Debug RSS spikes

The bot pings: memory ceiling crossed. Half an hour later the pod was OOM-killed. You want to know what the curve looked like and what the service was doing at the spike — before SSHing anywhere.

When to use

  • RSS or CPU anomalies that fire often enough to want a fast triage loop.
  • You don’t have a metrics stack, or the metrics stack doesn’t sample fast enough to see the spike.
  • The process forks subprocesses (Playwright, ffmpeg, multiprocessing pool) — own RSS lies, you need the aggregate.

The recipe

1. Tune the detector to the shape of your problem

import snitchbot
from snitchbot import AnomalyConfig, RssAnomalyConfig, CpuAnomalyConfig

snitchbot.init(
    "checkout",
    anomaly=AnomalyConfig(
        rss=RssAnomalyConfig(
            max_mb=800,           # ceiling — hard limit, severity=error
            spike_ratio=1.6,      # spike — current / baseline >= 1.6
            min_spike_mb=80,      # ignore spikes smaller than 80 MB
            duration="1m",        # current window
            baseline_duration="20m",  # baseline window
        ),
        # If the service spawns child processes, opt into aggregate detection.
        total_rss=RssAnomalyConfig(max_mb=2000, spike_ratio=1.8),
        cpu=CpuAnomalyConfig(max_percent=90, spike_ratio=2.5),
    ),
    sample_interval_sec=5,  # default; raise to 10–15s for low-overhead idle services
)

max_mb is the ceiling mode — fires when current RSS exceeds the limit. spike_ratio is relative growth — current window mean ÷ baseline window mean. Set any of max_mb, spike_ratio, drop_ratio to None to disable that mode independently.

2. Attach context that survives into the alert

Inside request handling, wrap the unit of work so the alert carries enough to reproduce:

with snitchbot.request_context(
    trace_id=request_id,
    user_id=user_id,
    endpoint="/checkout/finalize",
):
    finalize_order(...)

If the anomaly fires while this block is on the stack, the alert renders a Context block with those keys. That is usually enough to find the offending request in the application log.

3. Pull a chart from the sidecar after the ping

In the chat, send to the bot:

/chart total_mem 30m
/chart cpu 10m

The sidecar renders an ASCII chart from its in-memory vitals deque. The window must fit the deque — see sample_interval_sec × maxlen in the warning the client logs at startup. Default config holds ~5 minutes; raise the window via custom anomaly durations or accept that the deque caps the chart.

What the reply looks like

📊 chart · checkout · total_mem · 30m
━━━━━━━━━━━━━━━━━━
RSS (30m)  cur=1842.3MB  min=412.8  max=1908.6
 1908.60  ┤                                ╭─╮  ╭─
 1659.27  ┤                              ╭─╯ ╰──╯
 1409.93  ┤                          ╭───╯
 1160.60  ┤                      ╭───╯
  911.27  ┤                ╭─────╯
  661.93  ┤      ╭─────────╯
  412.60  ┼──────╯
2026-05-05 12:33:11 UTC -> 2026-05-05 13:03:12 UTC

How to read it:

  • Y-axis is the metric value in its native unit — MB for mem/total_mem, percent for cpu/total_cpu (one core = 100%), absolute count for fds/threads.
  • X-axis is time, oldest sample on the left, newest on the right. The two timestamps under the chart bound the window.
  • cur / min / max in the header are computed over the full window — cur is the most recent sample, not the rightmost printed point (the chart is downsampled to fit the configured chart_width).
  • A flat line followed by a slope is the typical leak signature: the service was idle, then started growing without recovery. A leak that releases looks like a sawtooth.
  • A single tall column with no trailing decay means the spike happened between samples — raise sample_interval_sec granularity or shorten the window; one missed sample at 5 s sampling is invisible at a 30-minute window.
  • If the chart says No live clients with vitals data., the sidecar has no client whose vitals_status == 'ok' for the resolved scope. In a forum-mode topic this is service-scoped (F7); in the General chat, it picks the first live client.

To compare own vs total, fire two requests:

/chart mem 30m
/chart total_mem 30m

A flat mem line under a rising total_mem line is the textbook subprocess leak — see Track memory across subprocesses.

Notes

  • max_history_seconds() is computed from your AnomalyConfig; the deque is sized to fit the longest baseline window across all enabled detectors. If you want /chart … 1h to work, set at least one detector’s baseline_duration to "1h".
  • total_rss and total_cpu aggregate the parent process and all descendants from psutil.Process.children(recursive=True). Children that exit between samples or deny access (AccessDenied) are silently skipped (V11). The /status row shows own / total / children — large gaps between own and total mean a leak in a worker, not in the parent.
  • The detector compares window means, not raw points. A single one-second spike won’t trip spike_ratio if the window is "1m" — that is the design. If you need point-in-time alerting, lower duration or use the max_mb ceiling.
  • The full anomaly model is in Configuring anomalies; per-class details in RssAnomalyConfig and CpuAnomalyConfig.