guide · 4 min read

Celery — one bot, every worker

Celery’s prefork pool spawns N worker processes after master startup. The post-fork hook used by module-level snitchbot.init() is fragile in this model; snitchbot.integrations.celery.install subscribes to Celery’s own worker_process_init signal instead, so every worker registers as its own client in the live dashboard.

Quickstart

from celery import Celery
from snitchbot.integrations.celery import install

app = Celery("tasks", broker="redis://...")
install(app, service="my-worker")

@app.task
def add(x, y):
    return x + y

Run with celery -A tasks worker -c 4. The dashboard shows four distinct rows — one per worker process — each with its own RSS, CPU, threads and fds.

Beat (scheduler)

Beat runs in its own process. Pass role="beat" so its lifecycle events stand out from worker events:

from snitchbot.integrations.celery import install

install(app, service="my-worker")  # workers
install(app, service="my-worker", role="beat")  # beat

(One install call per role; both are no-ops in the wrong process.)

Troubleshooting

Q: I see only one row in the dashboard, but I have N workers.

Check that install runs at module load time — the signal must be connected before workers are forked. If you call install inside a function that only runs in the master (e.g. a CLI entrypoint), the signal is not connected in workers and only the master registers.

Q: Does this work with the eventlet / gevent pool?

Yes — those pools are single-process; one row in the dashboard is correct.

What’s next

  • init() — full kwargs reference; everything you can pass to init() you can pass as kwargs to install().
  • Per-service topics — combine Celery workers and HTTP services in one supergroup.