Running background jobs with FastAPI, Celery, and Redis

The first time an endpoint takes eight seconds because it's sending email, resizing an image, and calling a third-party API inline, you learn the same lesson everyone does: some work doesn't belong in the request. FastAPI is fast, but it can only respond as quickly as the slowest thing you make it do before returning.

The fix is to hand slow work to a background worker and return immediately. Here's the setup I use.

The pieces

Three moving parts. FastAPI takes the request and enqueues a job. Redis holds the queue. A Celery worker, running as a separate process, pulls jobs off the queue and does the actual work. The API and the worker never block each other.

# celery_app.py
from celery import Celery

celery = Celery(
    "worker",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

The broker is where jobs wait. The backend is where results go if you want to check on them later. I put them on separate Redis databases so a flush of one doesn't wipe the other.

Defining a task

A task is a plain function with a decorator. This one generates a report, which is exactly the kind of thing that's too slow to do inline.

@celery.task(bind=True, max_retries=3)
def generate_report(self, user_id: int, month: str):
    try:
        data = crunch_numbers(user_id, month)
        pdf = render_pdf(data)
        store_pdf(user_id, month, pdf)
        notify_user(user_id)
    except ExternalServiceError as exc:
        raise self.retry(exc=exc, countdown=30)

bind=True gives the task a reference to itself so it can retry. max_retries=3 with a countdown means a flaky external service gets three more attempts, thirty seconds apart, instead of failing the whole job on the first hiccup. This retry behaviour is most of why I reach for Celery rather than rolling my own queue.

Enqueuing from FastAPI

The endpoint kicks off the task and returns a job id straight away. The client polls or waits for a webhook.

@app.post("/reports")
def request_report(body: ReportRequest):
    task = generate_report.delay(body.user_id, body.month)
    return {"job_id": task.id, "status": "queued"}

.delay() serializes the arguments, drops the job on Redis, and returns in microseconds. The response goes out before the worker has even picked the job up. That's the whole point: the user gets an instant "we're on it" instead of staring at a spinner for eight seconds.

Checking status

Because we set a result backend, the client can ask how a job is doing.

@app.get("/reports/{job_id}")
def report_status(job_id: str):
    result = generate_report.AsyncResult(job_id)
    return {"status": result.status, "ready": result.ready()}

Status moves through PENDING, STARTED, SUCCESS, or FAILURE. For user-facing work I usually skip polling and have the worker push a notification when it's done, but the status endpoint is handy for dashboards and debugging.

The tenant trap

If your API is multi-tenant, this is where things quietly break. The worker runs outside the request, so it has no idea which tenant a job belongs to unless you tell it. The tenant id has to travel in the job arguments.

@celery.task
def generate_report(user_id: int, month: str, tenant_id: str):
    with tenant_scope(tenant_id):
        ...

Never rely on ambient request state inside a task, because there is no request. I made this mistake once and a report ran with an empty tenant context, which returned nothing and looked like a bug in the number crunching. It took an hour to realize the query filter was doing its job and the tenant was simply missing. I wrote up the general pattern in multi-tenant SaaS in ASP.NET Core; the same rule applies in Python.

Running it

In development you need two processes: the API and at least one worker.

uvicorn main:app --reload
celery -A celery_app worker --loglevel=info --concurrency=4

--concurrency=4 runs four jobs in parallel per worker. For CPU-bound work like PDF rendering, match it to your core count. For I/O-bound work that mostly waits on network calls, you can go higher. In production both run as separate containers and you scale the workers independently of the API, which is the real advantage. A traffic spike that needs more report generation gets more workers, without touching the API tier.

When you don't need Celery

For genuinely light work, FastAPI has BackgroundTasks built in, which runs a function after the response is sent, in the same process. That's fine for firing off one email. The moment you need retries, need the work to survive a restart, or need to scale workers on their own, that's when Redis and Celery earn their place. Fire-and-forget in the API process loses the job if the process dies; a job on Redis waits patiently for a worker to come back.

Keep the request fast, push the slow work to a queue, and pass every piece of context the worker needs in the payload. That's the whole discipline.