Background Jobs: Queues, Retries and Dead Letters
Anything slow, flaky or non-essential belongs in a queue. Here is how to run one without losing work.
Sending emails, resizing images, calling third-party APIs and generating reports are all poor fits for a request handler. They are slow, they fail for reasons outside your control, and the user does not need to wait for them. Moving them to background jobs keeps the application responsive and makes failures recoverable.
A queue decouples producing work from doing it. The web process enqueues a small message such as "send welcome email to user 42" and returns immediately. A worker process picks it up, does the work and acknowledges it. If the worker crashes mid-job, the message becomes visible again and another worker retries.
Retries need a policy. Use exponential backoff with jitter so a struggling downstream service is not hammered by synchronized retries. Cap the number of attempts and send exhausted jobs to a dead-letter queue where they can be inspected and replayed once the root cause is fixed. Alert when the dead-letter queue grows.
Because jobs will run more than once, they must be idempotent. Check whether the email was already sent, whether the file already exists, whether the record was already updated. Store job state in the database rather than in memory.
Keep jobs small and observable. Log start, end, duration and outcome with the job ID. A dashboard showing queue depth, processing time and failure rate tells you about problems long before users do.
