Job Queues Done Right: Retries, Dead Letters and Idempotent Everything

Job queues are where web applications hide their true complexity. The request/response layer is a well-lit showroom; the queue is the warehouse out back where emails, exports, webhooks, image processing and that one nightly reconciliation actually happen — and where failures go unnoticed until a customer asks why their invoice never arrived. After building and un-breaking a lot of these (Laravel/Redis, Hangfire, RabbitMQ consumers), here's the design playbook that separates a queue from a job graveyard.

Rule 1: jobs are messages, not closures with feelings

A job's payload should be IDs and intent, never objects and state: SendInvoiceEmail(invoice_id: 4211), with the handler re-fetching current truth at execution time. Serializing the whole invoice into the payload bakes in a snapshot that's stale by the time the job runs — the queue-flavored version of the fat-event mistake from the events post. (Laravel's SerializesModels gets this right by storing keys and re-querying; know that it does, because the "job saw old data" mystery is usually someone passing arrays around it.) Corollary: a job that no longer makes sense at run time — invoice deleted, user gone — should discard itself quietly, not throw and retry into a wall.

Rule 2: assume every job runs 0, 1, or 3 times

At-least-once delivery plus retries means duplicates are policy, not accident. Every handler needs an idempotency answer, and "it's probably fine" isn't one:

public function handle(): void
{
    $invoice = Invoice::find($this->invoiceId);
    if (! $invoice || $invoice->email_sent_at) {
        return;                                    // natural idempotency guard
    }

    Mail::to($invoice->customer)->send(new InvoiceMail($invoice));

    // Claim the fact atomically — affected-rows is the truth
    Invoice::whereKey($invoice->id)
        ->whereNull('email_sent_at')
        ->update(['email_sent_at' => now()]);
}

For side effects on external systems (charges! legal documents!), pair the guard with idempotency keys passed to the provider. And the "0 times" case: jobs get lost — Redis restarts with weak persistence, deploys race enqueues. Critical flows deserve a reconciliation sweep ("invoices issued >1h ago with no email_sent_at → re-enqueue") — the outbox pattern's pragmatic cousin.

Rule 3: retries are a designed policy, not a default you inherited

public int $tries = 5;
public function backoff(): array { return [30, 120, 600, 3600]; }   // exp + spread
public function retryUntil(): DateTime { return now()->addHours(6); } // hard deadline

public function failed(Throwable $e): void
{
    // The job is now DEAD — this must be an event somebody sees.
    Log::error('invoice email permanently failed', ['invoice' => $this->invoiceId, 'e' => $e]);
    // + metric increment + alert routing
}

Decisions embedded there: exponential backoff (immediate retries against a down provider are a self-DDoS), a retryUntil so time-sensitive jobs die honestly instead of firing "your table is ready" at midnight, and — the big one — failed jobs as first-class events. Distinguish retryable errors (timeouts, 429s) from permanent ones (validation, 404s): permanent failures should skip the retry ladder entirely ($this->fail($e)) rather than waste five attempts discovering the same 422. The failed-jobs table is your dead-letter queue: it needs an owner, an alert on growth, and a weekly review ritual — a DLQ nobody reads is just a slower /dev/null.

Rule 4: queues are lanes, and lanes have speed limits

One default queue means the 20-minute export ahead of every password-reset email. Segment by latency class, not by feature: critical (user-facing, seconds matter — password resets, broadcasts), default, and heavy (exports, media, bulk syncs) — with worker pools sized per lane so heavy can saturate without touching critical. Two ops rules complete it: timeouts below the worker's kill threshold (a hung job should die informatively, not zombify the worker), and memory-bounded workers that restart themselves (--max-jobs, --max-time) because long-lived PHP/.NET workers accumulate state entropy no matter what anyone promises.

Rule 5: a queue without metrics is a rumor mill

Four numbers, per lane, on a real dashboard: depth (backlog), oldest-job age (the honest latency metric — depth 10,000 of fast jobs is fine; depth 12 with a 40-minute-old job is an incident), failure rate, and throughput vs. worker capacity (are we keeping up, structurally?). Horizon gives Laravel folks most of this out of the box; everyone else wires it into the metrics stack in an afternoon. Alert on oldest-job age and failed-count growth — those two catch every queue pathology I've ever debugged, usually before users do.

The checklist

□ Payloads = IDs + intent; handlers re-fetch truth, discard stale work
□ Every handler idempotent (guard column / unique claim / provider key)
□ Backoff + retryUntil designed; permanent errors fail fast
□ DLQ owned, alerted, reviewed — with a replay path
□ Lanes by latency class; workers sized & self-recycling per lane
□ Oldest-job age on a dashboard with a pager attached

Get these six right and the warehouse out back runs itself — which is precisely the point of having one. Choosing the broker underneath it all is its own decision; the RabbitMQ-vs-Kafka post handles that argument. And if your failed_jobs table currently has 40,000 rows and no owner — I know that archaeology well.