Event-Driven Architecture Without the 1 A.M. Chaos

The pitch for event-driven architecture is seductive: modules that don't know about each other, systems that absorb new features by just adding listeners, natural audit trails. The reality I've debugged at 1 a.m. is also real: invisible control flow, events lost in a broker nobody monitors, and a customer charged twice because a consumer replayed. Both pictures are true. This post is about getting the first one without the second.

What an event actually is (and isn't)

An event is a statement of fact, past tense: OrderPlaced, InvoicePaid, UserRegistered. It happened; the publisher doesn't care who's listening. Contrast with a command — SendWelcomeEmail — which is an instruction aimed at someone specific. Mixing these up is the root of most event-driven pain: when you publish SendWelcomeEmailRequested as an "event", you've built RPC with extra steps and none of the decoupling. Rule of thumb: if removing all consumers would make the publisher incorrect, it's not an event, it's a command wearing a costume.

The delivery guarantees nobody reads until it's too late

Whatever broker you use — RabbitMQ, Kafka, SQS, even Redis streams (comparison here) — you get at-least-once delivery in practice. Exactly-once is a marketing term for "at-least-once plus deduplication you still have to build". Two consequences fall out immediately:

1. Every consumer must be idempotent. Processing InvoicePaid twice must not send two receipts. The pattern is the same as webhook consumers: record the event ID in a table with a unique constraint, skip on conflict.
2. Publishing must survive crashes. The classic bug: you commit the DB transaction, then the process dies before publishing the event. Now the order exists and nobody ever heard about it. Enter the transactional outbox:

// Everything in ONE database transaction:
DB::transaction(function () use ($order) {
    $order->save();

    Outbox::create([                      // same DB, same transaction
        'event_type' => OrderPlaced::class,
        'payload' => $order->toEventPayload(),
        'occurred_at' => now(),
    ]);
});

// A relay (queue worker or cron) reads the outbox and publishes
// to the broker, marking rows as dispatched. Crash anywhere?
// The row is still there; the relay retries. At-least-once, honestly.

This one table eliminates the entire class of "the event never fired" incidents. It's twenty lines of code. I put it in every event-driven system on day one.

Eventual consistency: the part you have to design, not endure

Once modules communicate through events, the read side lags the write side — usually by milliseconds, occasionally (during incidents) by minutes. The mistake is pretending it doesn't; the craft is designing for it:

• UI honesty. After checkout, show "Order received — confirmation on its way", not a fake-synchronous "everything is done". Users tolerate async fine; they don't tolerate lies that unravel.
• Read-your-own-writes where it matters. The user who just changed their address should see the new address immediately — serve that read from the write model, and let the projections catch up for everyone else.
• Monitoring lag as a first-class metric. Consumer group lag (Kafka) or queue depth (RabbitMQ) belongs on the same dashboard as error rate. A consumer that silently stopped is the event-driven equivalent of a 500 — except nothing turns red unless you make it. Alert on lag and on absence, exactly like log-absence alerting.

Schema evolution, or: events are forever

An event published today may be replayed in two years by a consumer you haven't written yet. That makes event schemas the most durable contracts in your system — treat them with more ceremony than your REST API:

• Additive changes only. New optional fields: fine. Renaming or removing: that's a new event version (OrderPlaced.v2), with both published during migration.
• Fat enough to be useful, thin enough to age. Include the IDs and the facts that were true at the moment ("amount charged: 4900"), not the entire aggregate. Consumers needing more can query — a stale snapshot in an old event is a subtle data-corruption machine.
• One schema registry of truth — even if it's just a well-reviewed Contracts/Events directory in the modular monolith. In-process events today and broker events tomorrow can share the exact same classes, which makes the migration to distributed almost boring.

Where I reach for events — and where I refuse

Yes: cross-domain side effects (order placed → email, loyalty points, analytics, warehouse — four listeners, zero coupling), integration between services, audit-heavy domains, and anywhere new consumers appear regularly. No: anything where the caller needs the answer to proceed (that's a synchronous call, own it), single-consumer flows that would be a plain queued job with less ceremony, and — the hill I'll die on — validation. "Publish OrderRequested, a validator listens and maybe publishes OrderRejected" turns a 400 response into a distributed scavenger hunt.

Event-driven is a sharp tool: it buys real decoupling and pays for it in visibility. Add the outbox, make consumers idempotent, watch the lag, version the schemas — and the 1 a.m. version of you inherits a system that merely works, which at 1 a.m. is everything.

Designing an event backbone, or excavating one that grew wild? Bring the event catalog — half the fix is usually deleting events that were commands all along.