Webhooks Done Right: Signatures, Retries and the 200-Fast Rule

Webhooks are the duct tape of the internet economy. Stripe tells you about a payment, GitHub about a push, Adapty about a renewal — all with the same trick: an HTTP POST to a URL you gave them. Simple enough that everyone implements them; subtle enough that almost everyone implements them wrong at least once. I know because I've been on both ends of the wreckage: the consumer that processed a forged event, and the producer whose retries hammered a customer's crashed endpoint into a deeper grave.

So here's the complete field guide — both sides of the wire, signatures included, with the code I actually use.

First principles: a webhook is an unreliable, unauthenticated POST

Strip the branding and a webhook has three inconvenient properties. It arrives at least once (so duplicates are normal, not exceptional). It arrives in no guaranteed order (the "completed" event can beat "created"). And it arrives from the open internet — anyone who learns your endpoint URL can POST whatever they like at it. Every good webhook practice is a response to one of those three facts. Every webhook incident I've seen traces back to ignoring one of them.

Signatures: proving the sender is the sender

The standard solution is an HMAC signature: producer and consumer share a secret; the producer computes HMAC-SHA256(secret, payload) and sends it in a header; the consumer recomputes and compares. No signature match, no processing. The good implementations (Stripe's is the reference) add a timestamp inside the signed content to kill replay attacks — an attacker who captures a valid delivery can't usefully re-send it hours later.

Producer side, in C#:

public static class WebhookSigner
{
    public static (string Header, string Payload) Sign(object evt, string secret)
    {
        var payload = JsonSerializer.Serialize(evt);
        var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();

        var signedContent = $"{timestamp}.{payload}";          // timestamp INSIDE the MAC
        var hash = HMACSHA256.HashData(
            Encoding.UTF8.GetBytes(secret),
            Encoding.UTF8.GetBytes(signedContent));

        return ($"t={timestamp},v1={Convert.ToHexString(hash).ToLower()}", payload);
    }
}

Consumer side — and the details here are load-bearing:

public static bool Verify(string header, string rawBody, string secret,
                          TimeSpan tolerance)
{
    var parts = header.Split(',')
        .Select(p => p.Split('=', 2))
        .ToDictionary(p => p[0], p => p[1]);

    var timestamp = long.Parse(parts["t"]);
    var age = DateTimeOffset.UtcNow.ToUnixTimeSeconds() - timestamp;
    if (Math.Abs(age) > tolerance.TotalSeconds) return false;   // replay window

    var expected = HMACSHA256.HashData(
        Encoding.UTF8.GetBytes(secret),
        Encoding.UTF8.GetBytes($"{timestamp}.{rawBody}"));

    // Constant-time compare — string == leaks timing information
    return CryptographicOperations.FixedTimeEquals(
        expected, Convert.FromHexString(parts["v1"]));
}

The three mistakes I've fixed in other people's verifiers, in order of frequency:

1. Verifying against re-serialized JSON. Your framework parsed the body, you serialized it back, one field reordered — signature dead. Always verify against the raw bytes as received. In ASP.NET Core that means reading the body before model binding; in Laravel, $request->getContent().
2. Using == for comparison. Early-exit string comparison leaks how many leading characters matched. FixedTimeEquals (or hash_equals() in PHP) exists for this.
3. No timestamp tolerance. Signature valid forever = captured payload valid forever. Five minutes of tolerance is plenty.

Consumer architecture: the 200-fast rule

Producers judge you by your status code and your latency. Slow endpoints get retried, retries pile up, and now you're being DDoSed by your own payment provider. The pattern that survives:

[HttpPost("webhooks/payments")]
public async Task<IActionResult> Receive()
{
    var raw = await Request.Body.ReadAsStringAsync();
    if (!WebhookVerifier.Verify(Request.Headers["X-Signature"], raw, _secret,
                                TimeSpan.FromMinutes(5)))
        return Unauthorized();

    var evt = JsonSerializer.Deserialize<WebhookEvent>(raw)!;

    // 1) Dedupe — at-least-once means duplicates WILL come
    if (!await _store.TryInsertAsync(evt.Id, raw))     // unique index on event id
        return Ok();                                   // seen it; ack and move on

    // 2) Queue the real work; ack immediately
    await _queue.EnqueueAsync(evt.Id);
    return Ok();                                       // < 100ms, always
}

Receive, verify, dedupe, enqueue, ack. The business logic runs from the queue where it can be slow, fail, and retry on your schedule. Storing the raw payload before processing also gives you the replay tool you'll eventually want: "re-process event evt_8842" beats "ask the vendor to resend".

Ordering gets the same treatment: never assume it. Either design handlers to be order-independent (fetch current state from the API instead of trusting the event's snapshot — the Stripe-recommended trick) or serialize per-entity in your queue.

If you're the producer

Building the sending side — say your SaaS notifies customers' systems — the bar is: sign everything (per-endpoint secrets, rotatable, shown once), retry with exponential backoff and jitter over hours not seconds, cap attempts and park failures in a dead-letter list the customer can see and replay from a dashboard, timeout deliveries at ~10s, and never follow redirects (a redirect to an internal address is an SSRF invitation). A delivery-log UI showing status, attempts and response codes will cut your webhook support tickets roughly in half — customers can see the 500 was theirs.

One more: send thin events. An id and a type, letting consumers fetch details from your API, ages better than fat payloads — it dodges stale-snapshot bugs and keeps your event schema from calcifying.

The checklist

• Consumer: verify raw-body HMAC, constant-time, with timestamp tolerance ✔
• Consumer: dedupe on event id, ack fast, process from a queue ✔
• Consumer: tolerate out-of-order; prefer fetch-current-state ✔
• Producer: per-endpoint secrets, backoff+jitter retries, dead-letter + replay UI ✔
• Both: log every delivery attempt with correlation ids — somewhere searchable ✔

Webhooks are one of those topics where an afternoon of design review prevents a quarter of firefighting. If your integration surface is growing teeth, I do exactly this kind of review — bring your signature scheme, I'll bring the red pen.