Webhook Delivery Platform

The problem

A webhook is an HTTP call that one service makes to a customer’s server to say that something happened, for example that a payment went through. That server will be down sometimes, slow sometimes and, once in a while, hostile. A webhook platform, the system that makes those calls, sits in the middle and has to deliver the event anyway.

Sending the HTTP request is the easy part. The hard part is knowing what is true when something fails halfway: the worker dies after sending and before writing the result, the queue is down, the database fails. And the platform cannot tell two situations apart: The request never arrived, or it arrived and the answer was lost.

The decision

Deliveries are at least once, never exactly once. When in doubt the platform delivers again, so a receiver may see an event twice and must never miss one that could have been delivered. Every attempt carries the same webhook-id, and the receiver deduplicates by it.

After that, the order of the steps is the design (ADR 1 (opens in a new tab), an architecture decision record: a short note that explains one choice and what was rejected):

1. Claim the delivery with a lease, a claim that expires, so a worker that dies does not hold it forever. It is its own database statement.
2. Make the HTTP call, holding no database transaction.
3. Record the attempt and the next state, in one short transaction.
4. Delete the queue message, only after step 3 committed.

A crash before step 4 leaves the message in the queue. A crash before step 3 leaves the delivery enqueued with a lease that expires. Either way the work comes back and is done again. The worst case is a second HTTP call, and nothing is silently dropped.

Four terms in the drawing. The Idempotency-Key is a header the publisher sends so that a repeated request counts once. The reaper is a background job that gives back deliveries stuck as enqueued. HMAC is a signature that lets the receiver check who sent the request. The guard that refuses private addresses is protection against SSRF, where a customer points the platform at the internal network.

I rejected three alternatives (ADR 1 (opens in a new tab)):

• Exactly once. It cannot be offered honestly over a network. Systems that claim it mean at least once plus deduplication, which is what this builds in the open.
• At most once. It turns every network blip into a lost event.
• Recording the attempt before the HTTP call. It narrows the window without closing it, and it holds a connection across a call that can take 15 seconds.

The second decision is the queue. PostgreSQL stays the source of truth, and SQS carries only delivery ids. At this scale PostgreSQL alone would be enough. I chose SQS for the managed dead-letter queue, for something to scale workers on, and because I wanted to learn it. If this were a product with no learning goal, the PostgreSQL-only queue would win (ADR 5 (opens in a new tab)).

The evidence

Every row of the failure table in the README (opens in a new tab) names the test or script that causes the failure. Four of them:

• A worker is killed after the request left and before the attempt is recorded. The receiver gets the request twice with the same webhook-id, and the delivery still ends delivered. WorkerCrashTest (opens in a new tab)
• SQS is down, or the database fails at the claim. Deliveries wait and then flow, and the message is kept. InfrastructureOutageTest (opens in a new tab)
• Twenty identical publishes at once create one event. ConcurrentPublishTest (opens in a new tab)
• A rolling update of the worker under load lost 0 events in 3 runs, on local Kubernetes (kind). With the shutdown grace cut to 1 second it still lost 0 but produced 45 duplicates, so the test can fail. Read the results (opens in a new tab) or the script (opens in a new tab)

Measuring also found a bottleneck that reading the code did not show. Against a slow receiver the worker delivered about 13 per second, although it was configured for 50 in flight, and nothing was saturated: CPU was at 3% to 7% and the database pool was idle. The cause was that each loop waited for the slowest of its 10 messages before reading again. Reading only into free slots took the same workload from about 13 to about 47 deliveries per second, with the same machine and three runs each (ADR 4 (opens in a new tab)).

Every figure here comes from one machine that also ran the app, the database and the receiver, so the absolute numbers are lower bounds. The run on EKS happened once, for one day, and shows that the stack works there.

Limits, and what I would change

• It is not exactly once. Receivers must deduplicate, and the README says how (opens in a new tab).
• There is no ordering. If event 1 fails and is retried in 30 seconds while event 2 succeeds, the receiver sees 2 before 1.
• There is no latency bound. Between publishing and the first attempt there is up to one second of dispatcher tick plus the trip through SQS.
• After the last attempt, about 47 hours, only a manual replay sends it again.
• One master key encrypts every endpoint secret. Whoever has the database and that key reads them all.
• At 800 events per second the PostgreSQL rate limiter is the first thing to give. An in-memory limiter held 800, but it is only correct for one process.