Webhook ingestion with delivery guarantees

1. The constraint

The sender does not care about you.

That is the whole design. A webhook provider has its own retry policy, its own timeout, its own opinion about what your response code means, and no interest in your deploy schedule. Some retry aggressively and turn one event into six. Some retry once and give up. Facebook’s will keep trying and then stop, and an event you dropped is simply gone.

So the receiver cannot be a normal HTTP service that does some work and returns. It has to be a service whose only job is to not lose things, with the actual work happening somewhere it cannot affect the response.

2. The decision everything else follows from

Accept and process are different jobs and they are separated by a durable write.

Verify, deduplicate, persist, then acknowledge. In that order, with no step moved for speed. The acknowledgement is a promise that the event is durable, and returning 200 before the write turns a routine restart into permanent loss that nobody detects, because the provider believes it succeeded and will never send it again.

The raw payload is kept, not just the parsed one. Parsers have bugs and schemas change underneath you. Keeping the original bytes means a parsing mistake discovered three weeks later is a replay rather than an apology, and the archive has repaid that storage cost more than once.

Failures go to a dead-letter path a person can drain, never back to the provider. A handler crash is our problem. Bouncing it upstream converts an internal bug into lost data and into a provider quietly reducing its opinion of your endpoint.

Two of these are enforced at the accept path rather than written down as guidance. An unsigned payload is rejected by the receiver, so “only verified events enter” is a property of the code that runs and not a rule someone remembers. A replayed event is recognised by its idempotency key before any handler sees it, so a duplicate cannot become a second record even if every handler downstream is careless.

The ordering itself is the weaker part and it is worth being plain about. The guarantee holds because the write happens before the acknowledgement, and what protects that sequence is review and the fact that the people who work on it know why it matters. That is thinner than it should be for an invariant this load-bearing.

3. Why this is one service and not one per platform

Every platform that takes callbacks needs the same six things: signature verification, replay protection, durable receipt, ordered-enough delivery, a dead-letter path and an audit of what arrived. None of those are product features. All of them are hard to get right, and all of them are silently wrong until the day they matter.

Written per platform, each team gets its own subtly different bug. Written once, the delivery guarantee is a property of the infrastructure and every platform inherits it, including the ones that had not thought about retries at all.

The cost is a shared component with several consumers, which means a change has a blast radius and the ordering compromise in failure 6.3 is a decision made on everybody’s behalf. That is a real trade and it is the right one for a guarantee this specific.

4. What the guarantee buys the product

A hundred percent receipt rate is an infrastructure number that becomes a product one.

The person at the far end of a dropped webhook is a customer who sent a message and got no reply. They do not know a queue exists. They know they wrote to a company on Facebook and nobody answered, and the operator on the other side never saw it arrive, so both ends of the conversation believe the other is ignoring them. Every lost callback is one of those.

Social and messaging platforms also treat endpoint reliability as a signal about the integration itself. Miss enough callbacks and delivery degrades, the integration gets flagged, and in the worst case a platform review arrives that costs more time than the engineering ever did.

The teams building on top stop writing compensating logic, which is the quiet saving. No reconciliation job sweeping for messages that never landed. No “refresh to check if we missed anything” button, which is a confession rendered as a control. No support burden from conversations that arrived half-formed, and no support agent apologising for something the infrastructure did.

That is what the accept path buys. It looks like plumbing and it is a product commitment about whether a customer’s message can vanish.

5. What I would do differently

Build the dead-letter drain interface at the same time as the dead-letter queue. The queue is a morning of work and the tooling to inspect and re-drive it is a week, so the queue ships first and then poison messages sit in it for longer than anybody would admit, because looking at them is awkward. A dead-letter path nobody can comfortably drain is a landfill.

Make the accept-path ordering a test, not a habit. A contract test that drives the receiver, kills it between the write and the response, and asserts the event survives would turn the invariant in section 2 into something a build can check. It is an afternoon of work guarding the single decision the entire guarantee rests on, and it does not exist, which is the gap I would close first.

Treat silence as a signal from day one. Failure 6.4 is still open and it is the one I would front-load in a rewrite. Every alert here fires on something going wrong, and the failure that actually costs you is an upstream that stops sending, which produces no errors at all. A per-source expected-rate check would have cost an afternoon.

The figures on this page are thin and that is deliberate. The receipt rate is measured. The throughput and latency numbers this note used to carry were the prototype’s and are gone rather than replaced with estimates. Where a system’s real numbers have not been published, saying so is better than reaching for a plausible one, which is the whole argument of erratum 7.11.