The 200 OK that means nothing
A webhook endpoint returns 200 and everyone moves on. Three weeks later a customer says their order never showed up in the ERP. Support pulls the logs, finds the delivery attempt, finds your 200, and closes the ticket as "delivered." The record is still missing.
That gap between "we returned 200" and "the work happened" is where most enterprise integrations quietly rot. Webhooks look simple because the happy path is one HTTP POST. The difficulty is everything the happy path hides: the receiver that timed out after doing half its work, the retry that arrived while the first attempt was still running, the two events that landed out of order, the signature header that stopped matching after a key rotation nobody documented.
None of this is exotic. It is just how networks and queues behave, and it applies to a Shopify-to-warehouse sync exactly as much as it applies to Stripe.
Delivery is at-least-once, so duplicates are normal
Every serious webhook provider guarantees at-least-once delivery. That phrasing is doing real work. The sender cannot know whether your 200 came before or after your process crashed, so when a response is missing or slow it sends the event again. You will get the same event twice. Not as an edge case. As a Tuesday.
Stripe documents retries with exponential backoff for up to three days in live mode. Shopify makes eight attempts across roughly four hours. GitHub does not retry at all; failed deliveries sit there until someone manually redelivers them. Twilio Event Streams keeps trying until success or a four-hour cutoff.
Those four policies disagree on almost everything except the conclusion: idempotency is not something you bolt on after launch. It is the contract you build against from the first commit.
Signatures prove who sent it, not whether you already handled it
A signature tells you the payload came from the provider and was not tampered with in transit. It says nothing about whether you processed that event five seconds ago. Teams that treat signature verification as their duplicate protection end up double-shipping orders and double-crediting refunds.
The two controls are separate and you need both.
Verify against the raw request body, before any JSON parsing. This trips up almost every framework that auto-parses the body by default. Once a parser has normalized whitespace, reordered keys, or coerced a number, the bytes you are hashing no longer match the bytes that were signed, and verification fails with a confusing "invalid signature" that has nothing to do with the secret. Capture the raw bytes, verify, then parse.
Check the timestamp. Stripe signs a timestamp alongside the body and rejects deliveries outside a five-minute tolerance. Without that window, a captured request stays valid forever and an attacker can replay it whenever convenient. The Standard Webhooks specification, maintained by engineers from Svix, Zapier, Twilio, Kong and others, builds the same idea into a webhook-timestamp header.
Deduplicate on the event ID. Stripe uses event.id, Shopify uses the X-Shopify-Event-Id header, GitHub uses X-GitHub-Delivery, and Standard Webhooks defines webhook-id for exactly this purpose. That identifier stays constant across every retry and every manual redelivery. Hash the payload body instead and you will miss true duplicates, because retry counters, timestamps and field ordering shift between attempts. Key on received_at and the first redelivery sails straight through as a fresh event.
Enforce the dedupe with a database constraint, not a read-then-write check. Two retries arriving in parallel both read "not seen," both write, and now you have processed the event twice. A unique index on the event ID makes the second insert fail, which is exactly the behaviour you want.
Persist first, then do the work
The change that pays off most is to stop doing business logic inside the webhook handler.
The handler should verify the signature, write the raw event to a durable table or queue keyed by event ID, and return 200. That is it. A separate worker picks up the row and does the actual work: update the order, move inventory, sync the invoice, send the confirmation.
This inverts the failure modes in your favour. When the business logic throws, the event is already safely stored and the provider is not retrying something you already acknowledged. When the worker crashes mid-run, it restarts from a known state rather than from a provider retry that may never come. When the downstream API is down for forty minutes, nothing is lost, because nothing was ever only in memory.
It also fixes the timeout problem. Providers give you a few seconds. Stripe's guidance is to acknowledge quickly and process asynchronously. A handler that calls three external APIs before responding will occasionally take six seconds, get marked as failed, and earn you a retry for work that already completed.
Order is not guaranteed, so stop assuming it
Webhook delivery order is not part of the contract. A customer.updated event can land after the customer.deleted that supersedes it. A refund can arrive before the charge it refunds.
Two patterns handle this. The first is to treat each event as a notification that something changed, then fetch current state from the provider's API before acting. You reconcile against the source of truth instead of trusting a message that may be stale by the time you read it. This costs an extra API call and removes an entire class of bug.
The second, when you genuinely need sequencing, is to carry a version or sequence number and drop events older than what you have already applied. Combine that with a periodic full reconciliation job and you stop depending on message order for correctness at all.
On the sending side, the mistakes mirror
If you are the one emitting webhooks, the same discipline applies in reverse.
Never trust a customer-supplied URL. An endpoint URL is an input that can point at 169.254.169.254, at an internal admin panel, or at a hostname that resolves to a private range only on the second lookup. Resolve, validate, and reject private and link-local addresses at send time, not just at registration time. This is server-side request forgery, and webhook configuration screens are one of the easiest places to introduce it.
Retry with exponential backoff and jitter. Without jitter, every receiver that failed at the same moment retries at the same moment, and you build your own denial of service against a partner who is already struggling. Spread the retries.
Cap the total retry window and give up loudly. Silent abandonment is worse than failure, because nobody knows to look. After the cap, park the event in a dead letter queue that a human can inspect and redeliver. Then alert on queue depth, not just on individual failures.
Sign with a key you can rotate. Send the current signature and the previous one during rotation, space-separated in the same header, and let receivers accept whichever verifies. Standard Webhooks documents this approach, and it removes the three-day outage that key rotation otherwise causes.
Set an explicit timeout on outbound requests that is shorter than your retry interval. Otherwise a hung connection eats the window and the retry never fires.
Log the things that answer "did it actually arrive?"
Most webhook logging answers the wrong question. It records that a request came in. It does not record whether the effect landed.
Log the event ID, the signature verification result, the provider name, the HTTP status you returned, how long the handler took, and a correlation ID you can trace into the worker. Then log the worker's outcome separately: succeeded, retrying, or dead-lettered, with the error.
With those two streams joined on event ID, the support question changes from "we think it was delivered" to "event evt_1Qx… was received at 14:02:11, verified, enqueued, and completed at 14:02:13." That is a different conversation entirely, and it is the one your customer actually wants to have.
Alert on the ratio, not the count. A spike in dead-letter depth, a sudden run of signature failures after a deploy, or a worker whose p99 processing time has tripled will all tell you about a problem before a customer does. Signature failures clustering right after a release usually mean somebody changed the secret in one environment and not the other.
A migration path that does not require a rewrite
You do not need to rebuild the integration platform to get most of the benefit. Sequence the changes by risk.
First, add event ID deduplication with a unique constraint. This is small, additive, and stops the double-processing that causes the most visible damage.
Second, move the business logic out of the handler into a worker fed by a durable table. Even a plain database table with a status column beats doing the work inline.
Third, verify signatures against the raw body and enforce a timestamp tolerance. If you currently skip verification, turn it on in report-only mode first so you can see what breaks before it breaks production.
Fourth, add the reconciliation job. Once a day, pull current state for everything that changed and repair drift. This is the safety net that makes the other three survivable, because it catches the failures you did not anticipate.
Fifth, instrument delivery and processing separately and alert on both.
None of these steps require a new vendor. The pattern is older than any of the platforms it now serves.
What it comes down to
Webhook reliability is not a library choice or a framework feature. It is a set of assumptions you either write down on purpose or discover in production at the worst possible time.
Assume delivery is at-least-once. Assume order is not guaranteed. Assume the 200 you returned does not prove the work completed. Assume every inbound URL is hostile until resolved and checked. Design against those assumptions and the integration stops being interesting, which is what you want from plumbing.
The alternative is the ticket that opens three weeks after the fact, when the customer has already lost confidence and the log only tells you that a request arrived.
