A 504 is not a failed payment
Somewhere on a mobile connection, a customer taps Pay. The request reaches the payment provider, the card is authorized, the charge exists. Then the load balancer gives up at 30 seconds and returns a 504. The browser shows a spinner that never resolves, the customer taps again, and the second request creates a second charge. Nobody did anything careless. The system simply had no way to tell a retry from a new order.
This is one of the more expensive bugs in e-commerce, because nothing in the flow reports a problem. It shows up as a support ticket, then a refund, then a chargeback fee, then a customer who tells people the store charged them twice. The engineering fix is small: one unique index, one row written before the outbound call, and a client that sends the same key when it retries. The work is in getting the order of operations right.
The failure is mechanical, so the fix is too. The rest of this piece is the sequence that stops a retry from becoming a second order, the record that makes it work after the provider forgets, and the tests that prove the whole thing holds under load.
Every payment call has three outcomes, and clients see two
A request that crosses a network can succeed, fail, or land in the gap where the server did the work but the client never learned about it. From the client's side, the second and third cases look identical: no response, a timeout, or a connection reset. That ambiguity is the whole problem.
Automated retries make it worse. Payment SDKs retry on network errors, load balancers retry on upstream failures, mobile operating systems replay requests when an app resumes, and customers tap the button a second time because nothing appeared to happen. A checkout that treats each incoming POST as a fresh intent will create one charge per attempt. Five retries, five charges, one very confused customer.
You cannot fix this on the network side. Timeouts are not going away, and a client that gives up after 30 seconds is behaving correctly. The fix is to make the operation itself safe to repeat, which means the server needs to recognize the second request as the same piece of work rather than a new one.
Idempotency keys, without the ceremony
An idempotency key is a value the client generates once per logical operation and attaches to every attempt at that operation. The server stores the key alongside the result. When a request arrives with a key it has already seen, the server returns the stored result instead of doing the work again.
The details matter because they define what the guarantee actually covers:
- The client generates it. If the server mints the key on each request, a retry arrives with a new key and the mechanism does nothing.
- It carries enough entropy to avoid collisions. Stripe's documentation suggests UUIDv4 strings or similar, allows up to 255 characters, and warns against putting email addresses or personal identifiers in the key, since keys end up in logs and error trackers.
- It is bound to the payload. Stripe compares incoming parameters against the original request and returns an error when they differ, which stops a key from being reused to charge a different amount.
- It expires eventually. Stripe prunes keys after at least 24 hours, and a key reused after pruning starts a brand new request. That window is a minimum, not a promise of exact expiry.
Twenty-four hours sounds generous until you remember that a retry queue with exponential backoff can run for a day, and a support agent might retry a stuck order the next morning. The provider's memory is not your memory, which is why the durable record has to live on your side.
Where implementations break
The key is generated when the customer retries
The most common version of this bug is front-end code that creates a fresh UUID inside the click handler. First click gets key A, the timeout fires, second click gets key B, and the server correctly processes two distinct operations. The key has to be created and persisted before the first request leaves the browser, then reused for every attempt at that checkout until the order reaches a terminal state. Session storage is enough for a single tab. A server-side cart or intent record is better, because it survives a refresh and gives you something to reconcile against.
Check, then act
A read followed by a write is not atomic. Two requests can both read "no record for this key" before either inserts one, and both proceed to charge. Under real traffic this happens more than teams expect, because retries often arrive within milliseconds of each other.
The fix is to claim the key before doing any work. Insert the row with a unique constraint on the key, and treat a constraint violation as "someone else owns this operation." A pattern that holds up in production is an insert with a conflict clause that returns whether the row is new, followed by a short lease on in-flight records so a crashed worker does not leave the key stuck forever. Claim, call, record, commit. Nothing about the provider call happens before the claim.
Keys that are not scoped to a customer or merchant
A globally unique key table is a quiet security problem. If keys are not scoped to the authenticated account, one tenant can replay another tenant's key and read back a stored response, including a payment identifier they should never see. Scope the uniqueness to the pair of account and key, and reject any request whose payload fingerprint does not match the stored one. That fingerprint should hash the amount, currency, line items, and customer reference. A mismatch is a 422, not a replay.
Duplicate webhooks, duplicate fulfilment
Payments settle asynchronously, so most stores fulfil the order from a webhook rather than from the API response. Webhooks are delivered at least once. The same event will arrive twice, and if your handler is a bare "mark paid, decrement inventory, send email," the second delivery ships a second box.
Deduplicate on the provider's event identifier, not on the payload. Store the event id with a unique constraint, and wrap the fulfilment steps in a transaction that commits the event id alongside the state change. A webhook that has already been processed should return 200 quickly and do nothing else. Returning an error to force a retry is fine when a real dependency failed, and harmful when the failure was your own duplicate check.
The key expires before the retry does
A key that vanished after 24 hours is not a safety net for a job that retries for three days. If your only record of a payment attempt lives in the provider's idempotency table, then once that table forgets, nothing in your system knows the charge already happened. The next retry creates a genuine second charge, and every layer involved believes it is doing the right thing.
The durable record belongs in your database. A row that says step three called the provider with key K, the provider object id was ch_123, and the amount was 4200 USD is your memory of the operation. It outlives the provider's window, it survives a provider migration, and it is the thing you reconcile against when a response goes missing.
The outbound call ledger
A ledger table turns an ambiguous failure into a state you can query. The columns that earn their place:
- The idempotency key, unique per account, plus a hash of the request payload.
- A state field with three values: in flight, succeeded, failed with a definitive provider error.
- The provider's object id and the amount in minor units, recorded from the response, not from the request.
- An attempt counter and a lease expiry, so a stuck in-flight row can be reaped instead of blocking the key forever.
- Timestamps for creation, last attempt, and settlement.
With that table, the checkout endpoint becomes a short state machine. On request, claim or load the row. If it already succeeded, return the stored response. If it is in flight and the lease is fresh, return a 409 and let the client poll. If the lease is expired, take it over, reconcile with the provider, and only then decide whether to call again.
Reconciliation is the part teams skip. Before retrying an unknown outcome, ask the provider what happened: search for the object by your own key or by a metadata field you control, or list recent charges for the customer within a narrow window. A read that confirms the charge exists beats a write that hopes it does not.
What the client needs to see
The API contract matters as much as the storage. Return three distinct outcomes, because the front end has three different jobs:
- Succeeded. The order is paid. Show the receipt and stop accepting attempts with that key.
- Definitively failed. The provider rejected the card. Show the reason, and let the customer start a new attempt with a new key.
- Unknown. The request went out and the answer did not come back. Show "we are confirming your payment" with a poll or a page refresh, never "payment failed." Saying failed when the charge may exist is how customers pay twice on purpose.
Two small front-end decisions remove a lot of duplicate volume. Disable the pay button on submit and keep it disabled until the state resolves. Use a POST, redirect, GET flow so a browser refresh on the confirmation page cannot resubmit the charge. Neither is clever, and both prevent a surprising share of the incidents that land in support.
Tests to run before real money moves
Most idempotency bugs pass a manual test with a fast connection. These five scenarios find the rest:
- Send the same request twice with the same key. Confirm one charge exists and the second response matches the first, including status code.
- Fire two requests with the same key at the same moment. Confirm exactly one charge, and confirm the loser gets a 409 or the same body rather than an error page.
- Kill the process between the provider call and the database commit, then retry. This is the only way to prove your reconciliation path works, because it is the failure your happy-path tests never hit.
- Reuse a key with a different amount. Confirm the API rejects it instead of charging the new amount or replaying the old one.
- Deliver the same settlement webhook twice. Confirm one fulfilment record, one inventory decrement, one email.
Run these against the provider's test mode with the same code path as production. A reconciliation script that only runs in a staging branch is not a reconciliation script.
What a duplicate charge actually costs
The direct loss is the refunded amount, which is the smallest part. Payment processors charge a fee on the original transaction that is not returned with a refund, and the interchange on a captured then refunded card is simply gone.
Then there is the dispute. Chargeflow's 2026 chargeback statistics put the average cost of a dispute at about $84, and note that duplicate charges and billing errors account for roughly 5 to 10 percent of chargebacks. Clearly Payments reports that merchants win only about 20 to 30 percent of disputes, so a duplicate charge usually means a loss plus a fee plus a mark against the account's dispute ratio, which processors watch when they set reserves.
The pattern is not limited to card payments. APQC research cited by expense management firms puts duplicate or erroneous payments at 0.8 to 2 percent of total disbursements, and IOFM estimates companies lose up to 1.5 percent of outgoing cash flow to duplicates. Those numbers come from accounts payable, where human approval steps add their own failure modes, but the mechanism is the same: an operation with no memory of itself.
Against that, the cost of the fix is a table, an index, and a few dozen lines of state handling. On most stacks the work is a day, and it can be retrofitted to an existing checkout without redesigning the payment flow.
Start with the order of operations
If a team takes one thing from this, it should be the sequence. Persist the key before the first request. Claim it atomically before any outbound call. Write down what the provider said, in your own database, before you tell the customer anything. Reconcile before you retry an unknown state. Those four steps cover the failures that a timeout, a refresh, and a nervous customer will otherwise turn into a duplicate charge.
Everything else, from key length to lease duration, is tuning. The ordering is what makes checkout safe to repeat.
