← All writing
FIELD NOTES / Architecture · Reliability

The retry is part of the API

A request can succeed even when its response never arrives. Design the retry path before it becomes a production incident.

The ambiguous outcome

A client sends a request to create an order. The server commits the transaction, but the connection drops before the response reaches the client. From the client's perspective, the operation failed. From the database's perspective, it succeeded.

Retrying is the reasonable next step. If the API treats every request as a new intention, that retry can create a second order. Reliability requires distinguishing a new operation from another attempt at the same operation.

Give the intention an identity

An idempotency key identifies a logical operation. The client generates it before the first attempt and reuses it on retries. The server scopes the key to the authenticated caller and operation, then associates it with the request and its outcome.

A useful record includes the key, a fingerprint of the relevant request fields, processing state, and a reference to the result. Reusing a key with a different payload should be rejected rather than silently returning an unrelated result.

Make concurrent requests safe

Checking whether a key exists and then inserting it is not sufficient: two requests can both pass the check. Enforce uniqueness in the database and define what happens when a duplicate arrives while the original operation is still processing.

The outcome record and the business change should commit together when they share a database. If the business operation commits but the outcome record does not, the next retry is ambiguous again.

External effects need their own plan

A database transaction cannot undo an email that has already been sent or a payment accepted by another service. Carry the operation identity across boundaries where supported. Otherwise, record the intention durably and let a worker deliver it, with reconciliation for uncertain outcomes.

Questions worth answering

  • How long are keys retained, and what happens after expiry?
  • Can the caller retrieve an operation's status?
  • What response is returned while the original request is running?
  • Which failures are safe to retry?

The goal is a clear contract for repeated attempts, not a promise that the network will behave.

KEEP EXPLORING

There's more
behind the code.

Explore all writing ↗
✳ Ask about Majid