Idempotency and resume
Idempotent LLM requests by body hash: why re-sending the same bytes never double-bills, what the ten-minute window is, and how a dropped stream is resumed.
The durability story rests on one mechanism: a request is identified by a hash of its body. Send the same body twice inside the window and the second call attaches to the first execution rather than starting a second one.
That is what makes an unmodified vendor SDK durable. The SDK's own retry — the thing it already does when a connection drops — is the resume mechanism. It never learns a Perdurance concept.
The window is ten minutes
A body hash keeps mapping to the same request for 600 seconds from submission. Inside it, an identical body is the same request. Outside it, an identical body is a new one, and your provider is called again.
Ten minutes is longer than a slow generation and shorter than a session, which is the balance it is struck at: long enough that a client which dropped and reconnected lands on its own execution, short enough that a genuinely repeated question tomorrow is not silently answered from a recording.
Identical means identical after canonicalisation
The hash is over the body parsed and canonicalised: object keys sorted, insignificant
whitespace gone. A re-serialised body whose JSON keys moved is the same request. Two things
are not normalised. Numeric spelling stays as you wrote it, so 1 and 1.0 are different
values, and array order is never sorted. Sending the same bytes is still the surest way to
land on the same request.
Ten minutes is what the hosted service runs. A dedicated deployment sets its own window, and one that has will say so in its own documentation rather than here.
What happens when the hash hits
| Record's state | Not streaming | Streaming |
|---|---|---|
running | Holds until it settles, then returns the stored answer | Replays the stored chunks, then live-tails to completion |
succeeded | Returns what was stored | Replays the stored chunks |
The provider is called once across all of it. This is what "retries never double-bill" means concretely.
A hash never hits a failed record. A terminal failure releases the idempotency claim, so
re-sending that body starts a new request, with a second platform fee and a second provider
call. The stored error is read by id with GET /requests/{id}. A failure that will be retried,
such as a provider 429 or a transport error, is not terminal. That record goes back to
pending, keeps its claim, and the same body still lands on it.
The two ways to come back
Re-send the identical body. Works on the synchronous routes, requires nothing to have been kept, and is what an SDK does by itself. Bounded by the ten-minute window.
Fetch by id. GET /requests/{id} works forever, or until your retention policy removes the
record. It needs you to have the id, which the synchronous routes do not hand you — see
Reading a request back.
For a stream, GET /requests/{id} with Accept: text/event-stream replays from the beginning,
and Last-Event-ID: 25 resumes at sequence 26. That is a sequence number, not a timestamp: it
addresses a chunk, so a reader that died knows exactly where it was.
Request statuses
| Status | Means |
|---|---|
pending | Stored and waiting for a worker to take it up |
running | A worker holds a lease on it and the provider call is in flight |
succeeded | The provider answered and the answer is stored |
failed | It will not be retried; error carries what the provider or the router said |
Alongside them, three fields say what execution has actually been through:
attempts— how many times a worker took the request up.recoveries— how many of those followed a worker dying mid-flight rather than a transient error. A non-zero value is the durability machinery doing its job, not a fault.next_retry_at— when the next attempt is due, if one is.
A worker that dies holding a request does not lose it. The lease expires, a sweep re-dispatches
it, and recoveries goes up by one.
What is not deduplicated
- Two bodies that differ in numeric spelling, such as
1against1.0, or in the order of an array. Canonicalisation sorts object keys and nothing else. - The same body in two different namespaces. A namespace is an isolation boundary and the claim does not cross it.
- The same body more than ten minutes later.
- The same body re-sent after a terminal failure. The claim was released with the failure.
- The same body after a routing change. The hash is taken after a
model_rewriteis applied and it covers the backend the rule chose, so repointing a rule makes the next submission a new request. The reverse also holds: two different names that rewrite to the same model on the same backend are one request, and the provider is called once for both.