Idempotency and resume
Idempotent LLM requests by body hash: why re-sending the same bytes never double-bills, what the ten-minute window is, and how a dropped stream is resumed.
The durability story rests on one mechanism: a request is identified by the bytes of its body. Send the same bytes twice inside the window and the second call attaches to the first execution rather than starting a second one.
That is what makes an unmodified vendor SDK durable. The SDK's own retry — the thing it already does when a connection drops — is the resume mechanism. It never learns a Perdurance concept.
The window is ten minutes
A body hash keeps mapping to the same request for 600 seconds from submission. Inside it, an identical body is the same request. Outside it, an identical body is a new one, and your provider is called again.
Ten minutes is longer than a slow generation and shorter than a session, which is the balance it is struck at: long enough that a client which dropped and reconnected lands on its own execution, short enough that a genuinely repeated question tomorrow is not silently answered from a recording.
Identical means byte-identical
The hash is over the bytes you sent. A re-serialised body with its JSON keys in a different order is a different request, and will be executed and billed as one. If you intend to resume, send the same bytes — not an equivalent object.
Ten minutes is what the hosted service runs. A dedicated deployment sets its own window, and one that has will say so in its own documentation rather than here.
What happens when the hash hits
| Record's state | Not streaming | Streaming |
|---|---|---|
running | Holds until it settles, then returns the stored answer | Replays the stored chunks, then live-tails to completion |
succeeded, failed | Returns what was stored | Replays the stored chunks |
The provider is called once across all of it. This is what "retries never double-bill" means concretely.
The two ways to come back
Re-send the identical body. Works on the synchronous routes, requires nothing to have been kept, and is what an SDK does by itself. Bounded by the ten-minute window.
Fetch by id. GET /requests/{id} works forever, or until your retention policy removes the
record. It needs you to have the id, which the synchronous routes do not hand you — see
Reading a request back.
For a stream, GET /requests/{id} with Accept: text/event-stream replays from the beginning,
and Last-Event-ID: 25 resumes at sequence 26. That is a sequence number, not a timestamp: it
addresses a chunk, so a reader that died knows exactly where it was.
Request statuses
| Status | Means |
|---|---|
pending | Stored and waiting for a worker to take it up |
running | A worker holds a lease on it and the provider call is in flight |
succeeded | The provider answered and the answer is stored |
failed | It will not be retried; error carries what the provider or the router said |
Alongside them, three fields say what execution has actually been through:
attempts— how many times a worker took the request up.recoveries— how many of those followed a worker dying mid-flight rather than a transient error. A non-zero value is the durability machinery doing its job, not a fault.next_retry_at— when the next attempt is due, if one is.
A worker that dies holding a request does not lose it. The lease expires, a sweep re-dispatches
it, and recoveries goes up by one.
What is not deduplicated
- Two bodies that differ only in whitespace or key order. Different bytes, different request.
- The same body in two different namespaces. A namespace is an isolation boundary and the claim does not cross it.
- The same body more than ten minutes later.
- A body whose model resolves through a
model_rewrite. The hash is taken before the rewrite, so the rewrite does not affect identity — but two different original names that rewrite to the same provider model remain two different requests.