LLM inference retries

Retrying an LLM inference call is not safe by default. A retried request is a second call to your provider: a second generation, a second bill, and — because sampling is stochastic — a different answer. The first call usually keeps running on the provider's side, so the retry does not replace it. It adds to it.

This page is about making the retry safe: what breaks, what a correct retry needs from the server, and what changes in your code (nothing, if you are using a vendor SDK).

Why a retried LLM request bills twice

Every HTTP retry rests on an assumption that inference violates. POST is not idempotent, so a client that does not hear an answer cannot tell these two apart:

Three of those four rows charge you again, and the client cannot see which row it is in. So it retries, and pays. On a long reasoning generation that is not a rounding error — it is the whole cost of the request, doubled, for a failure that had nothing to do with the model.

What a safe retry requires

A retry is safe when the server can recognise the second call as the same call and attach it to work already in progress rather than starting more. That needs three things, and a provider API gives you none of them:

1. An identity for the request that the client can reproduce without having been told one.
2. A record of the execution that outlives the connection that started it.
3. A way to read the result back once the connection is gone for good.

Perdurance supplies all three, and the first is the one that makes it work with clients that have never heard of it.

How Perdurance makes the retry idempotent

A request is identified by the bytes of its body, for ten minutes from submission. Send the same bytes again inside that window and you attach to the execution already running; the provider is called once, no matter how many times your client asks.

attempt 1  ──POST body────▶  hash(body) → new record  ──▶ provider call starts
           ◀──connection dies──                              (still running)

attempt 2  ──POST same body──▶  hash(body) → same record ──▶ no second provider call
           ◀──the stored answer, or a live tail of it──

Nothing in that requires the client to hold state. It does not need an idempotency key it must generate, store and re-send — the body it is already re-sending is the key. That is precisely what the SDK's built-in retry does, which is why an unmodified SDK becomes durable by pointing its base_url at Perdurance.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.perdurance.dev/acme/prod/v1",
    api_key=PERDURANCE_KEY,
    max_retries=5,   # now safe to raise: each retry attaches, it does not re-generate
)

# If the connection drops, the SDK re-sends the identical body and lands on the
# execution already in flight. One provider call, one bill.
answer = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Explain idempotency."}],
)

What a retry does, per state

The second call's behaviour depends only on where the first one got to.

Note the streaming row: a retry mid-generation does not restart the generation and does not lose the part that already arrived. It replays what was stored and then joins the live tail.

Retries you should still make, and ones you should not

Retry the transport. A dropped connection, a read timeout, a 502 from something between you and us — re-send the identical body. That is free.

Do not retry by changing the body. A retry that re-serialises the request, adds a timestamp, regenerates a UUID or reorders JSON keys is a different request by definition, and will be executed and billed as one. Keep the exact bytes.

Do not treat 504 held_too_long as a failure. A synchronous call is held for at most the deployment's synchronous hold (five minutes by default). When a generation outlasts that, the connection is released with a 504 — but the execution continues and its answer is stored. The message carries the request id. Re-send the identical body, or read it back by id. Submitting a different body here is the one mistake that reliably costs real money.

Do not retry 422. A dialect mismatch is a configuration problem, and no record is written. See Routing.

Retries you do not have to make at all

Some retries are not yours. When a worker dies holding a request — a pod evicted, a node drained, a deploy rolling — its lease expires, the request is re-dispatched, and the execution picks back up without any client involvement. The record counts it:

A non-zero recoveries is the durability machinery working, not a fault to alert on.

The ten-minute window

The body hash maps to the same request for 600 seconds from submission. Inside it, identical bytes are the same request. Outside it, identical bytes are a new one and your provider is called again.

That length is a balance: longer than a slow generation, shorter than a session. Long enough that a client which dropped and reconnected finds its own execution; short enough that the same question asked tomorrow is not silently answered from a recording.

For anything longer-lived than the window, use the id. GET /requests/{id} works until your retention policy removes the record — see Idempotency and resume.