Unstable AI inference

Most AI inference failures are not model failures. The model produced an answer; something between your process and that answer did not survive. A dropped socket, a serverless timeout, a backgrounded tab, a pod eviction, a rate limit at the wrong moment — none of them are the model's fault, and all of them cost you a generation you already paid for.

This page is a catalogue of those failures, what each one costs, and which of them go away when the execution outlives the connection that started it.

Why AI inference is less stable than the rest of your stack

An ordinary API call answers in tens of milliseconds. An inference call answers in seconds to minutes, and streams for most of that. That single difference breaks assumptions the whole transport stack is built on:

• A long-held connection is a fragile connection. Every load balancer, proxy, NAT and mobile radio between you and the provider has an idle timeout and a lifetime, and a two-minute generation gives all of them time to act.
• There is no cheap retry. A failed GET costs nothing to repeat. A failed generation costs the full price of the generation, twice, and answers differently the second time.
• The work is not resumable by the protocol. HTTP has no way to say "carry on from token 4,000". A dropped stream is a stream restarted from nothing.
• The failure is invisible from the outside. A connection that dies at 40 seconds looks identical whether the provider never saw the request or is 40 seconds into answering it.

The six failures

1. The client disconnects mid-stream

A phone enters a tunnel. A laptop lid closes. A tab is backgrounded and the browser throttles its timers. The socket goes and the tokens produced after it are gone with it — on a normal provider connection there is nothing listening, so there is nothing kept.

Perdurance persists each chunk as it arrives, whether or not anything is reading. When the client comes back it replays from the beginning, or resumes at an exact sequence number:

GET /requests/01J8F2ZK9QX3M4NBVWT7
Accept: text/event-stream
Last-Event-ID: 25

The reply starts at sequence 26 and live-tails to completion. That is a chunk address, not a timestamp, so a reader that died knows exactly where it was.

2. The client process dies

A deploy rolls, a container is OOM-killed, a worker is evicted. The generation in flight has no owner and no record. Nothing about the request survives except what you happened to log before you sent it.

With the request written down before the upstream call is made, the execution has an identity that does not depend on the caller existing. GET /requests/{id} works afterwards — from another process, another machine, another day.

3. Something in the middle times out

Idle timeouts of 30, 60 or 100 seconds are the default in most proxies, load balancers and API gateways. A generation that thinks for 90 seconds before its first token meets them all.

The synchronous routes hold a connection for up to the deployment's synchronous hold — five minutes by default — and when a generation outlasts even that, the connection is released with 504 held_too_long. That is not a failure. The execution continues and the answer is stored; the message carries the request id. This is the single most misread status in the API, because the reflex it invites — submit again with a different body — is the one action that starts a second provider call. See Errors.

4. Serverless hits its execution limit

A Lambda, a Cloud Run request, an edge function: each has a hard ceiling it cannot argue with, and a long generation walks straight into it. Holding a socket open for two minutes inside a function billed by the millisecond is also expensive on its own terms.

Split the work across two invocations:

# Invocation 1 — submit, return immediately
curl -X POST "$PERDURANCE_URL/requests?dialect=openai" \
  -H "Authorization: Bearer $PERDURANCE_KEY" \
  -H 'Content-Type: application/json' -d "$BODY"
# → 202 { "request_id": "01J8F2ZK9QX3M4NBVWT7", "status": "pending" }

# Invocation 2 — later, from anywhere
curl "$PERDURANCE_URL/requests/01J8F2ZK9QX3M4NBVWT7" \
  -H "Authorization: Bearer $PERDURANCE_KEY"

Neither invocation holds a connection for the length of the generation, and the generation does not care that neither is running.

5. The retry after an ambiguous failure

This is the expensive one, and it is the one your code performs whether or not you wrote it: the OpenAI and Anthropic SDKs retry automatically by default. A retry after a connection error is a second provider call for work that is very likely still in progress.

A body hash makes the second call attach to the first execution instead. LLM inference retries is the long form.

6. Transient errors from the provider or a worker

429 from a provider at its capacity, a 5xx, an upstream connection dropped part-way. These are the failures a retry genuinely fixes — but only if the retry is free.

A worker that dies holding a request does not lose it: the lease expires, a sweep re-dispatches it, and the record's recoveries counter goes up by one. A stream being replayed while that happens carries an event: failed with status: "superseded" — the dead attempt's partial output is discarded rather than spliced onto the new one, and re-opening the stream gives you the new attempt from its beginning.

What is still unstable afterwards

Being honest about the boundary matters more than the list above.

• The provider being down is still the provider being down. A stored 502 upstream_failed is a durable record of a failure, not a way around it.
• The model's output is still non-deterministic. Durability makes a generation retrievable, not reproducible in the sense of producing the same answer if genuinely re-run.
• Your tools are still not idempotent. See Agent SDK retries.
• Ten minutes is the window for body-identity. After it, identical bytes are a new request. The id works for as long as your retention policy keeps the record.

Sizing it up for your own traffic

The failures worth counting are the ones you are already paying for and probably not measuring: generations abandoned mid-stream, automatic SDK retries, and 504s treated as failures. All three are visible per key and per model in GET /usage, alongside the difference between requests accepted and provider calls actually made — which is the number that tells you what deduplication is saving. See Usage and storage.