Frequently asked questions

Change two lines: point your existing OpenAI or Anthropic client at your namespace URL, and give it a Perdurance API key instead of your provider key. The routes have your provider’s own names and take your provider’s own bodies, and the answer comes back byte for byte as the provider sent it, so nothing else in your code changes. Raising max_retries is now safe, because an identical retry attaches to the execution already running rather than starting a second one.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.perdurance.dev/acme/prod/v1",
    api_key=os.environ["PERDURANCE_KEY"],   # sar_ab12cd34_… — not your provider key
    max_retries=5,                          # safe: a retry attaches, it does not re-generate
)

answer = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Say hello."}],
)

The long form →

POST to /requests with a dialect query parameter. It stores the request, answers 202 with a request id straight away, and runs the execution without you — which is the shape a batch job wants, or any caller that cannot hold a socket open for a long generation. The body is byte-for-byte the one the synchronous route takes, which is why the dialect moves into the query string: the same request submitted either way hashes the same and deduplicates together.

curl "$PERDURANCE_URL/requests?dialect=openai" \
  -H "Authorization: Bearer $PERDURANCE_KEY" \
  -H 'Content-Type: application/json' \
  -d '{ "model": "gpt-4.1", "messages": [{ "role": "user", "content": "Say hello." }] }'

# → 202 { "request_id": "01J8F2ZK9QX3M4NBVWT7", "status": "pending" }

The long form →

GET /requests/{id}. It answers with the stored record: the status, the execution history, and the provider’s own response embedded exactly as it was stored rather than re-serialised. This works for as long as your retention policy keeps the record, from any process and any machine — the id is the only thing you need to have kept. A failed request carries an error field instead of a response, and one that has not finished carries neither, which is what makes "no answer yet" a fact about the shape rather than something you parse for.

curl "$PERDURANCE_URL/requests/01J8F2ZK9QX3M4NBVWT7" \
  -H "Authorization: Bearer $PERDURANCE_KEY"

# {
#   "request_id": "01J8F2ZK9QX3M4NBVWT7",
#   "status": "succeeded",
#   "attempts": 1, "recoveries": 0, "next_retry_at": null,
#   "response": { "id": "chatcmpl-…", "choices": [] }
# }

The long form →

Ask the same retrieval URL for text/event-stream and it replays the stored chunks from the first one, however long ago they were stored, then live-tails to completion if the request is still running. Each event carries its chunk sequence number as its SSE id; send the last one you saw back as Last-Event-ID and the replay starts at the one after it. That is a chunk address rather than a timestamp, so a reader that died knows exactly where it was. Strip the id lines and what remains is exactly what the provider sent.

# from the beginning
curl "$PERDURANCE_URL/requests/$ID" \
  -H "Authorization: Bearer $PERDURANCE_KEY" \
  -H 'Accept: text/event-stream'

# or carry on from chunk 25 — the next event you receive is 26
curl "$PERDURANCE_URL/requests/$ID" \
  -H "Authorization: Bearer $PERDURANCE_KEY" \
  -H 'Accept: text/event-stream' \
  -H 'Last-Event-ID: 25'

The long form →

Point the framework’s underlying provider client at your namespace URL and leave every retry setting alone. An agent stack usually retries at three layers — the vendor SDK, the framework step, and a supervisor or queue — and none of them knows the others already tried, so one logical step can become eight provider calls. When every model call is identified by its body bytes, a retry at any layer re-sends those bytes and attaches to the execution already running. No agent code changes and the framework learns nothing about Perdurance.

from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model="gpt-4.1",
    base_url="https://api.perdurance.dev/acme/prod/v1",
    api_key=os.environ["PERDURANCE_KEY"],
    max_retries=5,
)

# LangGraph's node-level retry policy now re-attaches instead of re-generating.
# The same one line works for the OpenAI Agents SDK, the Vercel AI SDK and
# anything else that takes a base URL.

The long form →

List them. The synchronous routes answer with the provider’s response and no envelope, so they deliberately do not hand you an id — an id in the body is a field a vendor SDK would have to be taught to ignore. GET /requests returns them newest first, with each row carrying the status, model, dialect, key, backend, token counts, stored size and timestamps. The bodies are absent on purpose: they are the largest columns in the table and a page of them would be megabytes.

curl "$PERDURANCE_URL/requests?limit=10" \
  -H "Authorization: Bearer $PERDURANCE_KEY"

# paginate with the opaque cursor you were handed
curl "$PERDURANCE_URL/requests?limit=10&after=$CURSOR" \
  -H "Authorization: Bearer $PERDURANCE_KEY"

The long form →

GET /usage returns the totals and a breakdown attributed to each API key, which is what makes a key the unit of cost accounting. Two of its numbers are deliberately different: accepted_submissions counts every request that arrived, provider_calls counts the ones that actually reached a provider, and the gap between them is what deduplication saved you. A token count of null means no answer under that key reported one, which is not the same as a provider reporting zero.

curl "$PERDURANCE_URL/usage" -H "Authorization: Bearer $PERDURANCE_KEY"

# {
#   "accepted_submissions": 1841,
#   "provider_calls": 1792,          ← 49 retries that cost nothing
#   "api_keys": [{ "api_key_id": "…", "total_tokens": 1103986 }]
# }

The long form →

The same container image against a Postgres, dedicated to one customer, in our cloud or in your own cloud account — and we operate it either way. Every setting is a CTX_ environment variable with a default, so a deployment that says nothing gets one process serving both the API and the executor. Secrets are files in a mounted directory rather than variables, because a variable is readable from /proc and inherited by every child process, and these are the keys to prompt bodies and sealed provider credentials. The binary applies its own migrations at startup before it binds, so a rolling update is the ordinary upgrade path and there is no migration job to sequence.

# CTX_ROLE defaults to "both": one process serves the API and the executor.
docker run \
  -e CTX_DATABASE_HOST=postgres:5432 \
  -e CTX_DATABASE_USER=router \
  -e CTX_DATABASE_NAME=router \
  -v /run/secrets/perdurance:/app/secrets:ro \
  -p 8080:8080 perdurance/router

# the secret mount is a directory of files, one per secret:
#   credential-key  api-key-hash-key  database-password

The long form →

When a lost connection costs you a generation you have already paid for. Four situations make that common: long generations — reasoning models and agent turns — that outlast a socket, a proxy timeout or a serverless execution limit; clients that disappear, such as mobile, edge hardware and backgrounded browser tabs; retries you cannot switch off because a vendor SDK performs them by default; and any requirement for a per-request transcript and per-key spend as a system of record rather than a dashboard.

The long form →

When your calls are short, your clients are stable, and re-running a failed request costs a cent — the failures it removes are ones you are not having, and it is one more hop in your request path. It is also the wrong tool if you want a hosted model, because it has none and you bring your own provider credential; if you need translation between provider dialects, which it deliberately does not do; if what you need made idempotent is your own tools and side effects rather than model calls; or if nobody outside your company may operate what your prompts pass through, because we operate every deployment, dedicated ones included.

Marginally, on the path where nothing goes wrong. It is a proxy: there is one more network hop and one row written before the provider is called, which is milliseconds against a generation measured in seconds, and it depends on where it runs relative to you and to your provider. On the path where something does go wrong it is much faster, because a dropped generation is replayed from storage instead of being produced again. Perdurance is not a way to make inference fast; it is a way to stop paying for it twice.

You can, and the core is not complicated: hash the request body, claim the hash in the same transaction that stores the request, call the provider, store the answer. If your calls are not streamed and deduplication is all you want, that is a reasonable afternoon. What takes longer is the rest of it — persisting stream chunks as they arrive so a reader can resume mid-generation, leases so a worker dying mid-call re-dispatches rather than stranding the request, replay that returns the provider’s own bytes so an unmodified SDK still works, and a ledger that tells submissions and provider calls apart.

The long form →

That is the same architecture and a defensible thing to build. Three differences are worth weighing. A queue makes callers adopt an enqueue-and-poll shape, whereas here a client still calls its provider’s own route with its own SDK and receives the provider’s own answer. A queue hands back a finished result, whereas a stored stream can be replayed and resumed by chunk sequence while it is still being produced. And the deduplication claim commits in the same transaction as the submission, which a separate broker cannot do. The cost is another service to run — one image and one Postgres, with no broker.

Any provider reachable through one of three backend kinds: openai_compat for OpenAI and the many servers that copy its API, anthropic for Anthropic’s Messages API, and openrouter for OpenRouter including its Anthropic-family models. The backend kind fixes the dialect and the dialect has to match the route the request arrived on. Your own provider credential is used under your own contract with that provider; Perdurance does not resell model capacity.

The long form →

No. No prompt, response or stored chunk is used to train, fine-tune or evaluate any model. Requests, responses and provider credentials are your content; provider keys are encrypted before storage and are never returned by any route, to the console, the CLI or to you. If that is still more trust than you want to extend, a dedicated deployment keeps your traffic out of the shared service entirely.

The long form →

They are the three things that make up a URL and decide where a request goes. A tenancy is your organisation and is the first path segment. A namespace is an isolated set of provider connections, routing rules, keys and stored requests — production and staging want different ones — and is the second segment, so every call is addressed to /{tenancy}/{namespace}/v1. A backend is a connection to a provider: an address, a dialect and your credential for it. A routing rule maps model names onto a backend, and a request whose model matches no rule is refused rather than guessed at.

The long form →

Not by default. A retried request is a second call to your model provider: a second generation, a second bill, and a different answer, because sampling is stochastic. The first call usually keeps running, so the retry adds to it rather than replacing it. A retry is only safe when the server can recognise the second call as the same call and attach it to the execution already in flight. Perdurance does that by identifying a request by the bytes of its body for ten minutes, so an identical retry calls the provider once.

The long form →

Yes. Both the OpenAI and Anthropic SDKs retry automatically by default — two attempts, with backoff, on connection errors and on 408, 409, 429 and 5xx responses. If you have never set max_retries to zero, your application is already retrying LLM inference and already paying for a second generation whenever the first was still running.

The long form →

The upstream call keeps running and every chunk continues to be stored. The execution is not tied to the connection that started it, so a phone entering a tunnel, a closed laptop or a killed process does not cancel the generation. The result is collected afterwards by request id, from any process, whole or as a replayed stream.

The long form →

Byte-identical request bodies, in the same namespace, within ten minutes of the first submission. The hash is taken over the bytes you sent, so a re-serialised body with its JSON keys in a different order is a different request and is executed and billed as one. Two bodies differing only in whitespace are also different, and the same body in two namespaces is two requests, because a namespace is an isolation boundary. A timestamp or a fresh UUID in a system prompt is the usual way this is defeated by accident.

The long form →

It is longer than a slow generation and shorter than a session. Long enough that a client which dropped and reconnected lands on its own execution rather than starting a second one; short enough that a genuinely repeated question asked tomorrow is not silently answered from a recording. For anything longer-lived, fetch the request by its id, which works until your retention policy removes the record.

The long form →

No. A 504 with the code held_too_long means a synchronous call gave up waiting, not that the execution stopped. The request is still running and its answer will be stored, and the error message carries the request id. Re-send the identical body to attach to the same execution, or read the result back by id. Submitting a different body instead is the one mistake that starts a second provider call.

The long form →

They are the execution history on a stored record. attempts is how many times a worker took the request up; recoveries is how many of those followed a worker dying mid-flight rather than a transient error; next_retry_at is when the next attempt is due, if one is. A non-zero recoveries count is the durability machinery doing its job, not a fault to alert on — a worker that dies holding a request loses its lease, a sweep re-dispatches the request, and the counter goes up by one.

The long form →