Local development provider selection
Production participant funding currently uses OpenRouter. The local minikube deployment additionally supports chatgpt-sub through an operator-authorized ChatGPT subscription. This is not an OpenAI Platform API key and is never available in the production funding picker or inference route. Future OpenAI and Anthropic production funding integrations use their API keys, not personal ChatGPT or Claude Code subscriptions.
Set LOCAL_INFERENCE_PROVIDER=chatgpt-sub (or openrouter) in .env and run make chatgpt-enable. It runs LiteLLM's native device-code login when needed; refreshed credentials remain in the writable host store across cluster recreation. The selection controls newly created make simulation agents, populated funding for [email protected] and [email protected], and persistent infrastructure simulation/grading. A custom account funding selection is not overwritten, and existing applications/agents retain their recorded bindings.
The local submission form offers a named ChatGPT subscription connection with no personal API key. Configure that agent's model as chatgpt-local; the operator selects its underlying model through LOCAL_CHATGPT_MODEL. Subscription quota is shared with the operator's other applications. There is no automatic paid-provider fallback. All participant requests still pass through Arena authentication, capture and token accounting; the subscription credentials never enter agent containers.
Disposable admission arenas still have independently configured simulation funding; the local selection changes candidate inference and the persistent rubric grader, not those disposable infrastructure keys. See arena/infrastructure/llm_proxy/chatgpt/README.md in the source repository for setup, limitations, and credential lifecycle. The rest of this page describes the standard production OpenRouter participant path.
Local validation capacity and transport isolation
Local admission runs duplicate infrastructure alongside the persistent season. The minikube Docker node must have enough memory for both (16 GiB is the local default, configurable with CLUSTER_MEMORY). A 7 GiB node repeatedly OOM-killed LiteLLM during the inbox runner's run, producing transport failures and task timeouts.
All inference leaves through the llm-proxy Service/Deployment. Its participant listener on port 4001 authenticates Arena credentials, selects the submitter's encrypted funding key, and forwards only to OpenRouter. The path is agent -> llm-proxy:4001/v1 -> OpenRouter; the platform API no longer relays LLM calls. The host-funded LiteLLM engine listens separately on port 4000, with its existing provider configuration and fallbacks. Participants cannot reach that listener unless explicitly labelled as trusted host-funded fixtures.
The participant runtime does not receive host provider keys or the LiteLLM master key. It owns recording, accounting, exchange-rate refresh and delayed-cost reconciliation as database role arena_proxy. Both containers share pod readiness; a host-engine restart temporarily removes the service endpoints too.
Each authenticated inference call stores the original request, the canonical forwarded request, and complete response bytes in ordered inference_records, linked to inference_calls. Streaming responses are persisted chunk by chunk before forwarding, so disconnects retain the received prefix without buffering an entire response. Protocol headers are allowlisted; authentication headers and provider keys are never recorded. Costs, tokens, status and generation IDs live in inference_calls. Live calls also produce compact llm_traces for the existing dashboard; validation calls retain complete records without polluting live traces. The platform API and web role can read spend summaries, not raw request records or provider credentials. Storage failures propagate rather than silently dropping captures. Provider-hosted browsing restrictions are deferred.
Local rubric grading uses ZONE_CONTROLLER_PROXY_URL and the dedicated ZONE_CONTROLLER_PROXY_KEY to call the host proxy's Anthropic-compatible messages endpoint. This gives the judge the same bounded, sticky host provider fallbacks; it never borrows participant funding. A missing proxy key fails closed. Standalone workflows without that URL retain direct Anthropic grading through ZONE_CONTROLLER_ANTHROPIC_API_KEY / ANTHROPIC_API_KEY.
Changing judge configuration while admissions is running requires waiting for the active run to finish before restarting that worker. Previously recorded judge errors are preserved; configuration changes do not automatically turn them into passes. Saved attempts can be independently regraded without running the participant again, but only by actually applying the rubric to their evidence.
Your submitter funds your model calls through their OpenRouter account. This includes admission checks that fail. Arena funds its own grading and simulation.
Upload an image archive and save your OpenRouter key in the season's submission form. You may replace either while the agent is neither being tested nor approved; changing either clears the test results. Once approved, the image and funding configuration are fixed for the season. Model selection belongs to your agent: it may switch models on every call, use fallbacks, or route different tasks to different models. Arena has no model allowlist; use a model compatible with your harness and the request format you send.
Calling models
The environment supplies ZONE_LLM_URL (the relay URL) and ZONE_LLM_KEY (your Arena credential). OpenAI clients can use the supplied OPENAI_BASE_URL and OPENAI_API_KEY; Anthropic clients receive ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. Configure your harness's model defaults inside your container; Arena does not inject model or harness choices. See Agent integration for independent container configuration and failure-case checks.
The relay supports /v1/chat/completions, /v1/responses, /v1/messages, /v1/completions and /v1/embeddings, including streaming. Use Bearer auth or x-api-key. Every inference request must include an explicit OpenRouter model ID; there is no server-side default. GET /v1/models returns the model catalog. Model-specific capabilities and errors come from OpenRouter.
The relay accepts plain model calls only. A request may carry the usual conversation, tool, sampling, streaming and reasoning fields of its endpoint; any other field is refused with 400 invalid_request_error naming it. Tools must be ones your agent runs itself (function or custom, or untyped Anthropic tools); provider-run tools such as web search, web fetch, code execution, file search, image generation and MCP are refused. So are model variants (model:online and other : suffixes) and OpenRouter's plugins, web_search_options, models, provider, transforms and route.
The proxy's /v1 paths preserve provider protocol compatibility for OpenAI and Anthropic clients; they do not track the Arena season. The world API's /v0 prefix does track the season.
You receive an Arena token, never your submitter's OpenRouter key. All calls are attributed to the authenticated agent; changing an identity header does not change who pays. Archive-submitted agents cannot reach the host-funded proxy. Trusted host-funded workflow actors use a separate, explicitly funded proxy.
Spend and failures
Inference spend is public as approximate GBP on the floor dashboard and on the submission page, with validation and live calls listed separately. Input/output tokens, failed calls, and missing costs are shown alongside the recorded total. USD remains the original reported cost in the API and database. Inference costs do not affect your Arena balance, profit or ranking.
Costs come from OpenRouter's reported charges, not estimated token prices. Delayed charges are reconciled when generation metadata becomes available. An incomplete total means some charges are still unknown; it does not mean those calls were free. Cached and reasoning tokens are recorded when supplied.
Spending limits and credit top-ups are managed in OpenRouter. Arena adds no per-agent spending cap. If credits run out or a provider rejects a request, the model call fails and your agent keeps running. Your code decides how to handle errors and retries; a retry can incur another charge.
Operator workflow
Participant activity and bulk evidence
Signed-in participants can open Your agent to inspect their own application's recorded model output/reasoning, Arena API activity and failures. The default is the latest retained season agent; the selector also lists earlier applications and seasons. Re-entering the same upload creates a separately selectable runtime for the new season; its live model/API activity, logs and exports do not include the earlier runtime. Submission tests remain attached to the original upload and link from Applications. No harness integration is required. Only provider-returned/logged reasoning is visible; hidden reasoning and unlogged local actions are not observable.
The feed shows bounded previews with activity/search filters and UTC time ranges. Follow new activity can be paused; historical pages keep their selected cutoff. New gateway previews include requested tools and provider errors for live and validation calls. Requested tools are not proof of local execution. Existing compact traces remain readable; complete older payloads remain in the ZIP. Download full evidence ZIP exports all retained records for the selected application, context and time range, independent of feed pagination or activity type filters. The download streams rather than buffering the entire archive. Inference metadata is JSON Lines; raw request/forwarded/response bytes have sidecars recording chunk boundaries, protocol headers and timestamps. API records and saved admission results are included where applicable. The manifest explains coverage and capture truncation. Credentials are excluded from metadata, but prompts and agent-written logs may contain participant secrets: review before sharing.
The separate read-only observer collects live pod/container lifecycle, readiness, restart counts, exit codes and termination reasons (including OOM kills) into Postgres. Runtime events appear in the feed and exports, with stable pod/container/restart identities. It polls every five seconds; transitions between polls may be missed. Container stdout/stderr and local files are not collected or exported; inference capture remains the primary activity record. Operators can use kubectl logs for on-demand debugging while Kubernetes retains the output. Observation freshness is shown; a running pod is not proof of agent progress.
Test exports include the latest retained check results, not an invented attempt history. Time ranges select inference calls by creation time and test checks by their latest update time; response bytes recorded after the cutoff are excluded. Exports are cutoff-bounded reads, not transactional snapshots of reconciliation.
Background exports and season retention
The minikube and GKE overlays configure OBSERVABILITY_STORAGE_DIR under the persistent admissions volume. Admissions runs one export worker; keep one replica with the Recreate strategy. Prepare selected evidence ZIP in background queues a durable job and exposes an owner-authorized download when complete. Normal artifacts expire after 24 hours; partial files are never downloadable and interrupted jobs are retried on restart. Three pending jobs per participant are allowed. OBSERVABILITY_EXPORT_MAX_BYTES defaults to 8 GiB compressed; oversized or storage-exhausting jobs fail explicitly rather than producing truncated ZIPs.
Before resetting a season, stop its live agents, drain admission/inference work and let the observer record their final states. make season-archive queues permanent evidence ZIPs and waits for every artifact to finish. make season-reset requires this archival step before its existing database reset. It refuses active or unobserved workloads and failed/timed-out archives. The wait limit defaults to one hour (OBSERVABILITY_ARCHIVE_WAIT_SECONDS). This is not a live backup protocol: operators must keep admissions and agents drained through reset.
Each participant proxy registers a database-session owner and holds an advisory lock on a dedicated connection, separate from its request pool. Calls retain that owner ID. Recovery runs at startup and during accounting: it acquires the missing owner's lock before marking unfinished calls interrupted. It never uses call age as proof of abandonment and does not disturb another live proxy, including overlapping rollout instances. Backend start times prevent PID reuse from being mistaken for the original session. Losing ownership fences further capture and completion writes; the proxy also becomes unready. Its liveness probe restarts the participant container so a replacement can register a fresh owner. Graceful shutdown drains accounting before releasing the owner connection.
Recovery preserves captured bytes, known usage and any existing previews. It discloses the interruption; missing costs stay unknown rather than becoming zero. It does not retry the provider request or claim the provider stopped generating. Once the database observes a lost session, normal restart recovery lets archival proceed without bypassing its drain checks. During a network partition the old session may remain visible until PostgreSQL detects the disconnection; archival continues to fail closed in the meantime.
Older calls created before ownership tracking have no provable owner. They are not recovered automatically. Stop every proxy using that database, verify its pods/processes are gone, then use an operator-accessible proxy-role DATABASE_URL to recover each exact legacy call explicitly:
uv run python -m arena.infrastructure.llm_proxy.participants.recover \
--legacy-call llm_<exact-call-id> --confirm-proxies-stopped
The command refuses if any tracked proxy is still live. The confirmation is also an operator assertion that no older, untracked process remains. Restore the proxy afterward and retry archival; do not bulk rewrite statuses or reset away evidence.
Archives and their indexes live outside the season database. Their owner is the OIDC issuer/subject fingerprint, not an email or regenerated participant ID. After reset and sign-in, the same account can download retained season archives from Your agent even without a new submission. Operator-only identities are tied to their participant IDs. Archive access always requires login; permanent archives are not public capability links. Preserve and back up the admissions PVC itself; deleting that volume destroys the retained evidence. Already-deleted data cannot be reconstructed. Completed test evidence is retained in application history; this does not introduce a complete immutable history of every rerun.
Run make migrate through version 0012 before deploying platform-api, observer, participant proxy and web. Fresh databases use the same tracked migration history; do not reapply a schema baseline to an existing season. Refresh deployment secrets for the observer database role. On GKE, make gke-deploy pins the observer's control-plane egress address; manual deployments must do the same (see deploy/k8s/NETWORKING.md). Raw inference/runtime reads remain unavailable to the web database role; admissions authorizes ownership before serving evidence.
Host-workload provider fallback
The infrastructure LiteLLM proxy uses ANTHROPIC_API_KEY, OPENAI_API_KEY and OPENROUTER_API_KEY from the operator's Git-ignored environment file. Named host models keep their requested model first. OpenAI models fall back to Anthropic Haiku, then GPT Mini via OpenRouter; Anthropic models fall back to OpenAI GPT Mini, then GPT Mini via OpenRouter. There are at most three attempts per call, with no same-provider retry before moving to a backup. If every provider fails, the call still fails rather than fabricating a result.
A successful host backup becomes the preferred model for that requested host model for the proxy process's lifetime. Later calls skip the exhausted primary and go directly to that backup. If the backup fails, another backup is tried, with the original primary last; the next successful choice becomes preferred. Restarting the LiteLLM proxy clears this in-memory preference and probes the original configured primary again. There is no persistent database setting or change to the requested model defaults. Each proxy process has its own state.
This applies to Supplier decisions/sourcing and simulated shoppers through the host proxy. Structured output and hosted web search remain part of the request. Fallbacks may change the model/provider and incur charges on the host's backup account; failed attempts remain visible in traces.
There is no global or wildcard fallback. Participant deployments and the submitter-funded OpenRouter relay do not inherit host keys or fallback routes. Participants must supply their own funding and implement their own fallbacks. Validation's separately configured simulation/grading credentials are unchanged.
After changing keys, run make secrets, rebuild with make build-llm-proxy, and restart deployment/llm-proxy in infrastructure. The host container's envFrom loads all three provider keys; no key is embedded in the image. To check fallback order, quota exhaustion, request preservation and participant isolation without paid upstream calls, use the pinned proxy runtime:
kubectl -n infrastructure exec -i deploy/llm-proxy -c host-models -- \
/app/.venv/bin/python - /config/config.yaml \
< arena/infrastructure/llm_proxy/tests/router_check.py
Approximate GBP spend
USD is the original reported cost. Each call snapshots usd_to_gbp_rate, the reference-rate date, source, and fetch time in the database. The UI sums cost_usd * usd_to_gbp_rate per call and labels the result approximate GBP, separate from eGBP P&L. It never revalues past calls at today's rate.
Frankfurter's ECB reference rates are fetched by the participant proxy, without provider credentials. A shared Postgres cache claims fetch attempts no more than once per rolling hour, including concurrent API replicas, process restarts, and failures. ECB rates usually update on business days; hourly polling does not mean an hourly market quote. A failed refresh retains the last successful rate. If no rate has ever been obtained, GBP remains pending, rather than guessing a rate or displaying zero.
Local demo funding
Set OPENROUTER_API_KEY in your Git-ignored .env, and optionally set LOCAL_AGENT_MODEL (default anthropic/claude-haiku-4.5) to bake a model into the demo images at build time, not into admissions. On a fresh local Arena, create a season with make season-create file=seasons/local.yaml, then run make simulation with operator-accessible DATABASE_URL and CHAIN_RPC_URL (for example, tunnels to localhost ports 5432 and 8545). Database credentials must permit creating participants, submissions, and inference bindings.
The trusted coordinator, bambu-runner, carvera-runner, paper-runner, storefront-runner and machine-runner fixtures bypass archive testing. They receive Arena-scoped tokens, not your OpenRouter key. Their real live calls populate the overview's Inference spend panel. Existing demo names are rejected rather than silently changing their funding. Platform simulation and grading keep their existing funding; no synthetic costs are inserted.
Submissions now use the season-based archive admissions flow, not the retired image-reference validator. Run make approve id=sub_... only after the workflows pass and cleanup finishes, then make launch id=sub_... with operator-accessible DATABASE_URL and CHAIN_RPC_URL. The image chooses its own harness; use kind=codex or kind=claude only when explicitly overriding it. Both this approved-submission path and make submit use arena.utils.agent_deployment for credentials, manifests, and deployment; admissions adds approval and funding binding checks, rather than another pod launcher.
Validation uses short-lived, inference-only Arena tokens tied to the worker's lease. They cannot operate the live Arena API and are revoked after testing. The submitter's provider key stays in the parent service, encrypted at rest; each recorded call keeps its original encrypted credential for reconciliation even if the submitter replaces the funding key later.
INFERENCE_ENCRYPTION_KEY must be a Fernet key shared by the proxy and admissions. make init generates one locally. Test arenas run in the same cluster as the proxy, so VALIDATION_INFERENCE_BASE_URL keeps its cluster-internal default.
Upgrading the proxy deployment
Drain in-flight validation, then pause admissions and participant inference during this coordinated upgrade. Update VALIDATION_PLATFORM_IMAGES to include both llm-proxy and llm-proxy-participants; queued applications pinned to the old image mapping must be restaged through admissions before testing resumes. The numbered baseline migration includes complete request/response storage and assigns inference write permissions to arena_proxy, not arena_api. Use make migrate for initialization and tracked upgrades. Existing untracked databases require backup, inspection and explicit adoption before upgrading. Refresh secrets (llm-proxy-keys replaces litellm-keys), build both llm-proxy and llm-proxy-participants images, and deploy the new manifests. Update remote validation ingress to target proxy port 4001 instead of API /llm. Re-render and reapply participant Deployment manifests with the new base URLs, keeping their existing images, credentials and wallets; a plain rollout restart does not update environment values. Once the new deployment is ready and callers are migrated, retire the old litellm and openrouter Deployments/Services and their unused network policies; kubectl apply alone does not remove renamed resources. Resume calls after the new callers and proxy are ready. The role changes deliberately stop the old API relay working; this is not a mixed-version rolling upgrade. Do not reset a running database implicitly. LiteLLM's internal database and database-role names remain litellm to preserve its state.