One pipeline. One URL change.
Your clients keep speaking their own API, so the only change is the base URL and the key. Behind it, one pipeline admits, screens, routes, attempts, settles, and records every request, always in the same order. The specification fixes that order once, and any page that disagrees with it is a bug.
# Before: one provider, wired straight in.
client = OpenAI(api_key=PROVIDER_KEY)
# After: the base URL and the key change. The call does not.
client = OpenAI(
api_key=GATEWAY_KEY,
base_url="https://gateway.example.com/v1",
)
# Streaming still streams, and tool calls still arrive as tool calls.
stream = client.chat.completions.create(
model="corporatesafety/standard",
messages=[{"role": "user", "content": "Summarize this ticket."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")
What happens to one request
The full pipeline has fourteen steps. These are the seven a reviewer needs, in the order the specification sets.
- Admit. Load shedding happens here, before anything is parsed.
- Authenticate. The record opens at this step, so an unauthenticated request leaves a refusal event but no record tied to a caller.
- Screen. Every part of the request is screened: messages, system instructions, tool calls, query parameters, and documents the Gateway can read text from. Screening runs before policy, so your rules can match what screening found.
- Decide. One policy version evaluates the whole request and produces one decision object. Grounding and focus run after it, and anything they add is screened again.
- Route. Candidates are filtered by policy, attestation, capability, headroom, and circuit state, then ranked. Budget is reserved after routing, because the cost estimate needs the chosen model's prices.
- Attempt. The Gateway tries eligible targets in order, screens the output on the way back, and holds any proposed tool call until your policy has judged it.
- Settle and record. Actual cost replaces the reservation, savings are attributed, and the record closes under its retention class, refused requests included.
Numbers an engineering review asks for
The latency figures are acceptance targets, written as NFR-001 and NFR-002, not measurements of your traffic.
| Question | Answer | Where to verify |
|---|---|---|
| How much latency does this add? | The acceptance target is an added p50 of 25 ms and an added p95 of 60 ms at the Gateway boundary, with caching and refinement off and screening on. No other latency figure in the specification is an acceptance target. | Latency budget section, NFR-001 and NFR-002 |
| Do we change client code? | Only the base URL and the key. The Gateway speaks each supported API natively, so your SDK, streaming, and tool calls keep working. | The exhibit above, on your own service |
| How does failover handle a stream mid-flight? | The cutoff is the first event that carries content or a tool call. Before it, the Gateway holds the opening events, so a switch to another provider stays invisible to your client. After it, the Gateway never splices a second model's output onto the first, and your client gets an error event in its own API's format, naming the interruption and how many tokens were delivered. | Streaming state machine |
| Who pays for a failed attempt? | An attempt that fails before that cutoff is not charged to the caller. A delivered stream is charged for what the provider reported, or for the delivered text counted with that model's tokenizer when the provider reports nothing. | Partial billing rules |
| Does cost quietly override our quality bar? | No. Routing filters on capability before it ranks on cost, and a cheaper model carries traffic only after your administrator accepts a graded recommendation. Failover within one alias is different: it stays within the quality tier the alias already declares, so it needs no acceptance. | Routing evidence in the dashboard |
| What is the admin API surface? | An HTTP JSON API under /admin/v1/, described by an OpenAPI 3.1 document the deployment serves, covering policy versions, credentials, aliases, budgets, traces, and evidence packs. |
/admin/v1/openapi.json |
| What about error codes and replay? | The error-code registry is a published, versioned contract, and a code is never reused or deleted. A retired code stays listed as retired. HTTP status is the same across every supported API, and only the native error format differs. | /admin/v1/error-codes |
| What are the guards under load? | A timeout budget limits the time to the first token and any stall between chunks. Before sending, a token-budget guard checks that the input plus the requested output fits the model's context window, and refuses early instead of paying for a cut-off attempt. A circuit breaker for each model opens after repeated failures, so routing prefers a healthy candidate. | Guard behavior under your own load test |
| What are the rate limits? | Requests a minute by plan: Starter 1,000, Growth 10,000, Scale 100,000, each sized for five times the peak load at the top of that plan's spend range. | Pricing |
| Do you force upgrades? | We support the current major version and the one before it. A new major version opens a 6-month migration window, during which the previous version keeps full support. After that, it receives security fixes and severity 1 support only. | Supported version window |
Two things reviewers check twice
- Tool calls are governed inside the stream. The Gateway offers the model only the tools your credential, workspace, alias, and data class permit. It strips, rewrites, or refuses the tools a client declared, and records a fingerprint of each one it allows. A tool call arriving in a stream is held until it is judged, then passed, rewritten, or blocked.
- Three names resolve to real models you chose. The published catalog is premium, standard, and economy. Your administrator maps each name to a real model, and the mapping holds when a provider retires a version, so nobody has to edit a model name in code that day. A power user can still list the full catalog.
Where it runs
Every topology has the same parts. They differ in how many independent failure zones each part runs across.
| Topology | What it looks like | Who operates what |
|---|---|---|
| Single site | Instances in one site, control plane active with one standby, one site for the stores. | The operator runs the application and the infrastructure in the hosted form. |
| High availability | At least three data-plane instances across two or more failure domains, stores replicated across domains, synchronous for the journal and the counters. | Same split as single site, with the failure domains declared. |
| Multi-region | Instances in each region, requests stay in region, one primary region for writes, regional stores, and a global budget authority that regions lease against. | Your region set governs replicas, backups, and restore targets. |
| Air-gapped | Data plane, control plane, and every store inside your enclosed network, plus a local secret store. | You own availability, upgrades, patching, backups, and key management. The operator still notifies you of Gateway vulnerabilities, and updates arrive as a signed offline bundle. |
Point one service at it.
We admit customers in cohorts, in waitlist order. Don't miss the next one. When your slot comes up, you change one base URL on one real service, run your own test suite against it, and read the record of every request it sent. If we stop earning our fee, leaving is the same URL change, reversed.