Model Gateway
A provider-agnostic router that keeps generating when a key rate-limits, blocks unsafe content twice before it can reach a child, replays cached streams chunk for chunk, and paces output on purpose. Fifteen models across four providers behind one request shape.
- Role
- AI Engineer — LLM platform
- Domain
- Shared serving layer, two products
- Providers
- Anthropic · OpenAI · Google · Friendli
- Runtime
- TypeScript · Express · Redis · AI SDK
- Status
- Live in production
Outcomes
What was
broken
The problem as it actually presented itself, and the constraints that shaped every decision after it.
Two products, four providers, fifteen models, and a pile of API keys with independent rate limits. Every provider outage or per-key 429 was a user-visible failure, and each product was solving it separately and differently.
One of those products is used by children. Moderation could not be a single call that fails open when the endpoint is slow — an unavailable safety check has to mean a blocked request, which is an availability cost somebody has to consciously accept.
And a meaningful share of traffic was repeated: the same generation request, the same prompt, the same parameters. Paying a provider twice for a byte-identical response is the easiest money in the system to stop spending.
What it
runs on
Models, retrieval, serving, and the operational layer that keeps it honest in production.
- Priority key pool
- Redis rate-limit ledger
- Per-model fallback chains
- Moderation endpoint
- Chat-based moderation
- Fail-closed policy
- Request-hash response cache
- Cached stream replay
- Token pre-counting
- TransformStream
- Back-pressure control
- Four pacing profiles
How it
works
The pipeline end to end — each stage, and why it earns its place in the latency budget.
- 01
Key pool with a shared ledger
Every model-and-key pair has a Redis record holding remaining requests and tokens, its priority, and an unavailable-until stamp. A rate-limit response marks the pair unavailable for a cooldown window; a hard authentication failure marks it permanently. Because the ledger is in Redis, every API process shares one view of key health.
- 02
Priority routing and fallback chains
Models carry an explicit priority. The router selects the highest-priority available pair with meaningful headroom left, and walks a per-model fallback chain when none qualifies. Degradation is ordered and predictable instead of being whichever key happened to be tried first.
- 03
Two-layer moderation, fail-closed
Layer one is the provider's dedicated moderation endpoint — fast, cheap, official. Layer two is a chat-based check with prompts written specifically for an under-14 audience, and it only runs if layer one passes. A violation in either layer blocks. An error in either layer also blocks. That is the point of the policy, not a limitation of it.
- 04
Cache the stream, not just the completion
A language-model middleware wraps both generation and streaming. The cache key is a hash of the full request parameters. On a streaming hit, the recorded chunk sequence is replayed with response timestamps rebuilt, so a cached stream is indistinguishable from a live one on the client.
- 05
Back-pressure aware streaming
Output flows through a transform stream with a bounded buffer that applies flow control before it fills, protecting memory when a client reads slower than the model generates. Four pacing profiles — from fast to deliberately slow — are selectable per request by header, query parameter, or body, because a long explanation reads better paced than dumped.
- 06
Plan enforcement before spend
Auth validation, plan lookup, and token counting all happen before the model call. A request that would exceed a tier's input or output ceiling fails immediately and cheaply, rather than after the expensive part has already run.
Decisions
and their cost
Every choice below bought something and gave something up. The second half is the part worth reading.
Fail closed on moderation
For a product used by children, a moderation service that is down means requests are blocked, not allowed. It trades availability for safety in the only direction that is defensible, and it has to be a stated policy rather than an emergent behaviour of the error handling.
Rate-limit state in Redis, not in memory
In-process tracking meant each API instance had to independently rediscover that a key was exhausted, burning a 429 per process per window. Moving the ledger to Redis made key health a shared fact. The cost is a Redis round trip on the routing path — cheap next to a wasted provider call.
Two moderation layers instead of one good one
The dedicated endpoint is fast and general; the chat-based check is slower and understands the specific audience. Running both in sequence adds latency to every request and catches the category of content that a general-purpose classifier scores as borderline. For this audience, that is the right trade.
Pace the stream deliberately
Tokens arrive in bursts, which reads as jittery. Four configurable pacing profiles let each surface choose — fast for code generation where the user wants the result, slower for explanations where the user is reading along. It is a product decision implemented in the transport layer.
How it was
measured
Nothing shipped on intuition. Each number below is produced by a repeatable harness that gates deploys.
Violation or error in either layer blocks the request.
Across four providers behind one request shape and one key pool.
Cooldown window per key, permanent marking on hard failure.
Streaming hits replay recorded chunks with timestamps rebuilt.
Back-pressure engages before the buffer fills, protecting memory.
Token counting and plan limits run before any provider spend.
Guardrails
and safety
What stands between a good demo and something you can leave running unattended.
Moderation is fail-closed: any violation or any error in either layer blocks the request before generation begins.
Layer two runs prompts written for an under-14 audience, not a general-purpose safety classifier.
Token counting and plan ceilings are enforced before the provider call, so an over-budget request fails cheaply.
Rate-limit state is shared across processes in Redis; a key exhausted on one instance is immediately known to all of them.
The streaming buffer is bounded and applies flow control before capacity, so a slow client cannot exhaust process memory.
Moderation decisions are logged with their triggering layer and violation category for review.
What I'd
carry forward
The parts that generalise — earned the expensive way, on this build.
Fail-closed has to be designed, not defaulted. Almost every naive error handler fails open — you get the safe behaviour only if you write it down and test the error path deliberately.
Shared state beats clever local heuristics. Once key health lived in Redis, an entire class of duplicated-discovery waste disappeared.
Cache the transport, not just the payload. Caching completions but not streams means the surface users actually use is the one paying full price.
Serving-layer decisions are product decisions. Pacing, degradation order, and what happens when a provider is down are all felt by users long before they show up on a dashboard.