base_url.200 OK full of garbage gets ejected: routing on quality, not liveness.At Capital One I worked on a resiliency engine that moved payment traffic between cells the moment one misbehaved. $780B a year flowed through decisions like "this cell is lying about being healthy." When I started routing my own LLM traffic across providers, the same failure shapes showed up within a week.
So I rebuilt the same answers, small:
llm-resiliency-router,
an OpenAI-compatible proxy in ~1,500 lines of mostly-stdlib Go. Point any
SDK at it by changing base_url. Five ideas carried over from payments.
Each upstream lives in a cell: a ring buffer of outcomes driving
healthy → degraded → ejected → half-open → healthy, with hysteresis both
ways. Ejection takes consecutive hard failures; recovery must be earned
through a probe. Kill a provider mid-traffic and requests shift before any
client sees an error — the failover demo is
eight seconds long and shows the whole thing.
The failure availability checks cannot see: a provider returning 200 OK
full of garbage. A background canary asks every upstream a fixed prompt set
— arithmetic, JSON compliance — and grades the answers. Canary failures
feed the same health cells as real traffic, so a lobotomized provider is
ejected within one cycle without ever returning an error
(recorded). This is the line between a router
and a load balancer: routing on quality, not liveness.
The C1 engine shipped in observe mode before it touched traffic. The router
does the same. In learn mode the full decision loop runs — would have
skipped, would have failed over — but traffic never shifts, and every
decision lands in decisions.jsonl with its inputs. When the log's judgment
matches yours, one config line turns it on. There is no decision the router
can't explain.
Some hangs no timeout catches: 200, headers, then never a first token.
With hedge_after: 1s, a request with no first token by the deadline races
the next healthy upstream — first token wins, the loser is cancelled and
not blamed by the health model. We killed it; it didn't fail. The
hedging demo shows a stalled primary answered in
one second instead of never. Off by default: hedging spends money.
The fake provider speaks OpenAI wire format with scripted failure profiles —
flaky, slow, hang, stall, lobotomized — as YAML. The same profiles
drive the component tests and the recorded demos, so the tests and the
marketing can't drift apart.
| What | Value |
|---|---|
| Size | ~1,500 lines of mostly-stdlib Go |
| PRs (#1–#11) | 11, every one gated |
| Upstream kinds | 3 — anthropic, openai, ollama |
| Default mode | learn |
Ejection (eject_after) | 3 consecutive hard failures (5xx / 429 / timeout / canary miss) |
Half-open probe (probe_interval) | 15s |
Degraded signal (window / degrade_p95) | p95 over 60s above 8s |
| Upstream header timeout | 30s |
Hedging (hedge_after) | 0 — off by default |
| Chaos profiles | healthy, flaky, slow, down, hang, stall, lobotomized, brilliant |
Defaults from config.yaml, main.go and chaos/profiles.yaml at the v0.1.0 tag.
The router was built end to end inside a personal agentic development flywheel: spec first, an AI pair with hard local hooks, a 30-second PR gate, goreleaser releases, weekly public DORA metrics. Eleven PRs, every one gated. That story has its own writeup.
Numbers current as of v0.1.0.
Code: github.com/jamesponwith/llm-resiliency-router · Next: An agentic development flywheel for one
