This paper describes the inference layer of a production system: how a request gets a model, how that model gets watched, and what a turn costs. I built the layer and I run it. The application it serves is a co-dungeon-master for a live tabletop campaign, which matters here only as a set of workload shapes, four of them: long-form document generation, interactive chat, short lookups, and tool calls against two live databases. The routing, the observability and the cost work would look the same behind a support desk or a document pipeline.
One process, two tool servers, a database per workspace
browser panel (static, no framework, no build step)
| SSE stream
one Node/TypeScript process
|-- agent loop: capability-routed model, streaming,
| a cache-partitioned system prompt
|-- MCP manager: two stdio tool servers
| |-- campaign server: the knowledge store, per workspace
| '-- rules server: the shared rules reference
'-- SQLite, one database per workspace
|
LiteLLM broker --+-- aggregator: a live catalogue of 367 models
'-- first-party provider: a second route to the same model
One Node/TypeScript process: an HTTP server, a streaming chat endpoint, an agent loop. The browser panel is HTML, CSS and vanilla JS, no framework and no build step; the two libraries it needs are vendored as single files. State lives in SQLite, one database per workspace.
The two tool servers in the heading are Model Context Protocol servers over stdio, their tools bridged into the model's tool list at connection time. One is the campaign server: the knowledge store, one connection per workspace, with typed entities, facts, relations and provenance on every assertion. The other is the rules server, a single shared connection for the rules reference. The model holds no state between turns. Continuity is a property of the database, so a turn reads what is true at the moment the answer is written rather than what was true when the conversation started.
Model access runs through a LiteLLM broker in front of an aggregator (OpenRouter) and a first-party provider. One SDK client reaches both, the application holds no provider master key, and the broker is where the retry ladder finds a second route to the same model.
The tool list is also the correctness boundary. Exposed tools narrow by scope: a session at one scope is never handed another scope's write tools, so the wrong write is unavailable rather than discouraged. That removes a failure which reads like a model problem, where a tool that cannot succeed at the current scope returns an error the model then has to explain. That is a worse turn than never offering the tool at all.
No model name appears in application code
This codebase started with one model pinned in source as "best", and it was already a generation stale on the day it shipped. That is what a pinned model name gets you.
So a request declares requirements, all machine-checkable:
- tool-calling support, because an agentic turn without tool calls is broken rather than cheap
- a context floor
- an output ceiling that will not truncate a long document; a model that publishes no ceiling is refused outright
- quality floors read from published third-party benchmark indices
Those requirements resolve at request time against the live catalogue, 367 models at the last refresh. When something better ships, it wins on the next refresh with no code change.
Separately, a request declares an objective. There are three, and they name no models.
- Most value. The most capability headroom over the requirement, per dollar.
- Soonest per dollar. Minimise seconds times cost.
- Most capability. The most headroom over the requirement, cost breaking ties only.
The two declarations are orthogonal, and keeping them apart is what fixed the early defects. Nearly every one was the two axes collapsed into a single control. Asking for maximum capability on a one-line lookup priced and filtered it as a full document. Asking for the fastest option on a document routed it through the lookup requirements, where a model that stops at four thousand tokens can be handed a twelve-thousand-token document to write. Both were the same bug showing up from different directions.
Nothing here trusts a published number
Everything in this section is instrumented locally, because each published number failed in a specific way the moment it was trusted.
Availability
Published uptime answers whether an endpoint responds, not whether your requests succeed. Rate limits, 429s, context-length rejections and tool-format quirks all read as "up". One model in the catalogue reported 100 percent uptime while failing calls; I watched it do both at once. So the gate is the application's own error rate: a counter per model, a quarantine that backs off as failures repeat, and any success clearing the record.
The check that had never fired
The same quarantine mechanism runs in a sibling system, and auditing it started with one question: how many times has this guard actually fired? Zero. Not once, since the day it was written. One side of its comparison used catalogue ids and the other used dispatch ids, two namespaces that never overlap, so the guard compared names that could not match, excluded nothing, and every failing model kept serving.
A safety check that has never fired is worse than an absent one, because you believe it is protecting you.
A unit test of the comparison misses this, because each side is correct in isolation. You catch it by asking the running system for the count.
Speed
Every latency and throughput field was null on every endpoint checked, 16 of 16. Measuring passively from real turns is circular: an unmeasured model never wins a speed-sensitive objective, so it is never selected, so it is never measured.
Active probing breaks the circle: one small streamed completion per candidate, timed the way a real turn is timed and fed into the same moving average. The probe is stripped down on purpose:
- no workload content, because the probe reaches models the requirements have not vetted
- no tools attached, because a tool schema is thousands of prompt tokens, and first-token latency belongs to the model and the route
- sequential, never parallel, because parallel probes contend for one connection and would measure the harness instead of the model
- never on the turn path
- a call against a ceiling is refused before spending rather than after, and every skip is logged, because a silent cap reads as "everything is measured"
The decision record
Every routing decision is auditable. Per workload, the health payload carries the model chosen, its capability margin over the requirement, cost per turn, margin per dollar, estimated seconds, and which tie-break decided. Automatic selection without a visible decision record cannot be debugged and cannot be defended.
Vendor descriptions are not measurements
One catalogue entry advertises optimisation for tool use and coding agents. It measures 18.9 on the agentic index against a median of 20.8 across the 127 models that carry a score: below median on the exact axis its own description sells. Only the benchmark floor catches that.
Cost is the completion, the cache and the threshold ladder
Ranking on advertised input price picks the wrong model. Completion is priced several times higher than input, so a model that is cheap to ask can be expensive to answer. Each workload therefore carries a representative turn shape and is costed on that shape. Three things sit underneath it.
The cache line is most of the input bill. A large stable prefix replays every turn at cache-read rates, so prompt caching is a partitioning problem. The system instruction splits into a shared invariant prefix plus context-specific blocks, and the invariant part goes first so it is one shared cache entry rather than one per variant. There is a hard ceiling of four cacheable segments. The design sits exactly at it, and that fact is written in the file so the next person does not learn it from an invoice.
Some rates go up, not down. 49 of the catalogue's 367 models raise their rates above a prompt-token threshold, and the ladder is filed under pricing despite charging more. One workload was mispriced by exactly 2x until rates were resolved at that workload's own prompt size.
Delivery time is part of the price. 28 catalogue entries are asynchronous batch endpoints: the same model at identical benchmark scores for half the price, delivered in hours. On any cost-ranked objective a batch endpoint wins permanently, and no quality floor catches it, because its scores are identical to the interactive sibling's. The router was already doing exactly that. The price was fine, but the arrival time was the defect.
What a turn actually costs
The same short lookup turn, with the same requirements satisfied, priced at two different objectives: roughly $0.047 at maximum capability against roughly $0.001 at soonest-per-dollar. About fifty times, for a turn where the cheaper model meets the requirement. That is the spread between objectives on one workload.
Measured cost per turn across the four workload shapes: about $0.374 and $0.324 for the two long-form document workloads, $0.0022 for interactive chat, $0.0009 for short lookups.
A full catalogue sweep plus a timing run over the candidate set costs about three hundredths of a cent, $0.0003.
Measured time to first token, from the active probe:
- Fastest measured candidate. 658 to 833 ms to first token, at roughly 170 to 216 tokens per second.
- Second candidate. 2.0 to 2.9 seconds, at 53 to 80 tokens per second.
- Third candidate. 11.6 seconds, and it had been winning a lane on price alone.
That third one is unusable for an interactive lookup, and nothing published would have told you.
Retry steps sideways, never mid-stream
Retry moves to another model that already cleared the same requirements, and it prefers a different route to the same model over a different model, so a billing or provider problem never changes which model is writing. Nothing swaps mid-stream.
Degradation is ordered and explicit: relaxed requirements first, then a small pinned fallback list whose benchmark values are recorded verbatim with a capture date rather than guessed.
Errors are honest. Nothing substitutes a plausible neighbour for the thing that failed. That rule was paid for: one endpoint served a parent-scope document under a child-scope URL, a 200 with the wrong body, so users were handed the wrong document confidently and no log anywhere recorded a failure. I spent a while treating the complaint it caused as an architecture problem. It was one misrouted URL.
19 offline invariant tests run with no network and no credentials, so they pass or fail the same way on a cold clone.
Talk to me about it
If this is the kind of problem your team is working on, I am glad to go through any of it in more detail, including the parts that are still wrong. The application behind it is a tabletop tool. The routing, the observability and the cost work are the parts that transfer.
With Gratitude,
DGB