This paper describes the inference layer of a production system: how a request gets a model, how that model gets watched, and what a turn costs. I built the layer and I run it. The application it serves is a co-dungeon-master for a live tabletop campaign, which matters here only as a set of workload shapes, four of them: long-form document generation, interactive chat, short lookups, and tool calls against two live databases. The routing, the observability and the cost work would look the same behind a support desk or a document pipeline.
One process, two tool servers, a database per workspace
browser panel (static, no framework, no build step)
| SSE stream
one Node/TypeScript process
|-- agent loop: capability-routed model, streaming,
| a cache-partitioned system prompt
|-- MCP manager: two stdio tool servers
| |-- campaign server: the knowledge store, per workspace
| '-- rules server: the shared rules reference
'-- SQLite, one database per workspace
|
LiteLLM broker --+-- aggregator: a live catalogue of 406 models
'-- first-party provider: a second route to the same model
One Node/TypeScript process: an HTTP server, a streaming chat endpoint, an agent loop. The browser panel is HTML, CSS and vanilla JS, no framework and no build step; the two libraries it needs are vendored as single files. State lives in SQLite, one database per workspace.
The two tool servers in the heading are Model Context Protocol servers over stdio, their tools bridged into the model's tool list at connection time. One is the campaign server: the knowledge store, one connection per workspace, with typed entities, facts, relations and provenance on every assertion. The other is the rules server, a single shared connection for the rules reference. The model holds no state between turns. Continuity is a property of the database, so a turn reads what is true at the moment the answer is written rather than what was true when the conversation started.
Model access runs through a LiteLLM broker in front of an aggregator (OpenRouter) and a first-party provider. One SDK client reaches both, the application holds no provider master key, and the broker is where the retry ladder finds a second route to the same model.
The tool list is also the correctness boundary. Exposed tools narrow by scope: a session at one scope is never handed another scope's write tools, so the wrong write is unavailable rather than discouraged. That removes a failure which reads like a model problem, where a tool that cannot succeed at the current scope returns an error the model then has to explain. That is a worse turn than never offering the tool at all.
No model name appears in application code
This codebase started with one model pinned in source as "best", and it was already a generation stale on the day it shipped. That is what a pinned model name gets you.
So a request declares requirements, all machine-checkable:
- tool-calling support, because an agentic turn without tool calls is broken rather than cheap
- a context floor
- an output ceiling that will not truncate a long document; a model that publishes no ceiling is refused outright
- quality floors read from published third-party benchmark indices
Those requirements resolve at request time against the live catalogue, 406 models at the last refresh. When something better ships, it wins on the next refresh with no code change.
Two thirds publish no agentic score at all, so a quality floor excludes them by measurement rather than by opinion. The last two columns are the traps: a batch endpoint undercuts its interactive twin at identical benchmarks, and a surcharge ladder makes the sticker rate the wrong rate.
Separately, a request declares an objective. There are three, and they name no models.
- Most value. The most capability headroom over the requirement, per dollar.
- Soonest per dollar. Minimise seconds times cost.
- Most capability. The most headroom over the requirement, cost breaking ties only.
The two declarations are orthogonal, and keeping them apart is what fixed the early defects. Nearly every one was the two axes collapsed into a single control. Asking for maximum capability on a one-line lookup priced and filtered it as a full document. Asking for the fastest option on a document routed it through the lookup requirements, where a model that stops at four thousand tokens can be handed a twelve-thousand-token document to write. Both were the same bug showing up from different directions.
Bars are exactly proportional; the last one is 1 of 406, held to a 3px minimum so a real quantity does not render as nothing. Taken from the lane's own decision record, which states the count it chose from. The requirements do the excluding; the objective only orders what is left. No model name appears in the code that produced this.
Nothing here trusts a published number
Everything in this section is instrumented locally, because each published number failed in a specific way the moment it was trusted.
Availability
Published uptime answers whether an endpoint responds, not whether your requests succeed. Rate limits, 429s, context-length rejections and tool-format quirks all read as "up". One model in the catalogue reported 100 percent uptime while failing calls; I watched it do both at once. So the gate is the application's own error rate: a counter per model, a quarantine that backs off as failures repeat, and any success clearing the record.
The check that had never fired
The same quarantine mechanism runs in a sibling system, and auditing it started with one question: how many times has this guard actually fired? Zero. Not once, since the day it was written. One side of its comparison used catalogue ids and the other used dispatch ids, two namespaces that never overlap, so the guard compared names that could not match, excluded nothing, and every failing model kept serving.
A safety check that has never fired is worse than an absent one, because you believe it is protecting you.
A unit test of the comparison misses this, because each side is correct in isolation. You catch it by asking the running system for the count.
Speed
Every latency and throughput field was null on every endpoint checked, 16 of 16. Measuring passively from real turns is circular: an unmeasured model never wins a speed-sensitive objective, so it is never selected, so it is never measured.
Active probing breaks the circle: one small streamed completion per candidate, timed the way a real turn is timed and fed into the same moving average. The probe is stripped down on purpose:
- no workload content, because the probe reaches models the requirements have not vetted
- no tools attached, because a tool schema is thousands of prompt tokens, and first-token latency belongs to the model and the route
- sequential, never parallel, because parallel probes contend for one connection and would measure the harness instead of the model
-
Time to first token, from the active probe Fastest candidate0.83 sSecond candidate2.9 sThird candidate11.6 sSlowest bar shown at each candidate's range. The third was winning a lane on price alone: fourteen times the wait of the fastest, for a turn a person is sitting through. Nothing published would have told you, because every latency field the catalogue exposes was null.
- never on the turn path
- a call against a ceiling is refused before spending rather than after, and every skip is logged, because a silent cap reads as "everything is measured"
The decision record
Every routing decision is auditable. Per workload, the health payload carries the model chosen, its capability margin over the requirement, cost per turn, margin per dollar, estimated seconds, and which tie-break decided. Automatic selection without a visible decision record cannot be debugged and cannot be defended.
Vendor descriptions are not measurements
One catalogue entry advertises optimisation for tool use and coding agents. It measures 18.9 on the agentic index against a median of 20.8 across the 135 models that carry a score: below median on the exact axis its own description sells. Only the benchmark floor catches that.
The router is also a component under test
Everything above measures the models. None of it measures the thing choosing between them.
Those are two different objects. "Every model clears the bar its workload sets" is a claim about the workers. "The right work reaches the right lane" is a claim about the dispatcher, and a system can satisfy the first completely while failing the second on most turns.
One accuracy number cannot express it, because the two ways to be wrong do not cost the same. Sending easy work to an expensive lane wastes the difference between two tiers. Sending an authentication change to a cheap one ships it unreviewed.
So the dispatcher carries its own labelled set: 27 task briefs with the correct category and tier recorded, each label traced to a line of the routing policy rather than to an opinion. Scoring reports two numbers and never merges them. One is expected wasted spend, in dollars. The other counts violations against the categories the policy calls absolute, and it stays a count, because a rare catastrophic miss averages away to nothing.
Measured against the live classifier on 2026-08-12, not a mock: the model alone violated 4 of 15 absolute cases and put 70.4 percent of turns on the correct tier. The failures shared a shape. A payments webhook described as scaffolding, an auth-middleware change described as a refactor, a PII migration suite described as test-writing. Each one a premium-only category wearing the vocabulary of a cheap one.
The policy already stated those floors. It stated them in the prompt, in English, where a floor is one consideration a model weighs against everything else in front of it. Moving them into code, as rules applied after classification, took the same responses to 0 of 15 and 85.2 percent.
A floor a model can vote on is a suggestion.
Stated rather than plotted: a count and a percentage share no axis, and drawing them on one would invent a scale. Category accuracy is unchanged at 92.6% in both columns, which is the point. The floors correct tiers, never labels.
Two explanations died on the way there, each probed before anything was built on it. Casual phrasing looked like the cause and was not: stripping it changed nothing. A clarified prompt changed nothing either, and on the one remaining case four models, from the cheapest in the catalogue to the strongest, returned the same answer. That unanimity was the finding. My label was wrong, not the router. The example moved two variables at once and I read the result as evidence about the one I had in mind. Correcting a label after seeing the results is the failure this kind of work is most prone to, so the reasoning now sits in the dataset beside the label.
The harness also found four defects that reading the routing code did not: a broker unreachable from the worker process, a token ceiling spent on reasoning before any content was emitted, no retry on a malformed answer, and a route knowable without the model failing whenever the model did. Each one needed something that actually called the router.
The first of those was not one defect. Configuration read directly from a process environment is undefined in the worker that never loaded it and correct everywhere else, so the same mistake sat in seven places, wrong only in production and wrong by falling back rather than by failing. The guard that closed it asserts the rule instead of naming the seven, which is the difference between a test that prevents the eighth and one that passes forever.
The measurement runs weekly rather than on demand, and a run that finds a violation raises a decision on the operator's panel instead of writing a file nobody opens. Each run appends its numbers, so a regression arrives as a trend rather than as a surprise.
Cost is the completion, the cache and the threshold ladder
Ranking on advertised input price picks the wrong model. Completion is priced several times higher than input, so a model that is cheap to ask can be expensive to answer. Each workload therefore carries a representative turn shape and is costed on that shape. Three things sit underneath it.
The cache line is most of the input bill. A large stable prefix replays every turn at cache-read rates, so prompt caching is a partitioning problem. The system instruction splits into a shared invariant prefix plus context-specific blocks, and the invariant part goes first so it is one shared cache entry rather than one per variant. There is a hard ceiling of four cacheable segments. The design sits exactly at it, and that fact is written in the file so the next person does not learn it from an invoice.
Some rates go up, not down. 58 of the catalogue's 406 models raise their rates above a prompt-token threshold, and the ladder is filed under pricing despite charging more. One workload was mispriced by exactly 2x until rates were resolved at that workload's own prompt size.
Delivery time is part of the price. 60 catalogue entries are asynchronous batch endpoints: the same model at identical benchmark scores for half the price, delivered in hours. On any cost-ranked objective a batch endpoint wins permanently, and no quality floor catches it, because its scores are identical to the interactive sibling's. The router was already doing exactly that. The price was fine, but the arrival time was the defect.
What a turn actually costs
The same short lookup turn, with the same requirements satisfied, priced at two different objectives: roughly $0.047 at maximum capability against roughly $0.001 at soonest-per-dollar. About fifty times, for a turn where the cheaper model meets the requirement. That is the spread between objectives on one workload.
Measured cost per turn across the four workload shapes: about $0.374 and $0.324 for the two long-form document workloads, $0.0022 for interactive chat, $0.0009 for short lookups.
Linear scale, deliberately. The bottom two are invisible at this scale because they are: a lookup costs about one four-hundredth of a document turn. A log axis would make them look comparable, which is the opposite of the point.
A full catalogue sweep plus a timing run over the candidate set costs about three hundredths of a cent, $0.0003.
Measured time to first token, from the active probe:
- Fastest measured candidate. 658 to 833 ms to first token, at roughly 170 to 216 tokens per second.
- Second candidate. 2.0 to 2.9 seconds, at 53 to 80 tokens per second.
- Third candidate. 11.6 seconds, and it had been winning a lane on price alone.
That third one is unusable for an interactive lookup, and nothing published would have told you.
Retry steps sideways, never mid-stream
Retry moves to another model that already cleared the same requirements, and it prefers a different route to the same model over a different model, so a billing or provider problem never changes which model is writing. Nothing swaps mid-stream.
Degradation is ordered and explicit: relaxed requirements first, then a small pinned fallback list whose benchmark values are recorded verbatim with a capture date rather than guessed.
Errors are honest. Nothing substitutes a plausible neighbour for the thing that failed. That rule was paid for: one endpoint served a parent-scope document under a child-scope URL, a 200 with the wrong body, so users were handed the wrong document confidently and no log anywhere recorded a failure. I spent a while treating the complaint it caused as an architecture problem. It was one misrouted URL.
19 offline invariant tests run with no network and no credentials, so they pass or fail the same way on a cold clone.
Talk to me about it
If this is the kind of problem your team is working on, I am glad to go through any of it in more detail, including the parts that are still wrong. The application behind it is a tabletop tool. The routing, the observability and the cost work are the parts that transfer.
With Gratitude,
DGB