# Routing, observing and paying for LLM inference in production

*LLM platform engineering, as practised in a system that runs every week.*
Updated 2026-08-03 · Eudai Gestalt Integrations, LLC

---

This describes the inference layer of a system I built and operate: how a request
gets a model, how that model gets watched, and what a turn costs. The application
it serves is a co-dungeon-master for a live tabletop campaign. That matters here
only as a set of workload shapes, four of them: long-form document generation,
interactive chat, short lookups, and tool calls against two live databases. The
routing, observability and cost work would look the same behind a support desk or
a document pipeline.

## One process, two tool servers, a database per workspace

```
browser panel (static, no framework, no build step)
   |  SSE stream
one Node/TypeScript process
   |-- agent loop: capability-routed model, streaming,
   |     a cache-partitioned system prompt
   |-- MCP manager: two stdio tool servers
   |     |-- campaign server: the knowledge store, per workspace
   |     '-- rules server: the shared rules reference
   '-- SQLite, one database per workspace
          |
   LiteLLM broker --+-- aggregator: a live catalogue of 367 models
                    '-- first-party provider: a second route to the same model
```

Node and TypeScript, one process: an HTTP server, a streaming chat endpoint, and
an agent loop. The browser panel is HTML, CSS and vanilla JS with no framework and
no build step, and the two libraries it needs are vendored as single files. State
lives in SQLite, one database per workspace.

The two tool servers in the heading are Model Context Protocol servers over
stdio, their tools bridged into the model's tool list at connection time. One is
the campaign server: the knowledge store, one connection per workspace, with
typed entities, facts, relations and provenance on every assertion. The other is
the rules server, a single shared connection for the rules reference. The model
holds no state between turns. Continuity is a property of the database, so a turn
reads what is true at the moment the answer is written rather than what was true
when the conversation started.

Model access runs through a LiteLLM broker in front of an aggregator (OpenRouter)
and a first-party provider, so one SDK client reaches both and the application
holds no provider master key. The broker is also where the retry ladder finds a
second route to the same model.

The tool list is also the correctness boundary. Exposed tools narrow by scope: a
session at one scope is never handed another scope's write tools, so the wrong
write is unavailable rather than discouraged. An instruction is advice. A tool
list is enforcement. That removes a failure which reads like a model problem,
where a tool that cannot succeed at the current scope returns an error the model
then has to explain. A worse turn than never offering the tool.

## No model name appears in application code

A pinned model name is a stale constant the day something better ships. The
starting state here proved it: one model sat in source as "best" and was already a
generation stale on the day it shipped.

A request declares **requirements**, all machine-checkable:

- tool-calling support, because an agentic turn without tool calls is broken
  rather than cheap
- a context floor
- an output ceiling that will not truncate a long document, with a model that
  publishes no ceiling refused, since unknown capacity is not sufficient capacity
- quality floors read from published third-party benchmark indices

Those resolve at request time against the live catalogue, 367 models at the last
refresh. When something better ships it wins on the next refresh, with no code
change.

Separately, a request declares an **objective**. There are three, and they name no
models.

- **Most value.** The most capability headroom over the requirement, per dollar.
- **Soonest per dollar.** Minimise seconds times cost.
- **Most capability.** The most headroom over the requirement, cost breaking ties
  only.

The two declarations are orthogonal, and keeping them apart is what fixed the
early defects. Nearly every one of them was those axes collapsed into a single
control. Asking for maximum capability on a one-line lookup priced and filtered it
as a full document. Asking for the fastest option on a document routed it through
the lookup requirements, where a model that stops at four thousand tokens can be
handed a twelve-thousand-token document to write. Same bug, two faces.

## Availability is observed, speed is measured, decisions are recorded

Everything in this section is instrumented locally, because each published number
failed in a specific way the moment it was trusted.

### Availability

Published uptime answers whether an endpoint responds, not whether your requests
succeed. Rate limits, 429s, context-length rejections and tool-format quirks all
read as "up". I watched a model report 100 percent uptime while failing calls. So
the gate is the application's own error rate: a counter per model, a quarantine
that backs off as failures repeat, and any success clearing the record.

### The check that had never fired

The same quarantine mechanism runs in a sibling system, and auditing it started
with one question: how many times has this guard actually fired? Zero. Not once,
since the day it was written. One side of its comparison used catalogue ids and
the other used dispatch ids, two namespaces that never overlap, so the guard
compared names that could not match, excluded nothing, and every failing model
kept serving.

A safety check that has never fired is worse than an absent one, because you
believe it is protecting you. A unit test of the comparison misses this, because
each side is correct in isolation. You catch it by asking the running system for
the count.

### Speed

Every latency and throughput field was null on every endpoint checked, 16 of 16.
Measuring passively from real turns is circular: an unmeasured model never wins a
speed-sensitive objective, so it is never selected, so it is never measured.

Active probing breaks the circle: one small streamed completion per candidate,
timed the way a real turn is timed and fed into the same moving average. Every
design decision in the probe is a refusal:

- no workload content, because the probe reaches models the requirements have not
  vetted
- no tools attached, because a tool schema is thousands of prompt tokens, and
  first-token latency belongs to the model and the route
- sequential, never parallel, because parallel probes contend for one connection
  and would measure the harness instead of the model
- never on the turn path
- a call against a ceiling is refused *before* spending rather than after, and
  every skip is logged, because a silent cap reads as "everything is measured"

### The decision record

Every routing decision is auditable. Per workload the health payload carries the
model chosen, its capability margin over the requirement, cost per turn, margin
per dollar, estimated seconds, and which tie-break decided. Automatic selection
without a visible decision record cannot be debugged and cannot be defended.

### Vendor descriptions are not measurements

One catalogue entry advertises optimisation for tool use and coding agents. It
measures 18.9 on the agentic index against a median of 20.8 across the 127 models
that carry a score. Below median on the exact axis its own description sells. A
floor catches that. A description does not.

## Cost is the completion, the cache and the threshold ladder

Ranking on advertised input price picks the wrong model. Completion is priced
several times higher than input, so a model that is cheap to ask can be expensive
to answer. Each workload therefore carries a representative turn shape and is
costed on it.

Three things sit underneath that.

**The cache line is most of the input bill.** A large stable prefix replays every
turn at cache-read rates. Prompt caching is a partitioning problem: the system
instruction splits into a shared invariant prefix plus context-specific blocks,
and the invariant part goes first so it is one shared cache entry rather than one
per variant. There is a hard ceiling of four cacheable segments, the design now
sits exactly at it, and that fact is written in the file so the next person does
not learn it from an invoice.

**Some rates go up, not down.** 49 of the catalogue's 367 models raise their
rates above a prompt-token threshold, and the ladder is filed under pricing
despite charging more. It had one workload mispriced by exactly 2x until rates
were resolved at that workload's own prompt size.

**Delivery time is part of the price.** 28 catalogue entries are asynchronous
batch endpoints: the same model at identical benchmark scores for half the price,
delivered in hours. On any cost-ranked objective a batch endpoint wins
permanently, and no quality floor catches it, because the scores are identical to
the interactive sibling's. It was already doing that. Arrival time was the defect.
The price was fine.

## What a turn actually costs

The same short lookup turn, with the same requirements satisfied, priced at two
different objectives: roughly **$0.047** at maximum capability against roughly
**$0.001** at soonest-per-dollar. About fifty times, for a turn where the cheaper
model meets the requirement. That is the spread between objectives on one
workload. It is not a claim that anyone was previously paying the higher number
for everything.

Measured cost per turn across the four workload shapes: about **$0.374** and
**$0.324** for the two long-form document workloads, **$0.0022** for interactive
chat, **$0.0009** for short lookups.

A full catalogue sweep plus a timing run over the candidate set costs about three
hundredths of a cent, **$0.0003**.

Measured time to first token, from the active probe:

- **Fastest measured candidate.** 658 to 833 ms to first token, at roughly 170 to
  216 tokens per second.
- **Second candidate.** 2.0 to 2.9 seconds, at 53 to 80 tokens per second.
- **Third candidate.** 11.6 seconds, and it had been winning a lane on price
  alone.

That third one is unusable for an interactive lookup, and nothing published would
have told you.

## Retry steps sideways, never mid-stream

Retry moves to another model that already cleared the same requirements, and
prefers a different route to the same model over a different model, so a billing
or provider problem never changes which model is writing. Nothing swaps
mid-stream.

Degradation is ordered and explicit: relaxed requirements first, then a small
pinned fallback list whose benchmark values are recorded verbatim with a capture
date rather than guessed.

Errors are honest. Nothing substitutes a plausible neighbour for the thing that
failed. That rule was paid for: one endpoint served a parent-scope document under
a child-scope URL, a 200 with the wrong body, so users were handed the wrong
document confidently and no log anywhere recorded a failure. I spent a while
treating the complaint it caused as an architecture problem. It was one misrouted
URL.

19 offline invariant tests run with no network and no credentials, because a guard
that only runs when credentials are present is not a guard.

## Most of the progress was subtraction

I built roughly 400 lines of discount-detection machinery, gated it, tested it,
and then deleted every line after checking the one assumption underneath:
aggregator prices are already net of discounts. A discount is just a smaller
number, every cost-weighted objective picks it up on the next refresh, and it
reverts by itself when the discount ends, so there was nothing to detect and no
state to unwind. The finished router is smaller than the one it replaced, and if
your team inherits a layer like this, that is the property to look for: nothing
in it encodes an opinion the data can already express.

## What is not finished

Timing data covers eight models so far. The rest fall back to cost, which is a
weak proxy for speed and is labelled as one in the code.

One wrinkle is flagged in the code rather than fixed: capability margin is
measured from the requirement floor, which makes that floor a quiet tuning knob on
the value calculation. An absolute reference would be better.

No published benchmark measures prose quality. I checked rather than assumed: the
catalogue carries two benchmark namespaces and neither has a creative-writing
category. So a human preference survives as a final tie-break beneath every
measured axis, and it defaults to empty. It can separate two models the numbers
have called equal. It cannot promote a worse one.

## Talk to me about it

If this is the kind of problem your team is working on, I am glad to go through
any of it in more detail, including the parts that are still wrong. The
application behind it is a tabletop tool. The routing, the observability and the
cost work are the parts that transfer.

With Gratitude,
DGB