Running LLMs in production: monitoring, fallbacks, latency, and cost

Getting a prompt working is the first 20% of an LLM feature. The other 80% is operating it: keeping it up when a provider isn't, fast enough to feel good, cheap enough to be worth it, and observable enough to debug. Production LLM engineering is its own discipline, and here's the core of it.

Trace everything

Every node, prompt, tool call, token count, and latency should be captured as a trace (see the LangSmith and observability posts). When an LLM app misbehaves, the code ran fine — the only way to understand what happened is to see the actual steps. A black-box LLM app is an undebuggable one.

Fallbacks and circuit breakers

Providers have outages and rate limits — plan for it. Circuit-break to a fallback model or provider when the primary fails or slows, so one vendor's bad day isn't yours (a gateway like OpenRouter, or your own routing, makes this straightforward).

Latency and cost levers

• Model tiering — try a cheap, fast model first and escalate only when it fails a check.
• Semantic caching — serve semantically similar queries from cache instead of re-calling the model.
• Prompt caching — reuse the stable prefix so you don't reprocess the same tokens (see that post).
• Streaming — collapse perceived latency even when total time is unchanged.

The KPI that actually matters

Not cost per token, not even cost per request — cost per successful task. A cheap model that fails and triggers two retries and a human handoff is more expensive than the right model once. Optimise for the outcome delivered, and the token math takes care of itself.

Anyone can call an LLM. Running one in production means tracing it, tiering it, caching it, failing over it — and measuring cost per success, not per token.