How we cut the token bill 40%: prompt caching and structured output

Agent systems burn tokens structurally: every step re-sends context, every retry re-pays it. We cut our bill roughly 40% with two unglamorous techniques, prompt caching and structured output, and the interesting part is that both made the system faster and more reliable, not just cheaper. Cost discipline in LLM systems is usually quality discipline wearing an accountant's hat.

Prompt caching rewards architectural tidiness. Caches key on stable prefixes, so the system prompt, tool definitions, and reference material must be byte-identical across calls, with volatile content (the current document, the current state) appended after the cache boundary. That forced us to separate what's constant from what varies, which is just good design that billing happened to enforce. Hit the cache and you skip re-processing the prefix: cheaper and lower-latency.

Structured output cuts spend from the other end. Free-form generation pays for prose, hedging, and restatement; constrained generation pays for fields. Forcing tool-call/JSON-schema output shrank responses, killed a whole class of parsing failures, and, the quiet win, killed the retries those parsing failures caused. A failed parse costs the whole call plus the retry; schema enforcement is retry prevention, which is the cheapest token you'll ever buy.

Meta-lesson: measure cost per completed unit of work, not per call. Ours dropped more than 40% because fewer calls failed. A cheap call that has to be made twice is an expensive call.