In rough order of leverage on a typical production LLM workload in 2026:
1. Prompt caching (60–80% cost cut on agent/chat) · 2. Model routing (50–70% cut on mixed workloads) · 3. Batch API for async work (50% cut on offline pipelines) · 4. Output-shape engineering (20–40% cut by trimming tokens) · 5. Semantic caching (30–60% on repetitive workloads, use carefully) · 6. Speculative decoding / streaming (not cost, but TCO via latency) · 7. Fine-tuning a small model (high payoff only above ~100M tokens/month per task).
If you are running LLMs in production in 2026 and your monthly bill keeps surprising your CFO, you are not alone. Almost every team we've seen running real inference at scale has the same problem in the first 6–12 months: the bill grows faster than the user base, the FinOps slack channel gets noisy, and someone proposes "let's just switch to a cheaper model" as if that were a strategy.
It isn't a strategy. Cost optimization is a stack, and "use a cheaper model" is one lever out of seven. Pulled in isolation, it usually trades off measurable quality for marginal cost wins. Pulled in combination with the other six, it's the last lever you reach for, not the first. This guide is the practical, opinionated playbook for engineers actually running LLM inference in production — not a vendor pitch, not a theoretical survey, just the patterns we've seen work across teams in our directory of 118 AI and tech companies.
Why LLM Bills Get Out of Control
Before pulling levers, get clear on where the money actually goes. From cost breakdowns of teams we've worked with, the dominant failure modes are surprisingly consistent:
| Failure mode | Typical share of waste |
| Frontier model for every request | 30–50% — 70% of requests don't need it |
| No prompt-cache hits | 20–40% — long system prompts re-billed in full each call |
| Bloated system prompts & few-shots | 10–25% — "let me also include 8 examples just in case" |
| Unbounded retries on failure | 5–15% — quiet but real, especially with agents |
| Real-time API for async workloads | 10–30% — batch would have been 50% cheaper |
| No output-token discipline | 10–20% — "respond with explanation" instead of structured output |
The point: before optimizing, instrument. You cannot cut what you can't see. Every production LLM workload needs per-request cost telemetry (input tokens, output tokens, cached tokens, model used, route taken, retry count) attributed by user, feature, and route. Most cost surprises are also instrumentation surprises — you find out the chatbot has been falling back to the frontier model for every retry only because somebody finally looked.
The Seven Levers, In Priority Order
Prompt caching for stable prefixes
Prompt caching lets the model reuse its internal processing of a stable prefix — system prompt, tool definitions, few-shot examples, retrieved context that doesn't change — across multiple requests. Cached tokens are billed at roughly 10% of standard input rate on Anthropic and similar discounts on other providers. The first-token latency drops 2–4x as a free bonus.
The practical move: identify the stable prefix in your prompts and mark it for caching. For chat apps, this is usually the system prompt plus tool definitions. For agents, it's the system prompt plus the stable scaffolding. For RAG, it's the system prompt — retrieved context is per-request, so it's the bit after the cache marker.
Model routing by request difficulty
The biggest waste in most production LLM systems is using a frontier model for every request when 70–80% of requests are routine. The fix is a router: a small, fast classifier (or a small LLM) that triages requests into "easy" and "hard" buckets, then routes each to the appropriate model tier.
The router itself can be embedding-based (cosine similarity to known easy/hard examples) or LLM-based (a fast Haiku-tier call that returns a difficulty label). The economics work because the router cost is 1–3% of the inference it replaces.
Build the eval first. Sample 200–500 real requests, label them by "did the small model produce a response indistinguishable from the large model?", and use that as your routing oracle. Anything below 95% quality on the routed-to-small bucket means the router is too aggressive — tighten it.
Batch APIs for anything non-realtime
Major providers offer batch APIs at ~50% discount on input and output tokens with a 24-hour completion SLA. If a workload doesn't need sub-minute latency, it should be on the batch API. Common candidates: nightly enrichment, bulk classification, evaluation runs, document processing, content moderation backlogs, dataset generation for fine-tuning.
The reason teams don't use batch APIs more is organizational, not technical. Real-time and batch are different code paths; engineers default to real-time because it's already wired up. Spending a week to route the obvious offline workloads to batch is a 50% saving on that workload type for as long as the workload exists. There are very few engineering investments with that ROI.
Output-shape engineering (cut output tokens)
Output tokens cost 3–5x more than input tokens at most providers. Most production prompts treat output length as an afterthought — "respond in JSON" with no schema constraint, or "explain your reasoning" when the user only needs the answer.
Concrete moves: use strict structured output (JSON schema or tool-use), give explicit max-token caps based on what the downstream consumer actually parses, drop "think step by step" from prompts where the steps aren't used downstream, and switch reasoning-style chains-of-thought to summary-only outputs for non-debug paths.
For one team we worked with, switching their classification pipeline from "respond with reasoning and category" to "respond with just the category code" cut output tokens by 90% and dropped overall cost on that path by ~25%. Same quality.
Semantic caching for repetitive workloads
Semantic caching stores full request-response pairs and returns a cached response when a new request is semantically similar (typically via embedding similarity over the prompt or query). Different from prompt caching: semantic cache skips the model call entirely.
Where it works: FAQ-style chatbots, support agents, search re-ranking, anything where the same question gets asked many ways and the correct answer doesn't depend on user context. Hit rates of 30–60% are realistic on those workloads.
Where it breaks: any workload where similar prompts can have different correct answers (personalized recommendations, anything user-context-dependent, anything time-sensitive). Set the similarity threshold high (cosine ≥ 0.95) and treat the cache as a probabilistic optimization with a fallback path — never as a source of truth.
Streaming + speculative decoding for latency-cost
Streaming the response doesn't cut token cost, but it dramatically improves perceived latency, which lets you stick with a smaller model that would otherwise feel slow. Speculative decoding (where a draft model proposes tokens and a target model verifies them) is increasingly available from providers and cuts wall-clock time at no quality cost.
These don't show up in your bill directly, but they expand the design space — you can route more workloads to smaller models without users noticing the wall-clock difference. The cost win shows up in Lever 2's column.
Fine-tuning a small model for a narrow task
Fine-tuning a small open-weight or hosted small model on your specific task can match frontier quality at 10–30x lower cost per request. The catch: you only get there after exhausting prompt engineering, model routing, and caching — AND you need enough volume on a single task to amortize the training cost and ops burden.
Rough break-even in 2026: ~100M tokens/month on a single workload type. Below that, prompt engineering + routing + caching wins on TCO. Above it, fine-tuning starts to pay back within 2–3 months. The hidden cost is operational: now you own model versioning, eval drift, prompt-template versioning, and rollback playbooks.
The AI Gateway Pattern (Why You Want One)
By the time you've implemented two or three of these levers, you'll notice the patterns are bleeding across every service that touches the LLM API. Caching code in three places. Routing logic forked twice. Eval coverage uneven. This is the prompt for centralizing on an AI gateway.
An AI gateway is a thin routing layer between your application and the LLM providers. It owns: model selection routing, prompt caching configuration, semantic cache, retries with backoff, fallback chains, per-route observability (latency, cost, quality), and provider abstraction. Application code calls the gateway, which calls the right provider with the right model and cache config.
The cost-relevant win: changes happen in one place. Swap providers, A/B test model routing, roll out a new caching strategy — all without touching application code. Most teams running >$10k/month of LLM spend in 2026 have some form of gateway, either built in-house or via products like the Vercel AI Gateway or similar.
The Evals Question You Can't Skip
Every lever above is gated on the same precondition: can you measure quality before and after?
Cost optimization without an eval is just degradation in slow motion. The team that ships aggressive model routing without an eval will discover six weeks later that their support chatbot is now giving subtly wrong answers, with no way to attribute the change. The team with a 200-example golden set can roll out routing, watch the metric for a week, and confidently keep or revert.
You don't need a sophisticated eval framework. The minimum viable version: 200–500 labeled examples from production, a script that runs them against any prompt/model combination, and a simple metric (exact match, BLEU, LLM-as-judge with a tight rubric). Add to it monthly. For deeper guidance see our LLM evaluation guide and AI agent evaluation guide.
What to Stop Doing
Stop — Picking the cheapest model first
It's the lever with the worst quality-to-savings ratio when used in isolation. Apply caching, routing, and output-shape discipline first. Only then ask: "given the routing rules, what's the right model for each tier?"
Stop — Treating LLM cost as a single number
"We spent $X on LLMs this month" is the wrong unit of analysis. Attribute by route, feature, and user. Most optimization wins come from finding the 3 routes that are 80% of the bill, not from trimming every route by 10%.
Stop — Unbounded agent loops
Agents that retry, re-plan, and re-call tools without a hard step ceiling will silently eat 10x their expected budget when something goes wrong. Every agent path needs a max-step limit, a max-token budget, and an alarm when either fires.
Stop — Engineering for a future model
"This will be cheaper when GPT-7 ships" is not a cost plan. Optimize for the model you're paying for today. Providers consistently ship better models — the savings will come anyway, on top of your work, not instead of it.
The 30-Day Cost Reduction Sprint
If you want to act on this in the next month, here's a sequenced plan.
Week 1 — Instrument
- Add per-request cost telemetry: input tokens, cached tokens, output tokens, model, route. Tag by feature.
- Build a "top 10 routes by cost" dashboard. The 80/20 is almost always present.
Week 2 — Cache and trim
- Enable prompt caching on every route with a stable prefix > 1k tokens.
- Audit each top-10 route for output-shape bloat. Trim with strict structured outputs and explicit max-token caps.
- Move any non-realtime workload to the batch API.
Week 3 — Eval and route
- Build a 200–500 example eval set for the top-cost route.
- Test a smaller model on that eval. If it hits ≥95% of frontier quality, route to it.
- Add a difficulty router for the route — small model for routine requests, frontier for hard ones.
Week 4 — Stabilize
- Set up regression alerts: cost-per-request, quality-on-eval, cache hit rate.
- Document the routing logic for each route in one place.
- Decide which workloads are big enough to justify fine-tuning over the next quarter.
Done well, a 30-day sprint cuts most production LLM bills 50–75% without measurable quality regression. The compounding part is that the infrastructure you build — eval set, gateway, telemetry — pays dividends on every future change to the system, not just this one.
FAQ
Looking for AI & ML engineering roles?
Browse jobs at companies building production LLM systems — with verified culture profiles, comp data, and engineering blogs to evaluate before applying.
Browse AI/ML Jobs → Explore AI Tools →