LLM Cost Optimization: Caching, Batch APIs and Model Routing

One million input tokens can cost you $3 or 15 cents. Same model, same question, same answer. The 20x gap comes from how you send the request: whether you make the model re-read the same text every time, whether you need the answer now or overnight, and whether every question goes to the most expensive model on the menu. Most of what gets called LLM cost optimization is the answer to those three questions.

The pattern we see in the field: when an API bill balloons, most teams start shopping for a cheaper model first. Yet when you break the bill into its line items, three or four architectural patterns show up that cut it by more than 70 percent without touching the model at all. This piece walks through those patterns (prompt caching, batch processing, model routing and semantic caching) with the numbers attached, and explains why streaming is not on the list. Our non-technical leader's guide to AI infrastructure decisions covered the "API or your own server" question; here we assume you picked the API and look at what that bill is made of.

A caveat up front: the prices below were compiled in September 2026 from secondary sources, and providers change their price lists often. The ratios in this piece (50 percent for batch, roughly 90 percent for cache reads) are solid as orders of magnitude; check the official pricing page before you sign anything.

Where does an API bill actually come from?

An LLM API bill has three line items: input tokens (everything you send to the model), output tokens (everything the model writes back) and request count. Output tokens cost 3 to 8 times more than input tokens depending on the model, most commonly 4 to 5 times. What inflates the bill, though, is usually repetition on the input side: the same system instructions, the same product catalog, the same examples sent again with every single request.

Make it concrete. A customer support bot carries a 6,000-token system prompt: return policy, shipping rules, tone guidelines, ten example dialogues. The customer's question is 200 tokens, the answer 250. About 95 percent of each message's input cost goes to those 6,000 tokens the customer never sees. At 60,000 messages a month, that is 360 million tokens re-read every month.

So the first question should not be "which model is cheaper". The first question is: how much of each request is repeated, and why are we paying for it every time?

Does prompt caching really cut the bill by 90 percent?

Prompt caching means the provider stores the unchanging opening section of your request on its side and reads it from memory on subsequent requests instead of processing it again. Cached tokens are billed at 10 to 25 percent of the normal input price across the three major providers, a discount of 75 to 90 percent. But the discount applies only to the repeated input portion; output tokens and the part that changes with every request are billed at full price.

The mechanics in brief: the provider recognizes the section of the request that is byte-for-byte identical from the start. On Anthropic you mark that section explicitly, on OpenAI recognition is automatic, on Google you create an explicit cache and reference it. The rule is the same on all three: unchanging content first, changing content last. Put today's date or the customer's name in the middle of your system prompt and everything after that point falls out of the cache.

Three details decide the bill:

• Write cost. On Anthropic, the first write to the cache costs 1.25x the normal price (5-minute lifetime) or 2x (1-hour lifetime). The cache pays for itself once the same content is read two or three times; a one-off request loses money. OpenAI charges no separate write fee; Google charges hourly storage for explicit caches.
• Lifetime. The default lifetime is measured in minutes (5 on Anthropic, refreshed on every read). A system that gets one request an hour gains nothing from the cache; a system that gets one a minute keeps it warm all day.
• Minimum size. Short prompts are not cached at all; there is a floor of roughly a thousand tokens that varies by model. A 300-token system prompt has no cache to speak of.

Back to the support bot: once the 6,000-token prompt is read from cache, most of the input cost drops to a tenth. The total cost per message falls by about 70 percent, because output tokens stay where they were. The "90 percent savings" headlines on vendor blogs look at the input line, not the whole invoice.

Batch APIs: anything that can wait 24 hours costs half

A batch API lets you submit requests in a single file and collect the results within 24 hours; in return, OpenAI, Anthropic and Google apply a 50 percent discount to both input and output tokens. Any job that does not need an immediate answer qualifies: overnight reports, product description generation, archive classification, monthly sentiment analysis.

The real strength of batch is that it stacks with caching. In Anthropic's pricing the multipliers chain: a $3 input token becomes $1.50 with batch, then 15 cents with a cache read on top. That is where the 20x gap at the top of this piece comes from. Not every token reaches that ratio in practice (the changing portion and the output tokens are billed at the plain batch price), but for repetitive, patient workloads an effective discount above 80 percent is a reasonable expectation.

The mistake we see most often: teams dismiss batch as "slow". Yet to have 5,000 product descriptions ready at 8 a.m., submitting them at 2 a.m. is enough. The 24-hour window is a ceiling; most jobs come back within hours. The jobs that do not fit batch are easy to spot: a user is waiting at a screen, or the result decides the very next step in real time.

Does streaming reduce cost?

No. Streaming delivers the answer token by token; it changes neither the total token count nor the bill. What it changes is perceived waiting time: the user sees the first word within a second instead of staring at a blank screen for 20. The only place streaming touches cost is indirect: cancelled requests and control over output length.

The indirect effect comes through two channels. First, if a user closes the tab halfway through or says "that's enough", you can cut the stream; tokens that were never generated are never billed. Second, because you measure output length as it streams, you can see which questions get needlessly long answers and set a ceiling with max_tokens. Since output tokens cost 4 to 5 times more than input, bringing the average answer down from 400 to 250 tokens often saves more than switching models would.

That is why content that lists streaming under "cost reduction" is misleading. The accurate classification: streaming is an experience tool, max_tokens is a cost tool, and the two work well together.

Semantic caching: the 95 percent hit-rate myth

A semantic cache returns the answer previously given to a similar question without calling the model at all. Unlike prompt caching, it does not look for identical text; it looks for questions that are "close in meaning". The 90 to 95 percent hit rates on vendor pages do not show up in production data: in real traffic, 15 to 30 percent of requests are repeats, and the false-hit risk (an old answer served to a different question) can rule this pattern out entirely in regulated industries.

Let the numbers speak. In a 2024 academic evaluation (MeanCache, arXiv), the widely used GPTCache tool produced 54 false hits on a contextual query set; the method it was compared against produced 3. In another test, on a query set with no repeats at all, every hit GPTCache found was wrong: it returned another question's answer because the similarity score was high. "How many days do I have to return an item?" and "How much does return shipping cost?" are close as text; their answers are not.

An independent analysis of production data puts the acceptable false-hit rate at around 2 percent in unregulated sectors and around 0.5 percent in finance and healthcare. That rests on a single source, but the conclusion that near-zero thresholds take semantic caching off the table for most serious work is robust.

There is a narrow zone where semantic caching makes sense: FAQ-type questions whose answers never change and where a wrong answer is cheap. Even there you keep the similarity threshold high and audit hits by sampling. The claim of 30 to 50 percent savings on chat traffic comes from a single vendor source; in our own projects we plan for 10 to 15 percent at most.

Model routing: not every question deserves the big model

Model routing means sending every request to a small, cheap model first and escalating only the hard or risky ones to the large model. The RouteLLM research reported an 85 percent cost reduction while preserving 95 percent of GPT-4 quality on benchmark tests. That is a benchmark result; in the field, savings range from 45 to 85 percent depending on the workload. The critical precondition is that you have defined what a "hard question" is.

There are three common implementations. The simplest is rule-based: "where is my package" and "return window" questions go to the small model, contract interpretation and complaints to the large one. The second is a cascade: the small model answers, and if its own confidence score falls below a threshold the same request goes to the large model; you pay twice, but rarely. The third is a learned router: a classifier trained on historical data about which questions the small model handled well.

The hidden price of routing is the test set. If you do not know which questions the small model handles adequately, 85 percent savings can turn into 85 percent quality loss. That takes a labeled question set and regular measurement. In multi-agent setups the stakes rise further: we have written about how multi-agent systems burn 15 times the tokens, and running most of those agents on a small model trims that multiplier substantially.

A worked example: 60,000 support messages a month

A 40-person online retailer runs a support bot that handles 60,000 messages a month: a fixed 6,000-token prompt, 200-token question, 250-token answer. On a Sonnet-class model ($3 input, $15 output per million tokens) the raw bill is about $1,340 a month. Applying caching, routing and then batch for overnight jobs, in that order, brings the same service down to around $200. The figures are a rounded, simplified version of a system we built; the ratios matter more than the exact dollars.

Step by step:

• Baseline: input 372 million tokens × $3 = $1,116; output 15 million tokens × $15 = $225. Total $1,341.
• Prompt caching: the 6,000-token prompt is read from cache at $0.30: 360 million × $0.30 = $108. The changing 200 tokens at full price: $36. Traffic is continuous, so the cache stays warm; write costs stay under $10 a month. Output unchanged at $225. Total about $380, a 72 percent drop.
• Routing: 70 percent of messages (shipping, returns, order status) go to a small model (roughly $1 input, $5 output; cache reads at $0.10). In that slice, output cost falls from $157 to $52 and the input line drops to about a third. Total about $210.
• Batch: the overnight jobs (topic-tagging all 60,000 messages and the weekly summary report) move to batch; this line was already small and drops from $40 to $20. New total around $190.

There is no semantic cache in this table: even if 20 percent of the retailer's questions were repeats, they were not willing to risk a wrong answer on topics like return policy. An 85 percent reduction without changing the model and while keeping quality measurement intact, and the largest slice came from the first step, about three days of engineering.

Frequently asked questions

Are prompt caching and semantic caching the same thing?

No. Prompt caching happens on the provider's side and stores the identical opening section of the request; the answer is still generated by the model, only the read gets cheaper. Semantic caching runs on your side and returns an old answer to a similar question without the model running at all. The first is safe and almost always worth doing; the second is risky and suited to a narrow zone.

Do batch and cache discounts stack?

On Anthropic, yes, the multipliers chain (0.5 × 0.1). On OpenAI, cached-input pricing appears to apply inside batch for the newer model generations; on Google it is also described as multiplicative, but the cache discount itself varies between 75 and 90 percent across sources. Read the "batch" and "cached input" rows together on your provider's pricing page.

How much quality do I lose by moving to a small model?

It depends on the question; there is no general percentage. On narrow tasks such as classification, summarization and information extraction, small models frequently match the large one; on multi-step reasoning and long-document interpretation the gap opens. The only reliable method is to measure both models side by side on a labeled set of 100 to 200 of your own questions.

How much code change do these patterns require?

For prompt caching, reordering the prompt and (on Anthropic) adding a marker is enough; usually a day's work. Batch needs a separate submit-and-collect flow; a few days. Routing, together with its test set and threshold design, is a one- to two-week project. That is also the order to apply them in: cache first, then batch, then routing.

If the bill drops 80 percent, do I still need to think about self-hosting?

For most small and mid-sized companies, no. Your own server enters the picture when the monthly API bill runs into the thousands of dollars and there is a data-sovereignty reason; we worked through when self-hosting an LLM genuinely pays off in a separate piece. The patterns here push that threshold even higher.

So what should you actually do?

• Split the bill into three lines. Repeated input, changing input, output. Whichever line is above 50 percent, that is where the first pattern goes.
• Turn on prompt caching this week. Fixed content first, variables last; do not expect anything from prompts under a thousand tokens.
• Make a "does this need to be now?" list. Every job where nobody is waiting at a screen goes to batch; in a typical company that is about a third of the volume.
• Set max_tokens, keep streaming for the experience. Measure your average answer length and put a ceiling on it.
• Build routing together with its test set. No labeled question set, no routing; try semantic caching only where a wrong answer is cheap.

In every project we have seen so far, the most expensive fix was changing the model and the cheapest was changing the order of the request; we have yet to meet a case where that ranking flipped. The spreadsheet we use to split a bill into its three lines goes out to any team that asks for it. An email is enough.