# Prompt caching, batching and model routing: what they actually save

> Cached input costs 2.5% to 10% of the input price, yet on an agent call that is about 55% off the bill at a 90% hit rate, and a loss below a fifth. Batch is a flat half. Routing has the biggest claims and the weakest evidence.

- URL: https://bardak.ai/posts/prompt-caching-batching-routing
- Author: Ivan Bardak (https://bardak.ai/pages/about)
- Published: 2026-11-19

An agent bill grows with use, and sooner or later someone asks what can be done about it. There are three standard levers: prompt caching, batch processing and routing some of the work to a cheaper model. Vendors describe each with its best number. This post works out what each saves on an agent workload, what it costs in latency, quality or engineering time, and in which order to pull them.

**The short answer:** cache first, batch what can wait, route last. Reading from the cache is billed at 2.5% to 10% of the input price, but on a typical agent call that turns into about 55% off the bill at a 90% hit rate and never more than about 63%, because output is not discounted; below a hit rate of about a fifth, caching with a write surcharge costs more than no caching. Batch is a flat half at all three big providers, for work that can wait up to a day. Routing has the biggest numbers on paper and the weakest evidence: the largest independent benchmark puts the best routers at up to 32% cheaper at equal accuracy and several routers at no gain, and in the one study that ran routers at every step of a coding agent the result ranged from 53% cheaper to three times dearer. It is also the only lever that can lower quality.

As of 16 November 2026: prices and cache rules change monthly, and the current ones are in [LLM API prices for agents, tracked](https://bardak.ai/posts/llm-api-prices).

## Why agents repeat themselves

An agent calls the model many times per task, and every call re-sends the instructions, the tool definitions and the steps so far. In Anthropic's example of a five-server MCP setup, 58 tools take about 55,000 tokens of definitions before the conversation starts ([Anthropic](https://www.anthropic.com/engineering/advanced-tool-use)). That prefix is identical from call to call, which is exactly what a cache is for. How to keep it short in the first place is in [MCP tool design](https://bardak.ai/posts/mcp-tool-design).

With prompt caching the provider keeps the processed start of your prompt for a while and bills much less when the next request starts with the same tokens. The rules differ:

**Caching rules by provider**

| Provider | Reading from cache | Writing to cache | Smallest prompt | Lifetime | On by default | With batch |
| --- | --- | --- | --- | --- | --- | --- |
| Anthropic | A tenth of the input price; a twentieth on Opus 5.5 and Sonnet 5.5; a fortieth on Fable and Mythos 5.1 and 5.2 | 1.25 times the input price for 5 minutes, twice for 1 hour | 512 tokens on the newest models; 1,024 to 4,096 on older ones | 5 minutes or 1 hour, renewed by each read | No: one cache_control field turns it on | Discounts stack; hits are best effort |
| OpenAI (GPT-5.6 and later) | A tenth of the input price; a twentieth on GPT-6.1 Sol and Terra | 1.25 times the input price for 30 minutes, 2.5 times for 12 hours | 1,024 tokens | At least 30 minutes, renewed by a read, or 12 hours | Yes | Price list shows cached input at half; the batch guide does not mention caching |
| Google (Gemini 4) | A twentieth of the input price (a tenth on Gemini 3 models) | No charge; explicit caches on Gemini 3 models pay storage by the hour | 4,096 tokens | Not stated | Yes | Price list shows cached input at half; the batch guide is silent on implicit caching |

Read fractions are of the standard input price. Older OpenAI models (GPT-5.5 and earlier) have no write charge. Gemini 4 models cache only implicitly, and Google calls that a discount with no guarantee. Source: Anthropic, OpenAI and Google caching, batch and pricing documentation, read 16 November 2026. Data: https://bardak.ai/data/prompt-caching-batching-routing/caching-rules-by-provider.csv

## What caching saves on an agent call

Take the call from [What an AI agent really costs](https://bardak.ai/posts/ai-agent-cost-per-task): 10,000 input tokens and 1,000 output tokens, on a model priced at $2 and $10 per million tokens, with cache reads at a twentieth of the input price and cache writes at 1.25 times. Those are the prices of Claude Sonnet 5.5 and GPT-6.1 Sol today ([Anthropic](https://platform.claude.com/docs/en/about-claude/pricing); [OpenAI](https://developers.openai.com/api/docs/pricing)). The cost of 1,000 such calls, my arithmetic:

**Cost of 1,000 agent calls under each lever**

| Setup | Cost |
| --- | --- |
| Caching on, nothing reused | 35.00 |
| No discounts | 30.00 |
| Caching, 50% of input read from cache | 23.00 |
| Half the calls on a model at a twentieth of the price | 15.75 |
| Batch | 15.00 |
| Caching, 90% of input read from cache | 13.40 |
| Batch and caching at 90% | 6.70 |

Each call: 10,000 input and 1,000 output tokens at $2 and $10 per million. Cache reads at $0.10 per million; all input that is not read from the cache is written to it at $2.50, the worst case. The routing bar assumes the cheaper model needs the same number of calls and succeeds as often, which is the thing to test. Source: My arithmetic from Anthropic and OpenAI list prices, read 16 November 2026. Data: https://bardak.ai/data/prompt-caching-batching-routing/cost-of-1-000-agent-calls-under-each-lever.csv

Three things follow from the chart.

- **The discount on the price page is not the discount on the bill.** Output is a third of this bill and caching does not touch it. Even if every input token came from the cache, the bill would fall by 63%, not 95%. At a tenth of the input price the ceiling is 60%.
- **The hit rate decides the result.** At 90% the bill falls by 55%, at 50% by 23%. With a write surcharge of a quarter on everything that misses, caching breaks even at a hit rate of about 21% and loses money below it. OpenAI now charges for writes on its current models and caches by default, so a prompt that never repeats pays the surcharge without anyone having decided to.
- **Nobody publishes a typical hit rate.** OpenAI's guide gives two examples it calls possible outcomes: about 70% for a single-turn judge and over 90% for a multi-turn agent ([OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching)). Anthropic says only that batch users see "cache hit rates ranging from 30 percent to 98 percent" ([Anthropic](https://platform.claude.com/docs/en/build-with-claude/batch-processing)). Google gives no figure. Your own number is in the usage data of every response, so measure it before you forecast a saving.

## What breaks the cache

A hit needs the start of the prompt to match exactly, so the hit rate is a property of how the prompt is built. The documentation of Anthropic and OpenAI lists the usual ways to lose it ([Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching); [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching)):

- **Something that changes sits near the start.** A timestamp or a user name in the system prompt makes every request a miss. Put stable content first and variable content last.
- **The tool list changes.** At Anthropic a changed tool definition invalidates the whole cache, and OpenAI counts tool order as part of the prefix. At OpenAI, restrict tools with the tool-choice settings instead of removing them; at Anthropic a changed tool choice keeps the cached tools and system prompt but not the cached messages.
- **History is rewritten.** Summarising, compacting or truncating earlier turns resets reuse from that point. Append, do not edit.
- **Settings change between calls.** The model, the reasoning effort and the output format are part of what is matched.
- **The gap is longer than the lifetime.** Anthropic's default cache lives 5 minutes, counted from the start of the request, so a step that waits on a slow tool or a human approval comes back to a cold cache. The 1-hour cache costs twice the input price to write.

For implicit caching Google gives two tips, common content first and similar requests close together, but no list of what breaks a cache and no lifetime ([Google](https://ai.google.dev/gemini-api/docs/caching)).

## Batch: a flat half, for work nobody is waiting for

All three providers take 50% off input and output for requests submitted as a batch. Anthropic says most batches finish in less than an hour and expire after 24 hours; OpenAI completes within 24 hours; Google targets 24 hours and expires a job at 48 ([Anthropic](https://platform.claude.com/docs/en/build-with-claude/batch-processing); [OpenAI](https://developers.openai.com/api/docs/guides/batch); [Google](https://ai.google.dev/gemini-api/docs/batch-api)). No model change and no quality risk: the price is waiting.

That rules out an interactive agent, and in practice any agent loop, since each of 20 steps would wait in the queue. It fits what surrounds agents: evaluation runs, document processing, extraction, re-indexing (the indexing bill in [The real cost of GraphRAG](https://bardak.ai/posts/graphrag-cost) halves this way). OpenAI and Google also sell a Flex tier at batch prices for ordinary requests that may be slow or refused, which suits background agents ([OpenAI](https://developers.openai.com/api/docs/guides/flex-processing); [Google](https://ai.google.dev/gemini-api/docs/flex-inference)).

Batch and caching stack at Anthropic, which says so and warns that hits inside a batch are best effort because requests run in any order. OpenAI's and Google's price lists show cached input at half price in batches, but their batch guides do not promise it: OpenAI's does not mention caching, and Google's speaks only of explicit caches, at standard cache rates. Budget for the batch half and treat cache hits inside batches as a bonus.

## Routing: the biggest numbers, the weakest evidence

Routing sends each request to the cheapest model expected to handle it; a cascade tries the cheap model first and escalates. The numbers people quote come from two papers. RouteLLM needed 13.4% of calls to go to GPT-4 to reach 95% of its score on MT Bench ([Ong et al.](https://arxiv.org/abs/2406.18665)). FrugalGPT cut cost by 59% to 98% at the accuracy of the best single model ([Chen et al.](https://arxiv.org/abs/2305.05176)). Both are narrower than they sound. RouteLLM's result is on 160 chat questions scored by an LLM judge, and its 3.66 times saving is against a random router, not against sending everything to GPT-4; trained on Chatbot Arena data alone, its routers did no better than random on two other benchmarks. FrugalGPT's 98% is the best of three datasets, a headline classification task, with 2023 models.

Independent benchmarks are more sober. RouterBench found that simple routers did not significantly beat a trivial baseline ([Hu et al.](https://arxiv.org/abs/2403.12031); most of its authors work at a router vendor). LLMRouterBench, with 33 models and 21 datasets, found the best routers up to 31.7% cheaper at the accuracy of the best single model, several routers that failed to beat that model, and a commercial auto-router, on its own pool of models, 24.7% below it ([Li et al.](https://arxiv.org/abs/2601.07206)). RouterArena ranked the built-in router of GPT-5 seventh of twelve ([Lu et al.](https://arxiv.org/abs/2510.00202)). All of these pick one model for a whole question or task; none routes the steps inside an agent.

For routing inside an agent I found two measurements with a cost figure, both from parties with a product. TwinRouterBench ran routers at every step of a coding agent on 100 SWE-bench Verified cases: one model alone resolved 74 for $54.73; the authors' trained router resolved 75 for $25.66, which is 53% less; another router saved nothing; an untrained rule-based one cost $172.56, about three times the cost of not routing ([Yang et al.](https://arxiv.org/abs/2605.18859)). It is one run on cases that were not a random sample, and the winning router belongs to the organisation that hosts the benchmark. LangChain's test of NVIDIA's router on 145 multi-step tasks came out 74% cheaper than the frontier model alone and 6 points less accurate, the judge that decides when to escalate took about a fifth of the routed bill, and the authors say the routed result was within noise of the cheap model alone ([LangChain](https://langchain.com/blog/switchyard-agent-routing-benchmark)).

Two more costs belong in the estimate. A cheaper model can need more calls and fail more often: in the SWE-bench data in [What an AI agent really costs](https://bardak.ai/posts/ai-agent-cost-per-task), the cheaper model made about 89 calls per issue against about 20 and cost five times as much per resolved issue. And routing fights caching: the model is part of what a cache matches, so after a switch in the middle of a task the new model has no cache for the history and processes all of it again, at the write rate where writes are charged ([OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching)). TwinRouterBench charges a cache write on every switch for that reason. Routing whole tasks, not steps, avoids it.

## The levers together

On the example task (20 calls, a 60% success rate) the cost per completed task falls from $1.00 to $0.45 with 90% of the input read from the cache and the rest written to it. The calculator below starts from the same task; it does not add the write surcharge, so it shows $0.43. Put in your own calls, tokens, prices and hit rate:

_An interactive agent cost calculator stands here on the web page at https://bardak.ai/posts/prompt-caching-batching-routing._

Whether the result beats a person doing the same work is a separate sum, in Agent or person.

## When not to use them

- **Caching:** prompts below the minimum length (512 to 4,096 tokens, depending on the model), prompts that rarely repeat within the lifetime, and prefixes that change on every request. With write charges, these cost more with caching than without.
- **Batch:** anything a person or another system waits for, and multi-step loops.
- **Routing:** when you have no evaluation set of your own tasks, when one weak step can sink the whole task, or when the saving would be smaller than the cache you lose by switching models.

## The rule: cache first, batch what can wait, route last

1. **Measure.** Log input, cached input, cache writes and output tokens per call, and whether the task succeeded. Hit rate is cached tokens divided by all input tokens; at Anthropic that total is cache reads plus cache writes plus input tokens, because its input field counts only the uncached part.
2. **Fix the prefix.** Tools and instructions first and unchanged, variable content last, history appended. Re-measure until the hit rate stops rising.
3. **Match the lifetime to the gaps.** If steps often wait longer than the default lifetime, pay for the longer one.
4. **Move everything nobody waits for to batch or flex.** Evaluations, indexing and document runs first.
5. **Only then test routing,** on your own evaluation set (how to build one), against the best single cheaper model, counting the router's own cost and the lost cache, and judged on cost per completed task. Route whole tasks before you route steps.

After the change, track three numbers monthly: the hit rate, the cost per completed task and the success rate. If the first falls, someone changed the prompt. If the third falls after routing, the saving was not real.

## How much weight this evidence deserves

**Appraising the evidence**

| Evidence | Type | Strength | Main limitation |
| --- | --- | --- | --- |
| Caching and batch documentation, price pages | Vendor | Authoritative for prices and rules | Silent on typical hit rates; rules change monthly; Google's pages disagree on batch |
| Hit-rate figures (OpenAI examples, Anthropic range) | Vendor | The only published figures | Examples, not measurements of typical use |
| RouteLLM, FrugalGPT | Papers | Peer reviewed, methods open | Chat and classification, older models, partly LLM-judged, flattering baselines |
| RouterBench, LLMRouterBench, RouterArena | Benchmarks | Same questions for every router; mostly the same models | One model per question or task, no routing inside an agent; RouterBench is by a router vendor |
| TwinRouterBench, LangChain test of NVIDIA's router | Vendor benchmarks | Real agent tasks, real spend | One run or five, small, own product wins or is sold |
| My cost chart | Own arithmetic | Transparent, reproducible | One call shape, list prices, worst-case writes |

Data: https://bardak.ai/data/prompt-caching-batching-routing/appraising-the-evidence.csv

## Sources

- [Anthropic. Prompt caching (read 16 November 2026).](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
- [Anthropic. Pricing (read 16 November 2026).](https://platform.claude.com/docs/en/about-claude/pricing)
- [Anthropic. Batch processing (read 16 November 2026).](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [Anthropic Engineering (24 November 2025). Introducing advanced tool use on the Claude Developer Platform.](https://www.anthropic.com/engineering/advanced-tool-use)
- [OpenAI. Prompt caching (read 16 November 2026).](https://developers.openai.com/api/docs/guides/prompt-caching)
- [OpenAI. Pricing (read 16 November 2026).](https://developers.openai.com/api/docs/pricing)
- [OpenAI. Batch API (read 16 November 2026).](https://developers.openai.com/api/docs/guides/batch)
- [OpenAI. Flex processing (read 16 November 2026).](https://developers.openai.com/api/docs/guides/flex-processing)
- [Google. Context caching (read 16 November 2026).](https://ai.google.dev/gemini-api/docs/caching)
- [Google. Gemini Developer API pricing (read 16 November 2026).](https://ai.google.dev/gemini-api/docs/pricing)
- [Google. Batch API (read 16 November 2026).](https://ai.google.dev/gemini-api/docs/batch-api)
- [Google. Flex inference (read 16 November 2026).](https://ai.google.dev/gemini-api/docs/flex-inference)
- [Ong et al. (2024, v4 2025). RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025. arXiv:2406.18665.](https://arxiv.org/abs/2406.18665)
- [Chen, Zaharia and Zou (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.](https://arxiv.org/abs/2305.05176)
- [Hu et al. (2024). RouterBench: A Benchmark for Multi-LLM Routing System. arXiv:2403.12031.](https://arxiv.org/abs/2403.12031)
- [Li et al. (2026). LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. arXiv:2601.07206.](https://arxiv.org/abs/2601.07206)
- [Lu et al. (2025). RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers. arXiv:2510.00202.](https://arxiv.org/abs/2510.00202)
- [Yang et al. (2026). TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing. arXiv:2605.18859.](https://arxiv.org/abs/2605.18859)
- [Tangedipalli and Singh (11 August 2026). How many of your agent's calls actually need a frontier model? LangChain blog.](https://langchain.com/blog/switchyard-agent-routing-benchmark)
