
Article 50 of the EU AI Act: Transparency Duties for Chatbots and Generative AI Features
30 September 2026
From AI PoC to Production in 90 Days: A Realistic Roadmap for an Enterprise AI Assistant
18 November 2026
When an AI pilot becomes a production system, the model bill tends to grow faster than the number of users. Prompts get longer, retrieval adds context, agents make several calls per task, and traffic that was a few hundred test requests becomes hundreds of thousands a month. Our article on the economics of enterprise AI systems explains where those costs come from. This article is about what you can do about them.
A handful of techniques account for most of the savings: prompt caching, trimming context and output, routing simple requests to a smaller model, batching work that can wait, and caching whole responses. Their combined effect depends heavily on the workload, which is why this article uses a worked example. The example is a model of a typical retrieval-based assistant, not a client workload. Every assumption is stated, the price multipliers come from the providers’ documentation as it stood on 14 November 2026, and the arithmetic is simple enough to repeat with your own numbers.
Start With Cost per Task, Not the Monthly Bill
The monthly bill tells you that costs grew. Cost per task tells you why. For a single task the formula is straightforward: the number of model calls, multiplied by the input tokens times the input price plus the output tokens times the output price. Output tokens are usually priced several times higher than input tokens, and the example below assumes a ratio of 5 to 1, so check the ratio on your own price list.
Before changing anything, log the following for every call: tokens split by component (static instructions and tool definitions, retrieved context, conversation history, user message, output), the model used, whether the prompt was served from cache, latency, and a quality score from a fixed evaluation set. Without the split by component you cannot tell which technique is worth applying. Our article on AI observability in enterprise systems describes how to collect this kind of telemetry.
The Workload Behind the Example
The example models an internal assistant that answers employee questions over company documents. The assumptions are deliberately round.
- 500,000 requests per month.
- Each request carries 2,500 tokens of static prefix (instructions, tool definitions, policy text), 1,000 tokens of retrieved context and 300 tokens of user message and short history, so 3,800 input tokens in total.
- Each answer is 400 tokens long.
- Costs are expressed in cost units, where one unit is the price of one million input tokens on the large model. Output costs 5 units per million tokens.
- A smaller model costs one fifth of the large model on both input and output.
- 20% of requests are offline jobs, such as nightly summarisation, that do not need an immediate answer.
On these assumptions the baseline, with every request on the large model and no caching, is 1,900 units for input and 1,000 units for output, so 2,900 cost units per month. We call that index 100. Multiply the units by your own price per million input tokens to get a currency figure.
Technique 1: Prompt Caching
Prompt caching reuses the work done on a prompt prefix that repeats between requests, so that the stable part of the prompt is billed at a reduced rate. Both major providers document it, and the rules are similar in outline.
Anthropic’s prompt caching documentation specifies that a cache write with the default five-minute lifetime costs 1.25 times the base input price, a write with a one-hour lifetime costs 2 times, and a cache read costs 0.1 times, with cheaper read multipliers on some models (0.05 times on Claude Opus 5.5 and Sonnet 5.5, and 0.025 times on the Fable and Mythos 5.x models). The minimum cacheable prompt length depends on the model and runs from 512 to 4,096 tokens, and shorter prompts are processed without caching and without an error. These multipliers stack with the batch discount described below.
OpenAI’s prompt caching guide states that caching is enabled by default for supported models and that the cached-input rate can be discounted by up to 95%. For GPT-5.6 and later the minimum cacheable length is 1,024 tokens, a write costs 1.25 times the standard input rate for the default 30-minute lifetime, and reads cost 0.1 times on most models. The guide gives a useful illustration: over ten requests, one write and nine full reads cost 2.15 times the normal input cost, against 10 times without caching.
Three practical rules follow. Put stable content first and variable content last, because the prefix is the cache key. Keep the system prompt and tool definitions byte-identical between requests, so no timestamps or request identifiers in the prefix. And monitor the cache hit rate, because a small change to a template can silently reduce it to zero.
In the example, 2,500 of the 3,800 input tokens are cacheable. We assume steady traffic keeps the cache warm, with a 95% hit rate, a read multiplier of 0.1 and a write multiplier of 1.25. The effective cost of the prefix becomes 0.95 × 0.1 + 0.05 × 1.25 = 0.1575 of its original value.
Technique 2: Trim the Context and Cap the Output
Every token you do not send is a token you do not pay for, and output tokens cost the most. Two changes are usually low-risk. The first is to improve retrieval, for example with better chunking, tuned top-k, reranking and de-duplication, so that the model receives 700 relevant tokens instead of 1,000 mixed ones. The second is to specify the answer format and set a sensible maximum output length, which in the example reduces the average answer from 400 to 300 tokens.
Prompt compression goes further. The LLMLingua paper reports up to 20 times compression with little loss of performance on the benchmarks it tested, but benchmark results do not transfer automatically to retrieval over your own documents. Treat compression as something to evaluate, not to assume. Both changes in this section also reduce latency, since shorter prompts and shorter answers are faster to process, but both need a quality check because trimming can remove exactly the passage the answer depended on. Our article on latency versus accuracy in RAG covers the trade-off in more detail.
Technique 3: Route Simple Requests to a Smaller Model
Most real traffic is not uniformly hard. Questions that ask for a policy number or a short factual lookup do not need the strongest model, while multi-step reasoning does. Routing sends each request to a model matched to its difficulty. The research literature gives a sense of the potential. The FrugalGPT paper notes that API fees for different models can differ by two orders of magnitude and reports that a cascade of models matched the best individual model, GPT-4 in their experiments, with up to 98% cost reduction. The RouteLLM paper, published at ICLR 2025, reports cost reductions of over 2 times in certain cases without compromising the quality of responses. These are results on research benchmarks, not promises for your traffic.
In practice there are three levels of sophistication. Start with rules based on task type or feature, since that needs no extra model. Add a lightweight classifier or router once you have logged enough traffic to train or tune it. And use a cascade, in which the small model answers first and the request escalates to the large model when confidence is low, to protect quality where the cost of an error is high.
In the example, 60% of requests are routed to the small model. A router call, made by the small model with 400 input tokens and 5 output tokens, is added to every request, and 5% of routed requests escalate to the large model and therefore pay for both calls. Routing is also the technique with the most dependence on evaluation: a misrouted request is a worse answer, so the router needs its own measurement.
Technique 4: Batch Work That Does Not Need an Instant Answer
Batch APIs process groups of requests asynchronously at a lower price. OpenAI’s Batch API offers 50% lower costs, a separate pool of higher rate limits and a 24-hour turnaround, and Anthropic’s Message Batches API also cuts costs by 50%. Because the turnaround can be up to a day, batching suits offline work only: nightly summarisation, classification of a backlog, evaluation runs, and enrichment of records. It does not suit anything a user is waiting for.
There is a second meaning of batching: packing several items into one prompt so that the instructions are paid for once. It can help for classification of many short items, but it complicates output parsing and can reduce accuracy on individual items, so test it before relying on it. In the example, 20% of requests are offline jobs that qualify for the 50% batch discount.
Technique 5: Semantic Response Caching
A semantic cache stores previous answers and returns one when a new question is sufficiently similar, judged by embedding similarity. The saving is the hit rate multiplied by the cost of the call, minus the cost of the embedding and lookup. The risk is a false hit: two questions can be worded alike and mean different things, or the right answer can depend on the user, the tenant, or the date.
If you use it, scope the cache by permission and tenant, give entries a lifetime that reflects how quickly the answer goes stale, set the similarity threshold conservatively, and measure the false-hit rate on logged queries before switching it on. Because the benefit depends entirely on how repetitive the traffic is, the example does not include semantic caching in its totals. As a rule of thumb, every 10% of interactive requests served from the cache removes about 10% of the interactive cost.
How the Savings Stack in the Example
The table applies the techniques in order, each on top of the previous ones. The cost index is relative to the baseline of 100.
| Step | What changes | Cost index | Saving vs baseline | Effect on latency | Quality gate before rollout |
|---|---|---|---|---|---|
| Baseline | All requests on the large model, no caching | 100.0 | 0% | Reference | Reference |
| Prompt caching | 2,500-token prefix cached, 95% hit rate | 63.7 | 36.3% | Lower time to first token | None expected, verify outputs are identical |
| Output cap | Average answer 400 to 300 tokens | 55.1 | 44.9% | Faster responses | Answers still complete on the evaluation set |
| Context trimming | Retrieved context 1,000 to 700 tokens | 49.9 | 50.1% | Faster responses | No drop in answer correctness on the evaluation set |
| Routing | 60% of requests to the small model, 5% escalated, router overhead included | 28.9 | 71.1% | Faster for routed requests | No meaningful drop per request category |
| Batching | 20% of requests run as batch jobs at half price | 26.0 | 74.0% | Up to 24 hours for batched jobs only | Applies to offline jobs only |
Caching is the largest single step because the static prefix is two thirds of the input tokens, and routing is the second largest because it applies to a large share of requests. Batching contributes least here because only a fifth of the traffic is eligible.
How Sensitive Is the Result?
A single end figure hides how much depends on the assumptions. Changing one assumption at a time, with the others held at the example values, moves the final cost index as follows.
| Assumption changed | Final cost index (example: 26.0) |
|---|---|
| Cache hit rate 70% instead of 95% | 32.1 |
| Cache hit rate 98% | 25.3 |
| Static prefix of 1,000 tokens instead of 2,500 | 32.4 |
| Only 30% of requests routable to the small model | 36.1 |
| 80% of requests routable | 19.3 |
| 20% of routed requests escalate instead of 5% | 30.1 |
| Small model at 0.4 of the large model’s price | 32.7 |
| Small model at 0.1 of the large model’s price | 22.7 |
| No batch-eligible traffic | 28.9 |
Combining several unfavourable assumptions (a 1,000-token prefix, a 70% hit rate, 30% routable traffic, 5% batch-eligible traffic and no context or output gains) gives a final index of 67.7, a saving of about a third. Combining favourable ones (80% routable traffic and a 98% hit rate) gives 18.7. The range is wide, and the share of routable traffic moves the result more than any other single assumption, which is a good reason to measure your own request mix before committing engineering time.
What to Measure So the Savings Are Real
An optimisation that is not measured can quietly degrade quality. Track the following continuously rather than once.
- Cost per task, and tokens per task by component.
- Cache hit rate for prompts, and for responses if you use a semantic cache.
- The routing distribution and the escalation rate.
- Median and 95th percentile latency, split by route.
- Quality on a fixed evaluation set, plus a sample of live answers reviewed by people.
- Cost anomalies, such as a sudden fall in hit rate after a template change.
Set the acceptable quality drop for each change before rollout, and keep a way to roll back. Our article on building feedback loops for enterprise AI explains how to turn production feedback into that evaluation set.
The Order We Would Apply Them In
The sequence follows risk: start where nothing about the answer changes and finish where quality needs the closest watching.
- Instrument the system and establish cost per task.
- Enable prompt caching and restructure prompts so that the stable content comes first.
- Cap output length and trim context, checking quality on the evaluation set.
- Move offline work to batch processing.
- Introduce routing, starting with simple rules and adding a classifier later.
- Consider a semantic cache last, and only for traffic that is demonstrably repetitive.
It is also worth saying when not to optimise. If monthly spend is small, the engineering time may cost more than the savings, and a stable workload with a high-value task may justify keeping the strongest model on every call. For the wider picture of cloud cost control, our article on Cloud FinOps and Green IT places model spend in the context of overall infrastructure cost.
Conclusion
LLM cost optimisation is mostly a matter of order and measurement. Caching and trimming save money without changing what the system says, routing and batching save more but need quality checks, and the combined result depends on your own request mix, prefix length and cache hit rate. The worked example shows how the techniques compound, and the sensitivity table shows why you should replace its assumptions with your own logs before deciding where to start.
References
- Anthropic. Prompt caching. Claude Platform documentation, accessed 14 November 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Anthropic. Pricing. Claude Platform documentation, accessed 14 November 2026. https://platform.claude.com/docs/en/about-claude/pricing
- Anthropic. Batch processing. Claude Platform documentation, accessed 14 November 2026. https://platform.claude.com/docs/en/build-with-claude/batch-processing
- OpenAI. Prompt caching. API documentation, accessed 14 November 2026. https://platform.openai.com/docs/guides/prompt-caching
- OpenAI. Batch API. API documentation, accessed 14 November 2026. https://platform.openai.com/docs/guides/batch
- Ong I. et al. RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025. arXiv:2406.18665.
- Chen L., Zaharia M., Zou J. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176, 2023.
- Jiang H. et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. arXiv:2310.05736.

