
Cutting LLM Costs in Production: Caching, Model Routing and Batching, with a Worked Example
16 November 2026
A successful AI proof of concept answers an important question: can the proposed approach solve the problem at all? It does not answer the harder questions that appear once the system is connected to real users, real company data and real business processes.
This distinction explains why the transition from AI PoC to production remains difficult even when the prototype itself performs well. Research conducted by IDC for Lenovo found that, across the organisations studied, only four out of every 33 AI proofs of concept progressed to production deployment. This should not be interpreted as evidence that 88% of the models technically failed. It shows how often promising experiments fail to become operational systems because the organisation, data, integrations or economics are not ready for the next stage.
For a clearly defined enterprise AI assistant, however, moving from a validated PoC to an initial production release within approximately 90 days can be realistic. The important qualification is validated. The business problem should already be known, the core AI approach should have demonstrated sufficient potential, and the company should be able to provide access to the people, systems and data required for production work. Ninety days is not a realistic promise for discovering the use case, cleaning years of inaccessible data, rebuilding legacy infrastructure and launching an enterprise-wide AI platform at the same time.
The objective of the 90-day roadmap is therefore not to turn an experiment into a perfect final product. It is to turn it into a controlled, observable and maintainable production system that can serve a defined group of users and generate enough evidence to justify further scaling.
What actually changes between an AI PoC and production?
A PoC can work with a prepared dataset, manually selected documents and a developer watching every request. Production cannot. An enterprise AI assistant has to operate with real permissions, changing data, unpredictable questions, unavailable dependencies and users who do not know how the system was designed.
That changes the engineering problem. A production-ready assistant needs authentication and access control, stable integrations with source systems, an evaluation framework, logging and tracing, cost monitoring, failure handling and an explicit escalation path when the system cannot complete a task safely. If the assistant uses retrieval, the team must also be able to determine what information was retrieved and why. If it can call tools or execute actions, every important action needs an appropriate level of control and auditability.
The difference is therefore not simply that production uses a larger model or more infrastructure. Production adds the controls required to operate the system when nobody from the development team is standing next to it.
The team required to move an enterprise AI assistant into production
A 90-day programme does not require a large AI department, but it does require several responsibilities to be covered from the beginning. The most important role on the client side is the business owner: someone who understands the workflow, can resolve scope questions and has authority to decide whether the system is delivering enough value to proceed. Without this role, engineering decisions accumulate while nobody can make the business decisions that determine whether the project should continue.
On the delivery side, the core team normally combines AI engineering with conventional software engineering. An AI or ML engineer is responsible for model behaviour, prompts, retrieval, evaluations and model-related experimentation. A backend or integration engineer connects the assistant to identity systems, databases, APIs and the applications in which employees actually work. Data engineering becomes important when company knowledge has to be continuously ingested, transformed or permissioned rather than uploaded once for a demonstration.
Platform, DevOps and security responsibilities also need to be covered, although they do not always require separate full-time roles. The same applies to frontend work when the assistant needs a dedicated interface. Domain specialists from the client organisation should participate throughout the project because they are usually the only people capable of distinguishing an answer that sounds plausible from one that is genuinely useful in the workflow.
For a bounded assistant, several of these responsibilities can be combined within a senior team. What cannot be combined away is ownership. Someone must own business value, someone must own the application in production and someone must have authority to stop a release when the evidence is not good enough.
Week 1: define the production problem, not the demo
The first week should establish exactly what will enter production. A useful scope is expressed as a business workflow rather than a broad capability such as “an AI assistant for our employees”. The team needs to know who will use it, what they are trying to accomplish, which systems provide the source of truth and what happens today without AI.
This is also when the baseline is recorded. If the assistant is intended to reduce the time required to resolve support cases, prepare offers or search internal documentation, the team needs to know how the process performs before AI is introduced. Otherwise, a successful launch can produce impressive technical metrics without proving that anything improved.
The first week should finish with a narrow use case, a named owner and measurable acceptance criteria.
Week 2: validate data, architecture, security and compliance constraints
Before hardening the PoC, the team should establish whether the environment in which it needs to operate is actually accessible. This means checking source data, authentication, permissions, API availability, hosting requirements, retention policies and any restrictions on sending information to external model providers.
The architecture should remain as simple as the use case allows. The purpose of this week is not to design every future capability. It is to identify the minimum production architecture that can support the first release without creating a dead end.
Regulatory questions also belong here rather than at the end of development. In the EU, for example, Article 50 transparency requirements have applied since 2 August 2026 to relevant interactive AI systems, including requirements to inform users when they are interacting with AI. Whether additional AI Act obligations apply depends on the particular system and use case, so classification should be completed before the release process is designed.
At this point there should already be a credible reason to stop if the required data cannot be accessed, the proposed integration violates security policy or the use case cannot support a defensible business case.
Week 3: turn the PoC into something that can be evaluated repeatedly
A demonstration is usually evaluated by people looking at a handful of outputs. A production system needs repeatable evaluation.
The team should therefore create a representative test set containing normal requests, difficult cases and known failure modes. For a knowledge assistant, this may test whether the correct evidence is retrieved and whether answers remain grounded in approved sources. For an assistant that performs actions, evaluation also needs to examine tool selection, arguments, permissions and the decision to escalate instead of acting.
The important change is that quality becomes measurable across versions. From this point onward, changing a prompt, model, retrieval configuration or tool description should not require guessing whether the system improved.
Week 4: build the first production-shaped vertical slice
By the end of the first month, the team should be able to run one complete workflow using real or production-representative data and at least the essential integration path.
This is not a full rollout. It is evidence that the architecture works end to end: a real user can authenticate, the system can access information according to that user’s permissions, the assistant can complete the bounded task, and the execution can be reconstructed afterwards.
This is the first major go/no-go gate. If the system only performs well when engineers manually prepare its context or bypass production constraints, the project has not yet crossed the PoC boundary.
Week 5: implement identity, permissions and enterprise integrations
Many AI assistants become much more difficult at exactly this point. The model may already generate good answers, but the information available to one employee is not necessarily available to another, and production APIs behave differently from mocked endpoints.
Permissions should travel through the AI workflow rather than disappear once the user enters the assistant. Retrieval should respect access boundaries. Tool execution should follow the privileges of the workflow. Sensitive actions should be explicit rather than hidden inside a broad autonomous capability.
This week often determines whether the prototype can genuinely become part of the organisation’s systems rather than remain a separate AI interface.
Week 6: make the knowledge and retrieval layer maintainable
If the assistant relies on internal knowledge, the project now needs a repeatable process for keeping that knowledge current. Documents should not be manually uploaded every time something changes. The team needs to determine how new content enters the system, how updates propagate, how obsolete content is removed and how metadata and permissions are preserved.
For RAG-based assistants, this is also the point at which retrieval quality should be evaluated separately from the final model response. A fluent answer can hide poor retrieval, making production problems difficult to diagnose later.
The objective is not simply to make the assistant answer correctly today. It is to prevent the knowledge layer from quietly deteriorating over the following months.
Week 7: add observability before the pilot begins
Monitoring should be part of the release architecture, not a task added after users complain.
For an enterprise AI assistant, useful observability extends beyond uptime and API latency. Engineers should be able to reconstruct the path of a request: which application version was active, which model was used, what context was supplied, what information was retrieved, which tools were called, what policies were applied and what response or action reached the user.
This is also the right moment to start measuring production economics. Instead of tracking token consumption in isolation, measure the cost of completing the business task successfully. A cheaper response that requires an employee to repeat the task or manually correct the result may have worse economics than a more expensive inference that actually completes the workflow.
Week 8: test failure, not only success
The system should now be tested against the situations the demo was designed to avoid. What happens when retrieval returns nothing useful? What if a source system is unavailable, a user asks for something outside the approved scope, a prompt injection appears in retrieved content or a model provider times out?
Failure behaviour should be designed explicitly. Depending on the workflow, the correct response may be to ask for clarification, refuse an action, use a fallback path, escalate to a human or temporarily disable a feature.
Rollback should also be practical by this stage. A team that cannot quickly return to a known working configuration after changing a prompt, model, retrieval strategy or orchestration rule does not yet have a safe production process.
Week 9: start a controlled pilot with real users
Only now should the assistant begin serving a small, clearly identified production cohort. The purpose of the pilot is not to collect general opinions about whether users “like AI”. It is to observe what happens to the real workflow.
Do users accept the outputs? Which tasks are abandoned? Where do they correct the assistant? Which questions trigger escalations? Does the process actually become faster or does the assistant introduce an additional verification step that cancels the expected benefit?
This evidence is usually more valuable than another round of offline prompt optimisation because it exposes assumptions that were invisible during development.
Week 10: correct the system using production evidence
The tenth week is for fixing what the controlled rollout has revealed. Some problems will belong to the model, but many will not. Poor results may originate in retrieval, stale documents, missing permissions, ambiguous interface design, tool descriptions or a workflow that asks the assistant to make a decision it should never have been responsible for.
Changes should return through the evaluation suite before being released. This creates the beginning of a production feedback loop: observe behaviour, diagnose the responsible layer, implement a change, test for regression and then deploy it in a controlled way.
Week 11: prepare operations, support and ownership
Before a broader launch, the organisation needs to know what happens on an ordinary Monday morning when the implementation team is not actively watching every request.
Operational responsibilities should therefore be explicit. Someone needs to own incidents, quality changes, data-source problems and model or prompt updates. The support path for end users should be clear, as should the distinction between an application incident and an AI quality issue.
Runbooks should cover the most important failure scenarios, and release procedures should define who can approve changes and when rollback is required. Production readiness is as much an organisational condition as a technical one.
Week 12: release progressively rather than switching everyone at once
A production launch should increase exposure gradually while the team watches both technical and business metrics. The exact rollout pattern will depend on the organisation, but the principle is consistent: additional users or workflows should be added only when the evidence from the previous stage remains within agreed limits.
At this stage, reliability, latency and application errors matter, but they should be evaluated alongside task completion, user corrections, escalation frequency, cost and the business KPI defined during the first week.
A technically healthy assistant that fails to improve the process is not a successful production deployment.
Days 85–90: decide whether to scale, narrow or stop
The end of the roadmap is a decision point, not a ceremonial project closing.
By this stage, leadership should be able to answer several questions with evidence. Does the assistant improve the target process? Does its quality remain acceptable on real cases? Are operating costs compatible with the value created? Can the organisation detect degradation and recover from failures? Do users incorporate the system into their workflow rather than work around it? Is there a team that can operate and improve it after the initial release?
If the answer is yes, the next phase can focus on scale: additional users, adjacent workflows, more data sources or greater automation. If only part of the business case has been proven, narrowing the scope can be the better decision. And if the economics, data or workflow do not support production use, stopping is a successful outcome of the validation process rather than something that should be hidden by continuing to invest.
When should an AI PoC be stopped before production?
The strongest AI programmes do not maximise the number of prototypes that reach production. They stop weak projects early enough to protect resources for better ones.
A project should not move forward simply because the demo is technically impressive. It should be paused or redesigned when no accountable business owner exists, when the expected outcome cannot be compared with a meaningful baseline, when representative evaluation remains below the agreed acceptance threshold, when critical data cannot be accessed under production security rules, or when the cost of operating the system is disproportionate to the value of the task.
The same applies when production testing shows that users must review almost every answer manually. Human oversight can be an intentional part of an AI workflow, but if the system merely adds another step without removing enough work elsewhere, the original business case needs to be reconsidered.
What should the budget include?
The cost of moving from AI PoC to production is not simply the cost of the model multiplied by the number of users. In many enterprise projects, model inference is only one part of the operating cost.
The implementation budget has to account for the engineering required to connect the assistant to real systems, prepare and maintain its data layer, implement permissions, build evaluation and observability, perform security work and support the initial rollout. It should also separate one-off delivery costs from recurring costs such as model usage, infrastructure, observability services, data processing and ongoing maintenance.
Greenlogic has historically published an average cost of EUR 9,200 for a two-week PoC in 2023. That figure is useful as a historical reference for the initial feasibility stage, but it should not be treated as a current 2026 production budget or extrapolated linearly across a 90-day programme. The scope of production work depends much more heavily on integration complexity, security requirements, data readiness and the level of autonomy given to the assistant.
A credible budget should therefore emerge from the first part of the roadmap, once those constraints are known. For organisations comparing vendors, this is also a useful test: an unusually precise production quote issued before anyone has inspected the data, systems and security requirements may reflect confidence in a standard package rather than understanding of the actual implementation.
Why a 90-day roadmap is different from simply extending the PoC
One of the most expensive mistakes is treating production as a longer version of prototyping. The objectives are different.
A PoC reduces uncertainty about feasibility. Production engineering reduces uncertainty about reliability, security, operations and business value. That is why much of the work described in this roadmap does not involve improving the model itself.
This distinction also prevents unnecessary rebuilding. A good prototype does not need every production control from day one, but it should make the assumptions behind the prototype visible enough that the team knows what must change before release. The closer those assumptions are to reality, the shorter the path to production becomes.
The experience reported in the Swiss financial sector illustrates the same principle from a highly regulated environment. Its 2025 study found data access, executive sponsorship, governance, IT-business cooperation and MLOps capabilities among the factors associated with successful deployments, while privacy, security, data quality and weak business-case evidence remained important barriers to scaling.
From AI PoC to production: what success looks like after 90 days
After 90 days, success should not be defined as “the AI is live”. The organisation should have a bounded production system whose performance can be measured, whose behaviour can be investigated and whose failures have an owner.
The team should know what the assistant costs, where it creates value and where it still requires human intervention. It should be possible to change the model, prompts, retrieval logic or application code without losing the ability to detect regressions. Users should know when they are interacting with AI and what to do when the system cannot complete a task. The business should have enough production evidence to decide whether further investment is justified.
That is the real transition from AI PoC to production. The model stops being the centre of the project. The workflow, operating model and measurable business outcome take its place.


