Perspective from engineers who operate these systems

Most retrieval-augmented generation systems are built once and evaluated once — against a small, hand-picked set of questions the team already knows the answers to. That evaluation tells you almost nothing about how the system behaves against the long tail of real queries, and it tells you nothing at all about how it behaves six months later, after the underlying documents have changed and the index has quietly gone stale.

The failure mode is rarely dramatic. It is a slow accumulation of wrong context: an index that was never re-embedded after a document update, a chunking strategy that splits a table across two fragments and destroys its meaning, a retriever that returns five plausible-looking passages when the correct answer required a sixth. Users do not report these failures as bugs. They simply stop trusting the system and route around it.

The systems that hold up in production treat retrieval as an engineering discipline with its own lifecycle, not a one-time integration. That means index freshness policies tied to the actual cadence of source-document change, evaluation sets that are refreshed continuously rather than fixed at launch, and citation-level traceability so any answer can be walked back to the exact source it came from. It also means measuring grounding accuracy specifically, separate from fluency — a confident, well-written answer that cites the wrong passage is a worse outcome than a system that says it does not know.

None of this is exotic. It is the same operational discipline applied to any other production data system. The organizations getting durable value from RAG are the ones that stopped treating it as a prompt-engineering exercise and started treating it as infrastructure.

Commercial LLM APIs are, correctly, the default starting point for almost every generative AI initiative. They remove months of infrastructure work and let a team validate whether a use case has value before committing to anything heavier. The mistake is treating that default as a permanent architecture decision rather than a starting one.

The economics shift once a workload crosses from experimental to core: consistent, high-volume traffic; latency requirements tight enough that network hops to a third-party endpoint matter; and — often the deciding factor — data that should never leave a controlled environment in the first place. At that point, the per-token price of a commercial API stops being the only cost. Add in the unpredictability of rate limits during peak load, the lack of control over model updates that silently change behavior, and the compounding cost of routing every sensitive record through an external boundary.

The build case is not about avoiding commercial models — it is about owning the parts of the stack that determine cost, control, and reliability at your specific volume: fine-tuning smaller, task-specific models on your own data, owning the serving infrastructure, and keeping evaluation in-house so quality is measured against your own benchmark rather than a vendor's marketing claim. Done well, an in-house reasoning model can match or exceed commercial tool-calling accuracy at a fraction of the inference cost, while giving the organization direct control over GPU allocation, data handling, and model behavior.

The right question is never build or buy in the abstract. It is: at what volume does this specific workload cross the line, and does the organization have — or is it willing to build — the platform discipline to operate a model in-house responsibly. Most organizations answer that question too late, after cost and data-control problems have already compounded.

The industry's attention on agentic AI has focused almost entirely on orchestration — how agents plan, call tools, and hand off to each other. That is the visible, demoable part of the system. It is also not where production deployments actually fail.

Production agentic systems fail because there is no reliable way to measure whether a given run succeeded. Tool-calling accuracy, task completion, and reasoning quality all require an evaluation harness capable of judging outputs that do not have a single correct answer — which means an LLM-as-jury approach, calibrated against a held-out benchmark, with disagreement rates tracked over time.

Building the agent before building this evaluation infrastructure produces a familiar pattern: early results look impressive in a demo, the team ships to production, and quality silently degrades as the agent encounters cases the demo never covered. Nobody notices until a downstream stakeholder does, and by then the fix requires unwinding decisions made without any way to measure their impact.

The organizations getting agentic AI right invert the build order. They build the evaluation harness and the held-out benchmark first, treat every agent or prompt change as a regression-tested release, and only then invest in expanding what the agent is allowed to do autonomously. Evaluation infrastructure is not the boring prerequisite to agentic AI — it is the thing that makes autonomy safe to grant in the first place.

Generating large volumes of synthetic training data is, at this point, a solved engineering problem — a sufficiently large teacher model can produce tens of thousands of examples in under an hour. The harder problem, and the one that determines whether the resulting model is actually good, is distributional realism.

Synthetic data generated without deliberate variation collapses toward a narrow band of phrasing, tone, and scenario — the teacher model's own defaults. A downstream model fine-tuned on that data learns the teacher's stylistic tics rather than the diversity of real users. The fix is a persona and scenario model built explicitly into the generation pipeline — dozens of dimensions of variation, sampled with enough randomness to cover realistic edge cases without producing incoherent examples.

The second, quieter problem is duplication. Large-scale generation runs produce near-duplicate examples far more often than teams expect — the same scenario phrased five slightly different ways. Left unaddressed, these near-duplicates inflate a training set's apparent size while contributing almost nothing to model quality, and can actively bias a fine-tuned model toward overrepresented patterns. A two-stage deduplication process — an efficient first pass on embeddings, a stricter second pass on the survivors — routinely removes five to ten percent of a batch that looked, at a glance, like clean data.

Neither of these problems is solved by a bigger teacher model or a larger generation budget. They are solved by treating synthetic data generation as its own engineering discipline, with the same rigor applied to variation and deduplication that a data engineering team would apply to any other production dataset.

Most AI governance frameworks are written as policy documents: principles, review checklists, approval committees. They are usually well-intentioned and almost universally ignored in practice, because they sit outside the actual engineering workflow. A policy that requires an engineer to manually document a model's risk profile in a separate system, weeks after the model has already shipped, will be treated as paperwork — because it is paperwork.

Governance that works is built into the pipeline itself, so that compliance is a byproduct of normal development rather than an additional step. Every model version carries its evaluation record automatically. Every production deployment is gated by an approval check that reads from the same system engineers already use to ship. Every prediction can be traced back to the model version, training data, and evaluation results that produced it, without anyone having to reconstruct that trail after the fact.

This is not a rejection of policy — a governance framework still needs to define what fairness, robustness, and human oversight mean for a given organization. But the enforcement mechanism has to be technical, not procedural, or it will not survive contact with a shipping deadline. The organizations with governance that actually holds up under regulatory scrutiny are the ones where an auditor can query the system directly, rather than requesting a document someone has to go write.

A successful proof of concept and a production ML platform are judged by almost entirely different criteria, and organizations that don't recognize this spend years re-litigating a decision that was already validated. The proof of concept answers: can a model solve this problem well enough to matter. The platform question is: can a team ship the next fifty models this fast, this reliably, without heroics.

The gap between the two is rarely model architecture. It is the unglamorous infrastructure around the model: config-driven pipelines instead of copy-pasted notebooks, a shared evaluation harness instead of a bespoke script per project, feature consistency between training and serving instead of silent skew, and a deployment process that doesn't require the original author to be present for every release.

The clearest signal of MLOps maturity is how long it takes a team to stand up a new model-training pipeline for an adjacent use case. In an immature environment, that is measured in weeks of custom engineering. In a mature platform, it's a matter of configuration — new domain, same infrastructure, hours instead of weeks. That single metric compounds: every week saved on setup is a week redirected toward the model quality work that actually differentiates the business.

Organizations that invest in this infrastructure before it feels urgent — while they are still running two or three models, not twenty — build the platform once. Organizations that wait end up rebuilding it under pressure, usually after a scaling crisis has already made the case for them.