AI Evaluation in Production: The Gap Most Enterprise Teams Ignore

5 min read

Key Claim: The gap between observability and evaluation is the most prevalent governance failure in enterprise AI deployments — teams can see their models running, but most cannot tell whether the outputs are correct.

Most enterprise teams deploying large language models in production have instrumented tracing and logging — they can see the requests coming in and the responses going out. What the majority cannot do is tell you, at any given moment, whether the outputs are correct, faithful to their source data, or degrading quietly. That distinction — between observability and evaluation — is where most production AI deployments currently fail.

Deployment speed is compounding the problem. Databricks reports a 327% increase in multi-agent AI workflows in just four months; the same report indicates that 78% of enterprises now use two or more LLM families in the same production environment. Quality infrastructure has not kept pace with this acceleration. As our earlier analysis of the enterprise AI agents deployment gap documented, the majority of organisations counting agents as “in production” lack the governance infrastructure to operate them reliably at scale.

LangChain’s 2026 State of Agent Engineering report found that 89% of organisations have implemented observability for their AI agents. Just 52% run offline evaluations, and only 37% run any form of online evaluation against live production traffic. Among teams that have confirmed production deployments, nearly 23% do not evaluate their agents at all.

What Observability Gives You — and What It Does Not

Observability tools for LLM applications — LangSmith, Langfuse, Arize Phoenix, Helicone — capture traces: the inputs, outputs, latency figures, token counts, and call chains for every model invocation. This is necessary and valuable. You can diagnose why a particular call failed, identify cost spikes, and trace a multi-step agent’s reasoning path.

What traces cannot tell you is whether the answer was right. A response that arrives in 200 milliseconds with normal token usage may still be factually wrong, hallucinated, unsafe, or subtly worse than last week’s outputs. Observability surfaces the plumbing. Evaluation measures the water quality.

The failure mode that results from relying on observability alone is well documented. Unlike traditional software, which throws 500 errors and triggers alerts, LLM applications degrade gradually. As practitioner Anil Ambharii described it in a 2025 post: responses become “subtly wrong, slightly fabricated, marginally unsafe, or quietly expensive” without triggering any alarm.

The Air Canada chatbot case is a clean example: the model fabricated a bereavement discount policy that did not exist, and a customer successfully claimed against the airline in court. OpenAI’s Whisper transcription tool, deployed by hospitals, has been documented to insert text never spoken by patients or doctors. In 2023, a New York lawyer submitted a court brief containing ChatGPT-generated case citations that did not exist. None of these failures would have appeared in a trace as an error.

Why Benchmark Scores Do Not Transfer

The first instinct for many teams is to rely on published benchmark results. This assumption fails for several structural reasons.

First, benchmark contamination: test set data leaks into training corpora, inflating published scores. Second, distribution mismatch: the documents, queries, and edge cases in a typical enterprise deployment differ substantially from curated benchmark datasets. Third, and most practically, benchmarks measure the model, not the system. A production deployment includes retrieval layers, prompt templates, chunking strategies, and post-processing logic. Each component introduces quality variation that a model benchmark cannot capture.

Offline evaluation against a static test set addresses the distribution problem but not the temporal problem. Model providers update and fine-tune models continuously. An offline evaluation that passed last month may not reflect today’s system behaviour. The same limitation applies to reasoning models: as our analysis of reasoning models in production showed, chain-of-thought outputs introduce quality dimensions that conventional benchmarks do not measure.

The Eval Framework Landscape

RAGAS is the most widely used open-source library for evaluating RAG pipelines, with approximately 25,000 GitHub stars under an Apache 2.0 licence. Its core metric suite covers faithfulness, answer relevancy, context precision, and context recall. The limitation is scope: RAGAS is a scoring library, not a platform.

LangSmith (LangChain’s platform) provides tightly integrated tracing and evaluation for teams building on LangChain or LangGraph. The Developer plan is free but capped at 5,000 traces per month; the Plus tier costs $39 per seat per month.

Braintrust positions itself as the most complete end-to-end evaluation platform: dataset management, offline evaluation, CI/CD enforcement, experiment tracking, and online scoring of live traffic. In February 2026, it raised an $80 million Series B led by Iconiq at an $800 million post-money valuation. Customers include Notion, Replit, Cloudflare, Ramp, and Dropbox.

Arize Phoenix is notable for being open-source and self-hostable while offering one of the more extensive online evaluation capabilities in the market — scoring live production traces and sessions in addition to offline evaluation, which is specifically relevant for agent evaluation.

LLM-as-Judge: What It Can and Cannot Do

Manual human review does not scale at production traffic volumes. The emerging standard for automated evaluation is LLM-as-judge: using a separate model to score the outputs of the production model against a natural-language rubric.

GPT-4 as judge achieves roughly 80% agreement with human evaluators; more sophisticated judge models reach approximately 85%. The known failure modes include position bias (tendency to favour the first answer), verbosity bias (inflating scores for longer responses regardless of accuracy), and self-enhancement bias (models rating outputs similar to their training distribution more favourably).

What Best Practice Looks Like

The organisations making production AI work treat evaluation as infrastructure, not a pre-launch checklist. Databricks reported in its 2026 State of AI Agents report that companies using evaluation tools get nearly six times more AI projects into production than those that do not.

The architecture has four components that need to be in place before a production deployment is considered complete: (1) a versioned evaluation dataset drawn from production traces; (2) evaluation results as a CI/CD gate; (3) online scoring of live traffic; and (4) human review on a structured cadence.

The most common omission is the third component. Organisations have observability (traces exist) and often have offline evaluations, but no mechanism to score live production outputs. The result is that quality can degrade between release cycles with no signal until a customer surfaces it.

Implications: What to Watch

Gartner formalised a new platform category in 2026: AI Evaluation and Observability Platforms (AEOPs), projecting that 60% of software engineering teams will adopt them by 2028, up from 18% in 2025. The forcing function is likely to be accountability pressure rather than technical maturity. The EU AI Act’s high-risk system requirements create an audit trail obligation that a trace log without evaluation scores cannot satisfy.

For engineering teams, the immediate priority is closing the gap between observability and evaluation: instrument online scoring on a sample of production traffic before the next release cycle, not as a separate initiative, but as the final component of what “deployed” means.

This article was produced with AI assistance and reviewed by the editorial team.