Enterprise AI Teams Can See Their Models Running. Most Cannot Tell Whether They Are Working.

Most enterprise teams deploying large language models in production have instrumented tracing and logging — they can see the requests coming in and the responses going out. What the majority cannot do is tell you, at any given moment, whether the outputs are correct, faithful to their source data, or degrading quietly. That distinction — between observability and evaluation — is where most production AI deployments currently fail.

AI evals production overview graphic

Teams that run AI evals production workflows quickly learn that observability alone does not answer whether a model is still doing its job. For a practical companion piece, see our guide to AI agent evaluation in production.

Deployment speed is compounding the problem. Databricks reports a 327% increase in multi-agent AI workflows in just four months; the same report indicates that 78% of enterprises now use two or more LLM families in the same production environment. Quality infrastructure has not kept pace with this acceleration. As our earlier analysis of the enterprise AI agents deployment gap documented, the majority of organisations counting agents as “in production” lack the governance infrastructure to operate them reliably at scale.

LangChain’s 2026 State of Agent Engineering report found that 89% of organisations have implemented observability for their AI agents. Just 52% run offline evaluations, and only 37% run any form of online evaluation against live production traffic, highlighting a widespread absence of AI evals production infrastructure. Among teams that have confirmed production deployments, nearly 23% do not evaluate their agents at all.

Evaluation gets harder once several agents hand work to each other, which is why AI evals production now sit alongside orchestration choices. Readers can compare options in our review of multi-agent frameworks in production.

What Observability Gives You — and What It Does Not

Observability tools for LLM applications — LangSmith, Langfuse, Arize Phoenix, Helicone — capture traces: the inputs, outputs, latency figures, token counts, and call chains for every model invocation. This is necessary and valuable. You can diagnose why a particular call failed, identify cost spikes, and trace a multi-step agent’s reasoning path.

What traces cannot tell you is whether the answer was right. A response that arrives in 200 milliseconds with normal token usage may still be factually wrong, hallucinated, unsafe, or subtly worse than last week’s outputs, which is why systematic AI evals production infrastructure is essential. Observability surfaces the plumbing. Evaluation measures the water quality.

The failure mode that results from relying on observability alone is well documented. Unlike traditional software, which throws 500 errors and triggers alerts, LLM applications degrade gradually. As practitioner Anil Ambharii described it in a 2025 post: responses become “subtly wrong, slightly fabricated, marginally unsafe, or quietly expensive” without triggering any alarm. Teams discover the problem when customers surface it, not when it occurs.

The Air Canada chatbot case is a clean example: the model fabricated a bereavement discount policy that did not exist, and a customer successfully claimed against the airline in court. OpenAI’s Whisper transcription tool, deployed by hospitals to generate appointment records, has been documented to insert text never spoken by patients or doctors. In 2023, a New York lawyer submitted a court brief containing ChatGPT-generated case citations that did not exist, resulting in sanctions and a $5,000 fine.

None of these failures would have appeared in a trace as an error.

McKinsey’s State of AI: Global Survey 2025, covering nearly 2,000 respondents across 105 countries, found that 51% of organisations using AI reported at least one negative consequence, with roughly one-third attributing consequences to AI inaccuracy — outputs that were wrong in ways the team did not catch before they caused harm.

Why Benchmark Scores Do Not Transfer

The first instinct for many teams is to rely on published benchmark results rather than dedicated AI evals production pipelines. If a model scores well on MMLU, HellaSwag, or their sector-specific equivalent, it should perform adequately in production. This assumption fails for several structural reasons.

First, benchmark contamination: test set data leaks into training corpora, inflating published scores in ways that are difficult to detect and that vendors have limited incentive to disclose. Second, distribution mismatch: the documents, queries, and edge cases in a typical enterprise deployment differ substantially from curated benchmark datasets. A model that handles general-knowledge questions well may struggle with internal policy documents, domain-specific terminology, or the specific failure modes of a given organisation’s data.

Third, and most practically, benchmarks measure the model, not the system. A production deployment includes retrieval layers, prompt templates, chunking strategies, context windows, and post-processing logic. Each component introduces quality variation that a model benchmark cannot capture. An LLM that achieves state-of-the-art on a retrieval benchmark may still produce confident hallucinations — authoritative-sounding incorrect answers — when paired with a retrieval layer that pulls poor context.

Offline evaluation against a static test set (a “golden dataset” drawn from production traces) addresses the distribution problem but not the temporal problem. Model providers update and fine-tune models continuously. Prompt templates change. Document corpora evolve. An offline evaluation that passed last month may not reflect today’s system behaviour.

The same limitation applies to reasoning models: as our analysis of reasoning models in production showed, chain-of-thought outputs introduce quality dimensions — faithfulness of reasoning steps, not just final answers — that conventional benchmarks do not measure.

The Eval Framework Landscape

The tooling for structured AI evals production workflows has matured considerably, though adoption has not kept pace.

RAGAS is the most widely used open-source library for evaluating retrieval-augmented generation (RAG) pipelines, with approximately 25,000 GitHub stars under an Apache 2.0 licence. Its core metric suite covers faithfulness (does the answer reflect the retrieved context?), answer relevancy, context precision, and context recall. The limitation is scope: RAGAS is a scoring library, not a platform. It produces metrics but has no experiment tracking, dashboards, or production monitoring capability.

LangSmith (LangChain’s platform) provides tightly integrated tracing and evaluation for teams building on LangChain or LangGraph. Every call automatically generates structured traces, and the annotation queue allows human reviewers to label production samples that can feed back into evaluation datasets. The Developer plan is free but capped at 5,000 traces per month; the Plus tier costs $39 per seat per month.

Braintrust positions itself as the most complete end-to-end evaluation platform: dataset management, offline evaluation, CI/CD enforcement (evaluation results gate deployments), experiment tracking, and online scoring of live traffic. In February 2026, it raised an $80 million Series B led by Iconiq at an $800 million post-money valuation. Customers include Notion, Replit, Cloudflare, Ramp, and Dropbox. Founder Ankur Goyal framed the core problem plainly: “Teams have never had less conviction about what will fail next.

When something does break, it’s never been harder to explain why.”

Arize Phoenix is notable for being open-source and self-hostable while offering one of the more extensive online evaluation capabilities in the market — scoring live production traces and sessions in addition to offline evaluation, which is specifically relevant for agent evaluation where multi-step reasoning chains must be assessed holistically.

The common pattern across these platforms is the shift from measurement to feedback loop: production traces generate evaluation datasets, evaluation results gate releases, and quality signals flow back into prompt iteration.

LLM-as-Judge: What It Can and Cannot Do

Manual human review does not scale at production traffic volumes. The emerging standard for automated AI evals production workflows is LLM-as-judge: using a separate model — typically a more capable or differently optimised one — to score the outputs of the production model against a natural-language rubric.

GPT-4 as judge achieves roughly 80% agreement with human evaluators, matching human-to-human consistency; more sophisticated judge models reach approximately 85%. For most evaluation tasks, this is sufficient to detect systematic quality degradation.

The known failure modes require active mitigation. Research has documented several systematic biases in LLM judges: position bias (a tendency to favour the first answer in a comparison, confirmed across multiple models and tasks), verbosity bias (inflating scores for longer, more detailed responses regardless of accuracy), and self-enhancement bias (models rating outputs similar to their own training distribution more favourably, as quantified by Ye et al.).

In specialised domains — healthcare, legal, financial — practitioners report that judge model agreement with human experts drops meaningfully compared to general-knowledge tasks.

The practical architecture that emerges from these constraints: offline evaluation with LLM-as-judge on curated datasets plus online sampling of production traffic, with human review focused on cases where the judge score falls into a low-confidence range. According to the LangChain survey, roughly 24% of teams running evaluations currently combine offline and online methods — still a minority.

What Best Practice Looks Like

The organisations making production AI work treat AI evals production frameworks as continuous infrastructure, not a pre-launch checklist. Databricks reported in its 2026 State of AI Agents report (covering data from 20,000+ organisations) that companies using evaluation tools get nearly six times more AI projects into production than those that do not, and those using AI governance see 12 times more.

These are vendor-sourced figures and should be read with that context, but the directional finding — that evaluation infrastructure correlates with successful production deployment — is consistent with the LangChain data.

The architecture has four components that need to be in place before a production deployment is considered complete.

1. A versioned evaluation dataset drawn from production traces. Not a synthetic benchmark — a set of real inputs and expected outputs derived from actual use of the system. Updated continuously as the production distribution shifts.

2. Evaluation results as a CI/CD gate. Before any prompt change, model swap, or retrieval configuration update is deployed, automated AI evals production gates ensure it passes defined threshold scores. This prevents the most common failure mode: improving one workflow while silently breaking adjacent ones.

3. Online scoring of live traffic. A sample of production requests is scored continuously — using LLM-as-judge, rule-based metrics, or task-specific classifiers — to detect quality drift between releases. Traces without quality scores are visibility without measurement.

4. Human review on a structured cadence. Automated judges calibrate to the wrong distribution without human correction. A weekly review of flagged low-confidence samples feeds corrections back into both the evaluation dataset and the judge’s calibration.

The most common omission in enterprise deployments is the third component. Organisations have observability (traces exist) and often have offline evaluations (a test set from before deployment), but no mechanism to score live production outputs. The result is that quality can degrade between release cycles with no signal until a customer or internal audit surfaces it.

Telemetry and Guardrail Architecture for AI Evals Production Systems

Deploying robust AI evals production infrastructure requires software teams to move beyond static benchmark scores toward continuous behavioral evaluation. When generative agents interact with volatile customer data, automated AI evals production pipelines must evaluate contextual drift, tool call schemas, and safety policy compliance. Leading engineering teams treat AI evals production metrics as continuous telemetry rather than one-time pre-deployment checklists.

Scaling reliable AI evals production frameworks demands low-latency judge models calibrated against verified human ground truth. To prevent silent degradation in live software environments, AI evals production systems run asynchronous shadow testing against sampled production inference traces. Rigorous AI evals production scoring catches subtle hallucination increases and semantic drift before customer-facing agent failures compromise service-level agreements.

Ultimately, organizations that institutionalize AI evals production workflows establish transparent confidence intervals for automated agent decision-making. By integrating AI evals production monitoring directly into continuous integration pipelines, enterprise platforms maintain deterministic quality standards as underlying foundation model weights and provider endpoints evolve.

Implications: What to Watch

Gartner formalised a new platform category in 2026: AI Evaluation and Observability Platforms (AEOPs), establishing that AI evals production capabilities have become enterprise table stakes, defined as tools that manage the nondeterministic nature of AI systems, automate quality benchmarking, and create a feedback loop from observability data back into evaluation. In its February 2026 Market Guide, Gartner projected that 60% of software engineering teams will adopt AEOPs by 2028, up from 18% in 2025.

A separate Gartner press release predicts explainable AI requirements will drive LLM observability investments to 50% of GenAI deployments by 2028, up from 15% today.

The forcing function is likely to be accountability pressure rather than technical maturity. Every production deployment running without evaluation instrumentation accumulates what amounts to infrastructure debt: an expanding corpus of unscored outputs that cannot be audited, defended, or learned from. As AI systems make decisions with legal, financial, or clinical consequences, the absence of a documented quality measurement process becomes a governance liability.

The EU AI Act’s high-risk system requirements create an audit trail obligation that a trace log without evaluation scores cannot satisfy.

For engineering teams, the immediate priority is closing the gap between observability and evaluation: instrument online scoring on a sample of production traffic before the next release cycle, not as a separate initiative, but as the final component of what “deployed” means.

This article was produced with AI assistance and reviewed by the editorial team.

Arjun Mehta

Arjun Mehta

Arjun Mehta covers AI infrastructure, semiconductor manufacturing, and the science-to-commercialization pipeline for Next Waves Insight. His reporting focuses on the technical and economic mechanics behind emerging technology — from chip fabrication constraints to the regulatory and funding pathways that determine which breakthroughs actually reach production. He has covered the technology sector for several years, with a focus on connecting primary research and hardware development to their downstream industry impact.

1 thought on “Enterprise AI Teams Can See Their Models Running. Most Cannot Tell Whether They Are Working.”

Leave a Comment