Why Cheap Inference Has Not Made Enterprise AI Profitable

6 min read

Key Claim: The 280-fold drop in inference costs since 2022 has not resolved enterprise AI economics because token price is a minor component of total cost — integration, evaluation, and the human oversight tax dominate the cost structure of production AI deployments.

The cost of running a frontier language model has collapsed 280-fold since late 2022. According to Stanford’s 2025 AI Index, querying a GPT-3.5-equivalent model cost $20 per million tokens in November 2022; by October 2024 the same capability cost $0.07. By April 2026, Grok-4.1 Fast prices input tokens at $0.20 per million. If AI were a commodity market, these numbers would signal a mature, competitive, and increasingly profitable industry. Enterprise P&L statements tell a different story.

The Token Price Collapse Is Real — and Mostly Irrelevant to Enterprise Economics

API prices have continued to fall sharply into 2026. Claude Opus 4.6 lists at $5.00 per million input tokens and $25.00 output; GPT-5.2 at $1.75 input and $14.00 output; Gemini 2.5 Pro at $1.25 input and $10.00 output, per IntuitionLabs’ current comparison. Industry-wide LLM prices dropped approximately 80% between 2025 and 2026 alone. These reductions are genuine and carry real implications for cost-sensitive use cases — but they address only one layer of the enterprise AI cost stack.

The deeper issue is that token costs are now the smallest component of total AI ownership for most deployed systems. Analysis from Xenoss breaks down enterprise AI TCO as follows: data engineering overhead adds 25–40% of total spend; model maintenance — drift detection, retraining cycles, version control — adds another 15–30%; integration with legacy systems carries a 2–3x implementation premium. The token API bill, after these multipliers are applied, often represents well under 20% of actual cost.

Manufacturing enterprises, per the same analysis, report AI ownership costs running 200–400% above initial vendor quotes. Eighty-five percent of organisations misestimate AI project costs by more than 10% before implementation begins.

GPU Utilisation: The Structural Drag Nobody Publishes

One figure that rarely appears in vendor press releases is actual GPU utilisation rate. Industry capacity planning benchmarks target 65–75% average utilisation with a 20–30% buffer reserved for demand spikes, per Introl’s 2025–2030 infrastructure planning analysis. In practice, the same analysis identifies 20–40% utilisation penalties at scale — meaning real-world enterprise workloads routinely run at 40–55% utilisation, not the 65%+ required for capital efficiency.

NVIDIA’s own orchestration data illustrates the gap: its Run:ai platform can achieve up to 2x GPU utilisation gains for enterprise inference workloads while cutting first-request latency by up to 61x versus cold-start deployments, per NVIDIA’s published announcement. The existence of a 2x improvement opportunity implies that the baseline — without orchestration — is operating at roughly half the theoretically available capacity. Cloud H100 rental runs $5,000–$75,000 per year per unit; at 40–50% utilisation, half of that spend is idle capacity.

Specialist inference hardware offers a partial remedy. On Llama 3.1 70B, Groq achieves 544 tokens per second at roughly $0.64 per million tokens; Cerebras delivers 445 tokens per second at approximately $0.60 per million tokens (16-bit precision), per IntuitionLabs’ chip comparison. These throughput figures are three to five times what standard GPU-based inference delivers, but they require architectural commitment and are unavailable for on-premises deployments.

Software-level optimisations close some of the gap. Combining speculative decoding with 4-bit weight quantization achieves a 2.78x speedup on Llama-3-70B on A100 GPUs versus standard serving, per May 2025 arXiv research. DistillSpec, which uses knowledge distillation to better align draft and target models before speculative decoding, yields an additional 10–45% speedup. Adobe’s deployment using NVIDIA’s Model Optimizer with quantization and TensorRT achieved a 40% reduction in total cost of ownership, per NVIDIA’s technical blog. These are meaningful gains — but they require engineering investment that itself adds to TCO.

The Providers Are Not Profitable Either

The enterprise AI profitability problem extends to the model providers themselves. OpenAI spent $9.4 billion on compute in 2025 while projecting losses of $74 billion by 2028 and no profitability until 2030, per Digitimes. Q3 2025 losses alone exceeded $11.5 billion. At the same time, Sequoia Capital partner David Cahn’s December 2025 update to his “$600 billion question” analysis concluded that the gap between infrastructure investment and end-user revenue had widened over the prior 18 months, per Preben Ormen’s summary analysis.

Anthropic presents a more structurally sound picture: the company targets a 40% gross profit margin, with approximately 80% of revenue from enterprise clients, and is projected to reach profitability by 2028, per Digitimes. Anthropic projected $4.1 billion in training costs for 2025, with a target of $2.10 in revenue per dollar of compute by 2028 — compared to OpenAI’s projected $1.60 — per Tanay Jaipuria’s revenue breakdown. But even Anthropic’s trajectory requires roughly two more years of subsidised growth before the books balance — which matters to enterprise buyers evaluating long-term vendor stability.

Total enterprise generative AI spend reached $37 billion in 2025, up 3.2x from $11.5 billion in 2024, per Menlo Ventures’ enterprise survey. Foundation model APIs alone consumed $12.5 billion of that figure. Despite this volume, only 5% of enterprises are seeing real, sustained returns at scale in 2026, per Master of Code’s ROI analysis. Most organisations achieving satisfactory returns do so within two to four years — three to four times the timeline of conventional technology deployments.

The Last-Mile Problem: Integration, Maintenance, and the Oversight Tax

The most consistently underestimated cost category is what might be called the oversight tax: the human labour required to supervise systems that are sold as autonomous. One documented case captured in Xenoss’s TCO analysis found a senior engineer spending 20 hours per month correcting AI agent errors and managing dependencies — equivalent to $8,000 per month in supervision cost for a nominally automated system. This is not an edge case; it reflects the current reliability profile of production agentic deployments.

Annual AI maintenance costs typically equal 15–30% of the original build cost. A system costing $100,000 to develop requires $15,000–$30,000 per year to maintain. Senior AI engineers with seven to ten years of experience command $300,000–$500,000 annually. These are not inference costs. They are fixed costs that do not fall when token prices fall. The a16z survey of 100 enterprise CIOs confirms this dynamic: AI spend has migrated from innovation budgets — which fell from 25% to just 7% of LLM spend — to permanent core IT line items, signalling that enterprises are internalising these costs as structural rather than experimental.

The compliance layer adds further pressure. EU AI Act violations carry penalties of up to EUR35 million or 7% of global turnover, per Xenoss’s analysis. GDPR-related AI incidents carry their own exposure. Governance infrastructure — auditing, monitoring, documentation — adds $100,000–$300,000 annually for mid-to-large deployments before a single token is processed.

What to Watch

The utilisation problem is the near-term lever. Closing the gap between theoretical GPU capacity (65–75% target) and actual enterprise utilisation (40–55% in practice) is the single most controllable cost variable. Intelligent orchestration — the Run:ai class of tools — and tighter workload scheduling are proven interventions with published performance gains.

Model routing will matter more than model selection. As the a16z CIO survey documents, “for most tasks all the models perform well enough — so pricing has become a much more important factor.” Routing cheap, fast models (such as Grok-4.1 Fast at $0.20/million input) to routine tasks while reserving frontier models for high-complexity requests is an architectural decision with directly measurable unit economics impact.

The oversight tax demands an accounting standard. No current enterprise AI cost model consistently captures human supervision hours as an operating cost. Until CFOs and procurement teams build this into budgets at the outset — rather than discovering it post-deployment — the gap between projected and actual AI TCO will persist.

Provider consolidation is a risk factor for enterprise buyers. OpenAI’s trajectory to profitability extends to 2030 on current projections. Enterprises signing multi-year agreements with providers carrying $74 billion in projected losses should treat vendor financial stability as a procurement variable, not an assumption.

The inflection point is conditional. Falling inference prices are necessary but not sufficient. The structural break-even requires: utilisation rates above 65%; inference optimisation (quantization, distillation) deployed as standard practice; human oversight costs attributed to AI systems accurately; and the integration premium shrinking as tooling matures. None of these are guaranteed to occur on the same timeline.

Further Reading

• Stanford HAI AI Index 2025 — cost per token historical data
• Menlo Ventures: 2025 State of Generative AI in the Enterprise — enterprise spending breakdown
• a16z: How 100 Enterprise CIOs Are Building and Buying Gen AI in 2025 — budget and model selection data
• Xenoss: Total Cost of Ownership for Enterprise AI — TCO component breakdown
• arXiv: Speculative Decoding Meets Quantization (May 2025) — inference optimisation benchmarks

This article was produced with AI assistance and reviewed by the editorial team.