Why Cheap Inference Has Not Made Enterprise AI Profitable

The headline unit economics of running a frontier language model have collapsed 280-fold since late 2022. According to Stanford’s 2025 AI Index, querying a GPT-3.5-equivalent model cost $20 per million tokens in November 2022; by October 2024 the same capability cost $0.07. By April 2026, Grok-4.1 Fast prices input tokens at $0.20 per million. If AI were a commodity market, these numbers would signal a mature, competitive, and increasingly profitable industry. Enterprise P&L statements tell a different story about unit economics.

The Token Price Collapse Is Real — and Mostly Irrelevant to Enterprise Economics

API prices have continued to fall sharply into 2026. Claude Opus 4.6 lists at $5.00 per million input tokens and $25.00 output; GPT-5.2 at $1.75 input and $14.00 output; Gemini 2.5 Pro at $1.25 input and $10.00 output, per IntuitionLabs’ current comparison. Industry-wide LLM prices dropped approximately 80% between 2025 and 2026 alone. These reductions are genuine and carry real implications for cost-sensitive use cases — but they address only one layer of the enterprise AI cost stack.

The deeper issue is that token costs are now the smallest component of total AI ownership for most deployed systems, skewing assumed unit economics. Analysis from Xenoss breaks down enterprise AI TCO as follows: data engineering overhead adds 25–40% of total spend; model maintenance — drift detection, retraining cycles, version control — adds another 15–30%; integration with legacy systems carries a 2–3x implementation premium. The token API bill, after these multipliers are applied, often represents well under 20% of actual cost.

Manufacturing enterprises, per the same analysis, report AI ownership costs running 200–400% above initial vendor quotes, disrupting projected unit economics. Eighty-five percent of organisations misestimate AI project costs by more than 10% before implementation begins.

GPU Utilisation: The Structural Drag on Unit Economics

One figure that rarely appears in vendor press releases is actual GPU utilisation rate, which fundamentally dictates unit economics. Industry capacity planning benchmarks target 65–75% average utilisation with a 20–30% buffer reserved for demand spikes, per Introl’s 2025–2030 infrastructure planning analysis. In practice, the same analysis identifies 20–40% utilisation penalties at scale — meaning real-world enterprise workloads routinely run at 40–55% utilisation, not the 65%+ required for capital efficiency.

NVIDIA’s own orchestration data illustrates the gap: its Run:ai platform can achieve up to 2x GPU utilisation gains for enterprise inference workloads while cutting first-request latency by up to 61x versus cold-start deployments, per NVIDIA’s published announcement. The existence of a 2x improvement opportunity implies that the baseline — without orchestration — is operating at roughly half the theoretically available capacity. Cloud H100 rental runs $5,000–$75,000 per year per unit; at 40–50% utilisation, half of that spend is idle capacity.

Specialist inference hardware offers a partial remedy. On Llama 3.1 70B, Groq achieves 544 tokens per second at roughly $0.64 per million tokens; Cerebras delivers 445 tokens per second at approximately $0.60 per million tokens (16-bit precision), per IntuitionLabs’ chip comparison. These throughput figures are three to five times what standard GPU-based inference delivers, but they require architectural commitment and are unavailable for on-premises deployments.

Software-level optimisations close some of the gap in inference unit economics. Combining speculative decoding with 4-bit weight quantization achieves a 2.78x speedup on Llama-3-70B on A100 GPUs versus standard serving, per May 2025 arXiv research. DistillSpec, which uses knowledge distillation to better align draft and target models before speculative decoding, yields an additional 10–45% speedup. Adobe’s deployment using NVIDIA’s Model Optimizer with quantization and TensorRT achieved a 40% reduction in total cost of ownership, per NVIDIA’s technical blog. These are meaningful gains — but they require engineering investment that itself adds to TCO.

The Providers Are Not Profitable Either

The strained unit economics of enterprise AI extend to the model providers themselves. OpenAI spent $9.4 billion on compute in 2025 while projecting losses of $74 billion by 2028 and no profitability until 2030, per Digitimes. Q3 2025 losses alone exceeded $11.5 billion. At the same time, Sequoia Capital partner David Cahn’s December 2025 update to his “$600 billion question” analysis concluded that the gap between infrastructure investment and end-user revenue had widened over the prior 18 months, per Preben Ormen’s summary analysis.

Anthropic presents a more structurally sound picture: the company targets a 40% gross profit margin, with approximately 80% of revenue from enterprise clients, and is projected to reach profitability by 2028, per Digitimes. Anthropic projected $4.1 billion in training costs for 2025, with a target of $2.10 in revenue per dollar of compute by 2028 — compared to OpenAI’s projected $1.60 — per Tanay Jaipuria’s revenue breakdown. But even Anthropic’s trajectory requires roughly two more years of subsidised growth before the books balance — which matters to enterprise buyers evaluating long-term vendor stability.

Total enterprise generative AI spend reached $37 billion in 2025, up 3.2x from $11.5 billion in 2024, per Menlo Ventures’ enterprise survey. Foundation model APIs alone consumed $12.5 billion of that figure. Despite this volume, only 5% of enterprises are seeing real, sustained returns at scale in 2026, per Master of Code’s ROI analysis. Most organisations achieving satisfactory returns do so within two to four years — three to four times the timeline of conventional technology deployments.

The Last-Mile Problem: Integration, Maintenance, and the Oversight Tax

In evaluating true unit economics, the most consistently underestimated cost category is what might be called the oversight tax: the human labour required to supervise systems that are sold as autonomous. One documented case captured in Xenoss’s TCO analysis found a senior engineer spending 20 hours per month correcting AI agent errors and managing dependencies — equivalent to $8,000 per month in supervision cost for a nominally automated system. This is not an edge case; it reflects the current reliability profile of production agentic deployments.

Annual AI maintenance costs typically equal 15–30% of the original build cost, degrading long-term unit economics. A system costing $100,000 to develop requires $15,000–$30,000 per year to maintain. Senior AI engineers with seven to ten years of experience command $300,000–$500,000 annually. These are not inference costs. They are fixed costs that do not fall when token prices fall. The a16z survey of 100 enterprise CIOs confirms this dynamic: AI spend has migrated from innovation budgets — which fell from 25% to just 7% of LLM spend — to permanent core IT line items, signalling that enterprises are internalising these costs as structural rather than experimental.

The compliance layer adds further pressure. EU AI Act violations carry penalties of up to EUR35 million or 7% of global turnover, per Xenoss’s analysis. GDPR-related AI incidents carry their own exposure. Governance infrastructure — auditing, monitoring, documentation — adds $100,000–$300,000 annually for mid-to-large deployments before a single token is processed.

What to Watch

The utilisation problem is the near-term lever. Closing the gap between theoretical GPU capacity (65–75% target) and actual enterprise utilisation (40–55% in practice) is the single most controllable cost variable. Intelligent orchestration — the Run:ai class of tools — and tighter workload scheduling are proven interventions with published performance gains.

Model routing will matter more than model selection. As the a16z CIO survey documents, “for most tasks all the models perform well enough — so pricing has become a much more important factor.” Routing cheap, fast models (such as Grok-4.1 Fast at $0.20/million input) to routine tasks while reserving frontier models for high-complexity requests is an architectural decision with directly measurable unit economics impact.

The oversight tax demands an accounting standard. No current enterprise AI cost model consistently captures human supervision hours as an operating cost. Until CFOs and procurement teams build this into budgets at the outset — rather than discovering it post-deployment — the gap between projected and actual AI TCO will persist.

Provider consolidation is a risk factor for enterprise buyers. OpenAI’s trajectory to profitability extends to 2030 on current projections. Enterprises signing multi-year agreements with providers carrying $74 billion in projected losses should treat vendor financial stability as a procurement variable, not an assumption.

The inflection point is conditional. Falling inference prices are necessary but not sufficient to achieve sustainable unit economics across enterprise deployments. The structural break-even requires: utilisation rates above 65%; inference optimisation (quantization, distillation) deployed as standard practice; human oversight costs attributed to AI systems accurately; and the integration premium shrinking as tooling matures. None of these are guaranteed to occur on the same timeline.

Further Reading

This article was produced with AI assistance and reviewed by the editorial team.

Sarah Chen

Sarah Chen

Sarah Chen covers the economics of the technology industry for Next Waves Insight — venture funding, enterprise AI adoption, and the business models emerging around frontier technology. Her work traces capital flows and corporate strategy to explain not just what companies are building, but why, and what it signals about where the industry is heading. She has a background in technology and financial journalism.

3 thoughts on “Why Cheap Inference Has Not Made Enterprise AI Profitable”

Leave a Comment