AI Coding Assistants: What the Controlled Studies Actually Show

4 min read

The productivity numbers quoted for AI coding assistants vary so dramatically — 20% to 10x depending on the source — that they have become nearly impossible to evaluate. The signal is real. So is the noise. In 2025 and early 2026, a clearer picture has started to emerge from controlled studies, adoption data, and post-deployment audits. The headline numbers are accurate in narrow contexts. The contexts matter enormously.

What the Controlled Studies Actually Measured

GitHub’s foundational Copilot study, conducted in 2022 and replicated with larger samples in 2024, showed developers completing defined coding tasks 55% faster with Copilot enabled versus a control group. The study’s methodology is rigorous — randomised assignment, blinded evaluation, controlled task scope. It is also deliberately narrow: participants were given isolated implementation tasks with clear specifications. Debugging existing codebases, designing system architecture, and reviewing others’ code were excluded.

METR’s 2025 autonomous coding evaluation takes a different approach. Rather than measuring speed on defined tasks, it measures the percentage of real-world software engineering tickets that AI agents can complete end-to-end without human intervention. In 2023, that figure was effectively zero for any non-trivial task. By early 2025, leading agents (Claude, GPT-4o with tools, and purpose-built systems like SWE-agent) completed approximately 50% of the evaluated tasks autonomously. The remaining 50% — characterised by ambiguous specifications, cross-system dependencies, and novel debugging requirements — remain resistant.

Where the Productivity Gains Are Real

The strongest evidence for productivity gains concentrates in four areas: boilerplate generation, test writing, documentation, and translation between languages or frameworks. These tasks are time-consuming, cognitively low-intensity, and well-represented in training data. A senior engineer who previously spent 40% of their time on boilerplate and documentation can realistically reclaim a significant fraction of that time.

The weakest evidence is in tasks that require deep contextual understanding of a specific codebase — debugging subtle interactions between components, identifying architectural tradeoffs in systems the AI has not seen, and making decisions that require understanding business context alongside technical constraints. These are also the tasks that consume the most senior engineering time and carry the highest value. The productivity gains from AI coding assistants are, in aggregate, most dramatic where the stakes are lowest.

The Code Review Problem

A pattern emerging in engineering teams that have deployed AI coding assistants at scale: code review workload is increasing, not decreasing. When developers use AI to generate implementation code, they tend to produce more code in a given sprint. That code still requires review. It also tends to require more careful review — AI-generated code can be syntactically correct and logically flawed in ways that are not immediately obvious.

Microsoft’s internal data, shared at Build 2025, indicated that Copilot suggestion acceptance rates average around 30% across enterprise users. This means developers are evaluating and rejecting roughly 70% of suggestions — a real cognitive load that does not appear in time-to-completion metrics. Engineering managers building productivity models based solely on completion speed are likely overestimating net throughput gains by a material margin.

The Junior-Senior Gap

Experience level significantly modulates the productivity impact of AI coding tools. Junior developers — those with 0–3 years of experience — consistently show the largest percentage productivity improvements in controlled studies. AI assistance compensates for gaps in pattern recognition and reduces the lookup overhead that dominates junior developer time. The same tools used by senior engineers show smaller, and in some cases negative, effects on complex tasks, where the AI’s tendency to generate confident but incorrect solutions creates additional verification burden.

This creates a calibration problem for engineering leaders. Team-level productivity metrics that average across seniority levels will show strong gains — but the gains may be concentrated in work that was already the cheapest to perform, while the most expensive and valuable senior engineering time sees marginal or no improvement.

What the Next 12 Months Looks Like

The autonomous coding agent category is moving faster than the assistant category. Systems that can execute multi-step software engineering tasks — cloning a repository, understanding the codebase, implementing a feature, writing tests, and opening a pull request — have gone from experimental to production-viable in the span of 18 months. Devin, SWE-agent, and Claude’s native computer-use capabilities are all targeting the same surface area: the software engineering ticket queue.

For engineering leaders, the practical implication is a distinction that matters: AI coding assistants augment individual developer throughput. AI coding agents potentially change headcount models. Both are in play simultaneously, and most organisations are staffed and structured for neither.