Does AI make developers faster? What the controlled studies say

Every AI transformation plan I review has a productivity number in it: 20%, 30%, sometimes 55%. It usually comes from a vendor deck, and it usually carries the whole business case. So it is worth asking what the controlled studies, the ones that compare developers with and without AI on purpose, actually found.

The short answer: it depends on the task, the developer and the year. On a small task written from scratch, developers with an AI assistant were more than twice as fast. Experienced maintainers working in their own large codebases were slower with AI in early 2025, and roughly even to modestly faster a year later. Large field experiments inside companies found about a quarter more completed work, with the largest gains among less experienced developers. None of these studies supports a single percentage for your organisation.

The studies at a glance

Small task from scratch: more than twice as fast

The most quoted number comes from the first controlled study of GitHub Copilot (Peng et al. 2023). 95 developers recruited on Upwork were asked to write an HTTP server in JavaScript as fast as they could. The group with Copilot finished in 71 minutes on average, the group without in 161: 55.8% less time, with a 95% interval of 21% to 89%.

What the number does not say matters as much. It is one standardised task, written from scratch, with no existing codebase to understand. The difference in how many developers completed the task successfully was not statistically significant, and code quality was not measured. Less experienced developers and those who code more hours a day gained the most.

Inside companies: about a quarter more work, most of all for juniors

The largest evidence comes from field experiments run by companies as part of normal work (Cui et al. 2025, published in Management Science). At Microsoft, Accenture and an anonymous Fortune 100 company, a random group of developers got GitHub Copilot and the rest did not, for months. Pooled across 4,867 developers, the authors estimate 26% more completed tasks with the tool, with a standard error of 10.3 percentage points.

Two details are easy to miss. Each experiment on its own was noisy; only the pooled estimate is clear. And less experienced developers both used the tool more and gained more, which matters if your organisation is mostly senior.

Google's own trial (Paradis et al. 2024) sits between the two: 96 engineers on one realistic, enterprise-grade task were about 21% faster with AI features, with what the authors call a large confidence interval. Engineers who spend more hours a day coding were faster. The authors warn against assuming the effect carries over to other tools, tasks or time periods.

Experienced maintainers: slower, then roughly even

The study that changed the conversation came from METR in July 2025 (METR 2025). Sixteen experienced open-source developers worked on 246 real issues in repositories they had contributed to for years, projects averaging over 22,000 stars and a million lines of code. Each issue was randomly assigned to allow or forbid AI. With AI allowed, issues took 19% longer.

The perception gap is the part every leader should see:

METR repeated the experiment from August 2025 with 57 developers and over 800 tasks (METR 2026). Returning developers were 18% faster and new ones 4% faster, but both intervals include zero. METR itself calls this weak evidence: developers increasingly refused to work without AI, 30 to 50% of them said they held back tasks they did not want to do without it, and time tracking broke down for people running several agents at once. METR's reading is that AI tools probably help these developers more in 2026 than in 2025, without a reliable number for how much.

All the time effects on one scale

Why the results disagree

• The task. Writing something small from scratch is where AI shines. Changing a large, mature codebase that you know better than any model does is where it struggled.
• The developer. Less experienced developers gained most in the Copilot and field studies. METR's participants were among the most experienced developers in their projects.
• The tools and the year. Peng tested 2022 code completion; METR tested early-2025 agents; the 2026 follow-up already struggled to measure people running several agents in parallel. Results age quickly.
• What is measured. Minutes on one task, tasks completed over months, or developers' own estimates. Only METR checked perception against measurement, and they diverged by about 40 points.
• What is not measured. Most studies do not measure quality, review load or defects after release. Faster code that takes longer to review or breaks more often is not a productivity gain.

How much weight each study deserves

What this means for a transformation plan

1. Do not put a single productivity percentage in the business case. Use a range, and say which kind of work it applies to. The evidence supports a range from no gain to large gains depending on task and experience.
2. Never measure with surveys alone. In METR's study, developers' beliefs and their measured time pointed in opposite directions.
3. Run your own controlled rollout. Give access to a random half of teams first, as Microsoft and Accenture did, and compare. A staggered rollout costs little and settles arguments.
4. Measure outcomes, not activity. Lead time, change failure rate, review time and defects after release, by team and by type of work, alongside throughput.
5. Plan for where the gains are. Expect more from new code, unfamiliar codebases and less experienced engineers; expect less, at first, from senior engineers deep in a system they know well.

Code migrations are one place where the gains are well documented, under the right conditions: see AI code migrations: what Airbnb, Google and Amazon actually reported. To turn a range like this into a funding decision, the structure in A GraphRAG business case, done the evidence-based way works for any AI investment.

Beyond coding, the same rule holds for any agent: measure reliability, not the demo; see Agentic AI from demo to production. For the governance side of an AI programme, see how the EU AI Act, ISO 42001 and the NIST AI RMF fit together, and for the Act's dates, EU AI Act dates and guidance, tracked. Company-level claims need the same scrutiny as productivity numbers: see 95% of AI pilots fail: what the MIT report actually measured and the case of Klarna's AI customer service.

Sources

• Peng, Kalliamvakou, Cihon and Demirer (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590.
• Paradis et al. (2024). How much does AI impact development speed? An enterprise-based randomized controlled trial. arXiv:2410.12944.
• Cui, Demirer, Jaffe, Musolff, Peng and Salz (2025). The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science.
• METR (July 2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.
• METR (February 2026). We are changing our developer productivity experiment design.