Who it's for
Engineering leaders asked to promise a speed-up from AI.
You'll be able to
Tell a rigorous study from a vendor claim, and say what the evidence supports.
More pull requests is not the same as more value. Most headline numbers count the first and skip the second.
The randomized studies
The only randomized evidence so far comes from METR, which studied experienced open-source developers.
In early 2025, tasks done with AI took 19 percent longer. The developers believed they had been faster.
In an update published in February 2026, returning developers were about 18 percent faster, and new recruits about 4 percent faster. Both ranges include zero. METR itself calls this "only very weak evidence" for the size of any gain. Part of the reason: 30 to 50 percent of developers held back tasks they did not want to do without AI.
The large telemetry studies
Bigger studies watch real teams instead of running trials. They show clear gains in output.
| Source | Kind | What it found |
|---|---|---|
| Microsoft Research, 2026 | Observational | About 24 percent more merged pull requests per engineer. Quality was not measured |
| DX, 2026 | Vendor | Throughput up 37 percent over a year. Change confidence down 6.1 percent |
| Cortex, 2026 | Vendor | Pull requests per author up 20 percent. Incidents per pull request up 23.5 percent |
| LinearB, 2026 | Vendor | Agent pull requests merge less often: 79 percent in the best organizations, against 92 percent for human ones |
None of these are trials. People who use AI heavily may also be the people who ship most, and that makes the AI effect look bigger.
Output went up. Whether outcomes went up, nobody has measured well.
Where they split, and where they agree
The claims split in three ways. Telemetry studies report 20 to 37 percent throughput gains, while the trials show zero to 18 percent and not significant. Vendors count pull requests, while METR measures time to finish a task. And the same vendor reports that show more output also show stability or confidence falling.
DORA's 2025 report found the same pattern: AI goes with higher throughput and lower stability. Its 2026 report on return on investment describes a J-curve. Returns start negative after rollout, and a positive return "is not a given".
On one point DORA, Cortex, DX and LinearB agree. AI amplifies the practice you already have. Gains show up where testing, review and platform foundations already exist.
Try it
Before you roll out more AI, write down three numbers for the last quarter: pull requests merged, how often a change failed in production, and how long a pull request waited for its first review. Check them again a quarter later. If the first rises and the others get worse, you have more output, not more value.