In early 2025 a research group had sixteen experienced open-source developers work on their own projects — projects they had been building on for five years on average. 246 tasks were randomly assigned to two conditions: with an AI tool and without. Beforehand, the participants estimated that AI would cut their completion time by 24 per cent. Afterwards, the same people said it had been about 20 per cent. What was measured was a 19 per cent increase in completion time.
The interesting number in that study is not the slowdown. It is the distance: some forty percentage points between what people say about their own work and what the stopwatch records. And that distance is not a one-off. It is the actual subject.
I write this as someone who introduces AI into organisations and is regularly asked how much it delivers. The honest answer is: it depends who you ask and what you measure — and those two answers come apart reliably.
The finding that did not hold
An article citing the 19 per cent and stopping there would be incomplete in 2026. In February 2026 the same research group discarded its study design. The reason is as mundane as it is instructive: participants had become hard to find. In their own words, they had observed “a significant increase in developers choosing not to participate in the study because they do not wish to work without AI”. Thirty to fifty per cent of developers said they were withholding certain tasks because they did not want to do them without AI.
Lose from a sample precisely those tasks where AI helps most, and you understate its effect systematically. The group's new, preliminary estimates are minus eighteen per cent for returning and minus four per cent for newly recruited participants — both with confidence intervals that include zero. The researchers themselves call their data “only very weak evidence”.
That is the kind of self-correction one would like to see more often in corporate presentations. And it shifts the question: if even the most careful measurement starts to slip once participants refuse to give up the tool, what does that mean for a number on a slide in a steering committee?
The gap runs both ways
It would be convenient to build the opposite thesis from this: people overestimate the value of AI. That is not true either. In a controlled experiment where developers were asked to implement an HTTP server in JavaScript, the group with an assistant tool took 71 minutes on average against 161 in the control group — a saving of 55.8 per cent. Asked how much faster they felt, participants said 35 per cent on average. Here the self-assessment lay below the measured effect.
Also worth noting is the width of the confidence interval in that work: 21 to 89 per cent. A point estimate of “55.8 per cent faster” sounds more precise than it is. Anyone passing such numbers on should pass the range on too.
So the gap between perception and measurement runs in both directions. What makes it reliable is not its sign but its existence. Self-report is no substitute for measurement — and measurement is no substitute for the conversation about why something feels different from how it is.
Where the gains are solid
The methodologically strongest work on the subject comes from Erik Brynjolfsson, Danielle Li and Lindsey Raymond and appeared in the Quarterly Journal of Economics in 2025. It examined the staggered rollout of a generative assistant to 5,172 customer support staff. The result: issues resolved per hour rose by 15 per cent on average — with substantial heterogeneity.
That heterogeneity is the real finding. Inexperienced and lower-skilled workers gained around 30 per cent, and 36 per cent in the lowest skill quintile. The most experienced workers gained almost no speed — and lost slightly on quality. New staff reached a level with AI after two months that took more than six months without it.
A pre-registered study by Shakked Noy and Whitney Zhang, published in Science in 2023, shows a similar pattern: 453 working graduates completed occupation-typical writing tasks, half of them with access to a language model. Completion time fell by 40 per cent, rated quality rose by 18 per cent — and differences in performance between participants narrowed rather than widened.
Both point the same way: AI mainly lifts the lower end. For organisations this is uncomfortable news, because it inverts the common expectation. The value arises less often where the strongest people become stronger, and more often where onboarding, standard cases and routine sit.
The jagged edge
Why do results differ so much? One answer comes from a pre-registered field experiment with 758 knowledge workers presented by Fabrizio Dell'Acqua and colleagues in 2023. Participants worked on eighteen realistic tasks, some with and some without a language model.
Inside what the authors call the jagged technological frontier, participants using AI completed 12.2 per cent more tasks, were 25.1 per cent faster and delivered significantly better quality. On a management task deliberately chosen to sit outside that frontier, the picture reversed: those using AI arrived at the correct solution less often.
The way this effect is worded shifted between working paper and publication — the working paper says 19 percentage points, the published version “19% less likely”. Those are not the same statistically, and both numbers circulate interchangeably in the debate. For practice, the point estimate matters less than the shape of the finding: the frontier is jagged, and you cannot see where it runs. Tasks that look similarly hard to a human sit on different sides of the line for a model.
What organisations feel — and what they measure
At the level of the organisation the pattern repeats on a larger scale. In McKinsey's most recent survey of 1,719 respondents across 97 countries, eight in ten say AI has improved their personal productivity. In the same dataset, only 37 per cent attribute any effect on earnings to AI — about the same share as the year before. Six per cent qualify as high performers with demonstrable bottom-line impact.
The DORA report 2025, which surveyed close to five thousand technology professionals, arrives at a related picture: ninety per cent use AI at work, more than eighty per cent report increased individual productivity — and at the same time only 24 per cent have a lot of trust in what the systems produce. Delivery instability, the rate of failed changes, remains stubbornly high.
One can read that as a contradiction. More likely it is none. Individual acceleration and system performance are different quantities. If everyone produces suggestions faster that then have to be reviewed, corrected and integrated, the sum can slow down while every individual feels faster. That is precisely the experience of organisations without clear handover points.
A number that moved markets
In the summer of 2025 one figure travelled the world: 95 per cent of AI projects were failing. It comes from a report by an initiative at the MIT Media Lab. Read the source and you find a different sentence: of the organisations surveyed, 95 per cent were getting zero return — which is not the same as a failed project. The basis was 52 interviews, 153 executive questionnaires and a review of publicly documented initiatives over six months. No peer review. The bar for success was a “marked and sustained” effect within six months — a standard an ERP system would also fail.
Within a day, technology stocks fell noticeably. A non-peer-reviewed figure with a sample of 153 had done more work than every controlled study in the field combined.
I do not cite this to dispose of the report. I cite it because it makes the point of this article: in the debate about AI productivity, it is not the most solid number that prevails but the most quotable one. Anyone carrying responsibility in an organisation has to make that distinction themselves — nobody else does it for them.
What follows for leadership
First: do not ask whether AI makes people more productive. The question cannot be answered. Ask for which task, for whom, and observable how. The evidence is clear for standard cases and onboarding; it is not clear for expert work on familiar material.
Second: separate perception from measurement — and collect both. Satisfaction is a real quantity; it keeps people engaged. It is simply not evidence of results. Ask only about feeling and you will reliably receive good news.
Third: measure at system level. Individual acceleration is easy to produce and easy to report. Whether lead time, error rate and rework have changed only shows at the end of the chain.
Fourth: decide in advance what you would have to see in order to believe it works. Ask that question after the pilot and you will always find a number that fits.
None of this is about distrusting the value of AI. It is demonstrated in several clean studies. It is about the size of that value being an empirical question — and about organisations currently answering it mostly with self-report. Answering it differently is an advantage no tool provides.
Note on AI use: research and drafting of this article were AI-assisted. All figures were checked against the primary sources; where working paper and publication differ, this is noted in the text. The selection, the interpretation and the conclusions are mine.
Sources
- Becker, J. / Rush, N. / Barnes, E. / Rein, D.: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, arXiv:2507.09089, July 2025. arxiv.org/abs/2507.09089
- METR: We are Changing our Developer Productivity Experiment Design, 24 February 2026 — selection effects and revised estimates. metr.org
- Brynjolfsson, E. / Li, D. / Raymond, L.: Generative AI at Work. In: The Quarterly Journal of Economics, vol. 140 (2025), no. 2, pp. 889–942. doi.org/10.1093/qje/qjae044
- Noy, S. / Zhang, W.: Experimental evidence on the productivity effects of generative artificial intelligence. In: Science, vol. 381 (2023). doi.org/10.1126/science.adh2586
- Dell'Acqua, F. / McFowland III, E. / Mollick, E. et al.: Navigating the Jagged Technological Frontier. Harvard Business School Working Paper 24-013, September 2023; published in Organization Science, DOI 10.1287/orsc.2025.21838. ssrn.com
- Peng, S. / Kalliamvakou, E. / Cihon, P. / Demirer, M.: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590, February 2023. arxiv.org/abs/2302.06590
- DORA / Google Cloud: State of AI-assisted Software Development 2025. Fieldwork June/July 2025, close to 5,000 respondents. dora.dev
- McKinsey & Company: The State of AI in 2026: On the Road to ROI, August 2026. 1,719 respondents across 97 countries. mckinsey.com (PDF)
- Challapally, A. / Pease, C. / Raskar, R.: The GenAI Divide: State of AI in Business 2025. MIT NANDA, July 2025 — not peer reviewed; 52 interviews, 153 questionnaires.
This article offers a professional assessment and does not replace advice on an individual case.
