See also:

Before performing the study, the developers in question expected the AI tools would lead to a 24 percent reduction in the time needed for their assigned tasks. Even after completing those tasks, the developers believed that the AI tools had made them 20 percent faster, on average. In reality, though, the AI-aided tasks ended up being completed 19 percent slower than those completed without AI tools.

By analyzing screen recording data from a subset of the studied developers, the METR researchers found that AI tools tended to reduce the average time those developers spent actively coding, testing/debugging, or “reading/searching for information.” But those time savings were overwhelmed in the end by “time reviewing AI outputs, prompting AI systems, and waiting for AI generations,” as well as “idle/overhead time” where the screen recordings show no activity.

Overall, the developers in the study accepted less than 44 percent of the code generated by AI without modification. A majority of the developers reported needing to make changes to the code generated by their AI companion, and a total of 9 percent of the total task time in the “AI-assisted” portion of the study was taken up by this kind of review.

[…]

On the surface, METR’s results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency.

Many of the existing coding benchmarks also focus on synthetic, algorithmically scorable tasks created specifically for the benchmark test, making it hard to compare those results to those focused on work with pre-existing, real-world code bases. Along those lines, the developers in METR’s study reported in surveys that the overall complexity of the repos they work with (which average 10 years of age and over 1 million lines of code) limited how helpful the AI could be. The AI wasn’t able to utilize “important tacit knowledge or context” about the codebase, the researchers note, while the “high developer familiarity with [the] repositories” aided their very human coding efficiency in these tasks.

Study finds AI tools made open source software developers 19 percent slower