We are Changing our Developer Productivity Experiment Design
metr.org
Our second developer productivity study faces selection effects from wider AI adoption, prompting us to redesign our approach.

Selected developer quotes:

“I’m torn. I’d like to help provide updated data on this question but also I really like using AI!” — a developer from the original study early-2025 when asked to participate in the late-2025 study.

“I found I am actually heavily biased sampling the issues … I avoid issues like AI can finish things in just 2 hours, but I have to spend 20 hours. I will feel so painful if the task is decided as AI-disallowed.” — a developer from the new study noting selection effects when choosing what tasks to include in the study.

“my head’s going to explode if I try to do too much the old fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.” — a developer from the new study noting selection effects when choosing what tasks to include in the study.

  • maegul (he/they)@lemmy.mlEnglish
    20·
    2 days ago

    I mean, it makes sense that it’s addictive, right?

    I also suspect it’s one of those things that just naturally splits people. For some, the addictiveness and appeal just don’t make sense. For others it’s irresistible.

    It’s part of the reason why I’m so doomer on the state of things, from a generally anti-AI/sceptical perspective. There’s just something compulsive that this kind of tool triggers in many people.

    • Pup Biru@aussie.zoneEnglish
      10·
      1 day ago

      the other insidious part about it is that pretty much everyone agrees: experience is critical to ensuring AI isn’t just producing slop… 30% of the time you get something that’s not just working, but well architected

      now when you get AI to do things, even if you go with the assertion that it’s quicker (which a lot of the time i doubt: task choice is also critical for effectively using AI to generate useful outputs), you’re grinding down on your experience… not only are you learning less, but you’re also letting your reasoning skills degrade because you’re not using them (this is a pretty well-documented effect in standard neuroscience afaik)

      imo, only use AI in situations where you’d put a library in, because the level of abstraction from the problem solving is similar