studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software
LLMs can’t tell the difference between code and comments in some cases. Older code bases that have had a number of hands touching it over the years have a lot of commented out code and dead methods. LLM sees that as good live code. Also there will be comments like ‘this is a stupid way to do this, but I don’t have time to fix it right now’. LLM has no sense of humor in this case.
And sure. I should clean all this up before hand, but who has time for that? Project has one, part time developer working on it now.I could’ve told them this, tbh.
Anyone can tell anyone anything, it doesn’t make it true or validated.
don’t forget, even a broken clock is right twice a day 😄
LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training (“dominant priors”), can often “outweigh” information we give it
LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want.
Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be “driving in fog”, with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the “big picture” – probabilistically, it’s a blur.
I’d guess that what all this stuff adds up to is, sustainable use of LLMs as a coding tool for nontrivial projects calls for an entirely reworked set of software development practices to conform to its limitations effectively, but the people in charge really really want and believe it to be a drop-in efficiency boost, and a big mess results. This reminds me a lot of the articles and arguments I’ve read over the years about low level vs high level programming languages and frameworks. Probably will play out a similar way.
A lot of tokens tends to help move the needle, building in guardrails, sophisticated review of both in process work & final outputs, and consensus helps raise the bar considerably.
That and sane, universally consistent, well structured, low tech debt, organizationally elegant, systematically evolved codebases. Something the grand majority of teams don’t have to begin with.
Nevermind being well documented, with excellent opinionated linting and static type enforcement configurations, with robust CI checks. More things many projects don’t seem to have.
Turns out that giving AI poorly engineered and maintained codebases just amplifies the poor engineering and low rigor already present. Something that’s endemic to our industry.
There are some cardinal sins within the annals of software development in the workplace. The relevant two here are:
- Do not build a scoreboard for productivity
- Do not equate keystrokes with effort or value rendered
These create perverse incentives that can really screw everything up, including creating permanent damage to company culture, products, and productivity. Anytime you see these things done, it’s because you have (or are) lazy-ass management.
Edit: I also just learned that this is an application of Goodhart’s Law.
Oh god a former company had the burndown chart on a big plasma screen at all times
output is up (more code, more commits, bigger diffs)
We’ve known measuring output by lines of code is counterproductive for a long time.
Has management known that?
A lot of problems seem to be downstream from “management are idiots and assholes”
At any minimally competent company they are aware.
However my company mostly has former engineers as engineering managers.
I’m sure there were pockets but it legitimately seemed like that dragon had been slain until recently. Trying to assign more meaning to scrum points has been in vogue for a while though.
My team assigns both hours and points to tasks. I’ve never seen the points used for anything but they still spend time on it.
That’s… an interesting approach.
That battle is perpetual. Lazy management sees a number that resembles a statistic, and try to use it as an easy metric for stuff it doesn’t represent. Points do aggregate into velocity, which is worth measuring. But on their own, points are a proxy for estimation in $SPRINT_LENGTH days. The way to manage up is to keep making it clear that the smallest unit of estimation in Agile is the sprint length; points are used to subdivide that but only to ensure that the sprint itself is not overloaded and thus an accurate estimate.
(story points)
Can you retell my company that?
For a small exorbitant consulting fee that legitimizes me, I can tell them that.
One study found a significant correlation between confidence in AI output and belief in the paranormal.
💀 💀 💀
That’s HILLARIOUS
Lol that’s funny. My belief in having a divinely created soul is exactly why I think humans can’t be replaced by these supercharged drunken parrots. :O
…But yeah this is probably referring to the ones who, under the right conditions, will start to sense “ghosts in the machine” and go down the rabbit hole to chatbot psychosis…
I am 100% genuinely not trying to start a fight, this a legitimate question to which I really would like to know the answer.
What is the difference between a divine creation and the "ghost in the machine "?
I will give context , faith is genuinely confusing to me.
I have no issue with personal faiths unless you are trying to force everyone to have the same faith as you.
I consider anything above small scale organisation of religion to be the worst thing that can happen to faith, because people are people and power corrupts.
I suppose my real question is , for a concept that prohibits proof as part of it’s definition, how do you determine that one faith (or system) is better than another?
I’m not the person you responded to, but I can try to take a stab at that…
for a concept that prohibits proof as part of it’s definition, how do you determine that one faith (or system) is better than another?
Objectively, you don’t. But people’s experiences go beyond the objective, and we all have our subjective experiences too. Faith is how we make sense of those.
The question then isn’t, which one is better, but whose other experiences and faiths do we relate to? We build friendships and alliances based on those. Not because they’re better, per se, but because we feel comfortable or safe with them.
From that perspective it makes a lot of sense that we value other humans, because they can make us feel understood and appreciated, and we can exchange ideas with them which in turn refines our faith. Now, some people get tricked into thinking you can do the same with machines, but well, I think you can guess where I too stand on that idea.
That was a very insightful answer. Well said! Thank you very much for replying. :)
I will have to contemplate a little bit, and respond to the question myself as well.
This doesn’t seem to cover there is also no LLM that doesn’t plagiarize, or where the training data appears to be compatible with such behavior (e.g. CC0). Now I don’t know what that means legally, but morally it seems to be tossing away other project’s licensing and I think for FOSS as a whole that’s no good.
Also something worth reiterating: https://machinelearning.apple.com/research/illusion-of-thinking LLMs apparently can’t do basic logical reasoning. Even a junior coder can do that. I’m always surprised anybody would let LLMs near their code, at all.
Of course it didnt cover that. This was an article on the efficacy of AI in writing software, not a treatise on ethical or legal concerns. I would completely lose trust in the author if they started piling on every reason why “AI Bad”, because it would be clear that they have an agenda
I recommed you reading this
It summarizes really good not only the moral, but also the legal problems of AI, vibecoding and “AI-assisted/AI-boosted” programming/engineering/development.
Someone’s ITIL certified
Great summary.
It’s obvious if you actually use software beyond the average literacy of a talking chimp. I use hundreds of apps across iphone, mac, and linux os’s. Literally none of them have noticeably increased in quality, stability, or feature-set beyond their average between 1-5 years ago.
Mac and iphone appear to have more bugs and shittier quality control than at any other point in the last decade.
Quality software is getting harder and harder to find thanks to all the slop-abandonware being promoted by slop-content and slop-SEO on slop-enshittified search engines.
I notice far more idiocracy-grade errors in digital content, cx, business processes, product listings, etc than ever before.
Weather forecasts have gone to dogshit in the last 2 years. Even same-day forecasts can shift on a dime unpredictably. It’s at the point where I check 3 apps. Until this year, I never had a time where I woke up to 0% chance of rain and sunny, then looked outside to see rain. Not a sun shower. A rainy day hour-long downpour.
Auto-generated subtitles are great for content that was never going to receive human attention, but they’re clearly being used to replace humans. At least once a week I notice a major contextual error that completely alters the perception of the line/scene, and there’s no way to submit corrections.
Art, culture, and knowledge are being actively corrupted, bastardized, and destroyed.
The future simultaneously sucks while being dumb as all fuck. Complete clown show run by the idiot criminals, rapists, pedophiles, scammers, and thieves.
Weather forecasts have gone to dogshit in the last 2 years. Even same-day forecasts can shift on a dime unpredictably. It’s at the point where I check 3 apps. Until this year, I never had a time where I woke up to 0% chance of rain and sunny, then looked outside to see rain. Not a sun shower. A rainy day hour-long downpour.
Supposing you’re right that weather forecasts have gone to dogshit (I’m skeptical because your evidence is anecdotal), there may be reasons beside AI for that.
Yeah I was gonna add the weather part could be entirely explained by climate change and fascisms war on science, but forgot.
I’d never heard of the 5G impact, so thanks for that. I consider it unlikely unless there were a significant change in 5G spectrum usage during the last year. 5G has covered most Australian major cities since 2021, and 5g coverage is essentially non-existent outside of populated areas (90% or more of Australia).
Weather forecasting is actually one area where AI should far exceed what humans could ever possibly achieve without it. It is impossible for humans to process and adapt to the changing patterns across hundreds or thousands of variables interacting with each other in real time.
I was gonna say - the bulk of weather forcasting is done by governments. Commercial weather channels/sites/apps draw from these predictions and give them a nice interface, and sometimes apply some proprietary prediction algorithm (eg, a lot of commercial weather services bump up chance of rain, because people are angrier when it rains when you said it wouldnt, versus doesnt rain when you said it would).
But I wouldnt expect government forecasts to have changed due to AI tools. Except maybe in the US due to the DOGE debacle - but even then, why fuck with it?
I wasn’t implying that LLM’s have damaged the science or statistics of forecasting. I was implying that they could have degraded the software at any point in the chain from data collection device firmware, to ingestion pipelines, to the app on your device; perhaps even impacting the meteorologists ability to perform quality work themselves.
Except maybe in the US due to the DOGE debacle - but even then, why fuck with it?
Changing ANY data source will always have some degree of impact on ALL statistical modeling that utilises it. Global meteorologists have depended on data captured from NASA/NOAA satellites in some capacity since the 1970’s, by virtue of Americas lead in the space race and soft power in offering this data freely to everyone since the 1970’s.
DOGE/Fascist crimes specifically targeted government scientific institutions and their ability to operate.
Yes it blows my mind everyone can’t see all the historically perfectly fine software starting to crumble to dust
The big one for me was rsync, maintainer started vibecoding it and the first release with those changes had a huge amount of regressions.
Yet it keeps getting held up as an example of “using AI tools properly”.
This aligns with my experience, largely. Of course it’s still my job to maximize LLM effectiveness within my organization. Which is a delicate balancing act to protect my teams from overeager executive leadership looking for huge gains.
My own summary is that AI can be an accelerator, but the harder you lean into it, the worse outcomes will be. No matter how much code is written, you still need actual human minds to understand it and they can only handle so much volume before getting overwhelmed.
Also, if AI gives you 20% productivity gains, but that 20% goes into playing with AI trying to get more, you haven’t really gained anything. Usage needs to be standardized rather than developers constantly negotiating with AI trying to coax out better outcomes.
The impression I’ve gotten, fooling around with it at home and talking to friends in tech and hearing from actual users online is:
A good developer can develop faster with it. Giving it small, discrete tasks for first drafts or throw away code (like bash scripting) can work well.
It’s better google. If you are trying to figure out if a function that will do a thing exists, or are trying to figure out what architecture would work best in a given situation, it can be helpful. But in these cases, it should be used carefully - dont ask it to do your work for you, ask it to give you options, pros and cons, and sources. But in this regard, it can do a lot to help an experienced developer become more productive faster in a stack or tool they are unfamiliar with.
Vibe coding is a real thing, and it can work. For internal tools in a small company, a non-technical person can create a mostly functional piece of software to get a job done. My expectation is that over time, these people will become real developers, as they end up dealing with bugs and edge cases in the vibe code they created.
At the top end of ai-for-software-development, there is some sort of something with automated iterative looping and verification, where a developer can translate a set of requirements into code, and then a collection of ai agents iteratively develop the code until it works as expected. This is what the tech bros seem really hyped on, and it does seem to work… but at the same time, my feeling is that this is how you get multimillion dollar AI bills. And presuming this is how big tech is developing their products - it seems prone to making inefficient, buggy code, so I don’t think it will be worth it long term.
This is a tale older than AI. Most of the AI productivity pushes I struggle to get adopted fail not because of AI bad or its too hard to do. They fail because of a broken CI/CD pipeline. They fail because some team thinks their process is sacred and unique.
human minds … can only handle so much volume before getting overwhelmed.
It is possible, you know, to consider flourishing employees not just not-burned-out ones.
Some of this does not line up with my lived experience pretty starkly.
Repo level markdown files with architectural guidance not working for example… I’ve found that works quite well.
Not perfectly well, but llms are designed specifically NOT to be perfect deterministic executioners. Still though, pretty well.
I have seen that in a jr engineers hands llms get to bad outcomes fast, and unintuitively (to leaders…) usage of llms in coding does not provide a path for a he engineer to upskill into a sr engineer. A sr engineer with llms though is almost always radically augmented regarding their output speed on task completion.
Repo level markdown files with architectural guidance not working for example… I’ve found that works quite well.
Same. AGENTS.md files and the like are quite effective. Especially if you’re reviewing the code and making the LLM help you update the markdown files when it makes a mistake to prevent the same type of mistake in the future. Having concrete examples of “good” vs. “bad” to illustrate each architectural rule goes a long way.
For any feature or bug fix that is “painting with the colors already in the tray”, it makes sense to let a LLM write the code. Humans will introduce new tech and new patterns out of boredom and turn the codebase into a big Frankenstein, but the LLM will just follow the architectural guidelines indefinitely.
Agree, I have them curated lessons.md anytime they make a mistake and have found that to be highly effective. Every now and then a lesson goes defunct and needs pruned, but I think that’s just part of the new swe skill set.
It’s possible to win lots of battles but still lose the war. You can ask Trump about that :)
I agree with your last paragraph. We had about 6 weeks of unlimited AI spend before the costs reached executive leadership, and in that time I saw the least experienced developers spend the most with the least to show for it.
But I will say that another factor is thinking that if you get 10% gains from a little AI, then a lot of AI will get you 100%.
But I find the article is right about repo-wide docs. At least on their own. I find having small markdowns (often in the form of skills/commands), focused on specific tasks reduces spend (especially when your execution agent is a low cost model, leaving the reasoning to dedicated agents) and gives better outcomes. Loading massive docs into every task reduces the attention to the task at hand and often confuses AI as the reasoning part of the model becomes overwhelmed and starts inferring wrong things confidently.
I suppose it heavily depends on the scale of the repo though. A large microservice with multiple upstream services it needs to call spends a lot tokens on API which is unnecessary for most tasks. And then it decides to use the wrong one… I have stories lol.
Repo level markdown files with architectural guidance not working for example
I’ve seen it become less and less effective as the size of the file(s) grew and as the codebase grew - they got increasingly more diluted or even lost in context compression. After several months of a 6 man team working on the project the rate at which they got ignored started affecting output a lot.
Lol. Who knew?
Hehehehe
LLMs struggle with negation. Telling them not to do something can often have the same effect as telling them to do it.
The future looks… unreliable.
Models may get more powerful, but not significantly more reliable. This it folks – work with what you’ve got!
We all know that true AGIs becoming smarter than humans seems inevitable, but that could be like a hundred years from now, if ever. What’s unclear is what will happen two years from now, involving matters having little to do with the technology & what it is capable of and instead more to do with the economy and what jobs will be available then.
AGI isn’t possible with current or near tech. Anyone who says otherwise is huffing paint or selling AI crap.
I literally said “if ever”, and also “seems” rather than “is”. I also never so much as implied current tech, with my comment about a hundred years from now.
You are reacting against what I never said.
Though I choose to upvote your comment anyway, since at least you said it rather than simply assumed it and moved on.
Wow, what a solid argument you’ve made. It definitely doesn’t reek of someone trying to hype themselves up in the face of an unknown threat.
We invented a next token predictor. That isn’t intelligence nor is it on the path to it, either. The word rocks aren’t any closer to real intellect than the math rocks were. You just think they are cause words are the things that humans use to communicate across time and space.
The only ones spouting nonsense that it is are those whose business models require AGI to be achievable within the next decade. But, ya know, wish in one hand and all that.
That’s better, but you’re still just defining things such that you have to be correct.
Humans spend 90% of their time just predicting the next token. All we need is to mix in a little bit more reasoning and bam, AGI!
See how annoying and hollow that rhetorical technique is?
The fact is that we have no idea whatsoever. We might reach AGI next week. We might never reach AGI. The Chinese government might be sitting on AGI right now. We have no idea, and this is completely new territory that has caused us to question our very understanding of intelligence.
The problem is that you’re not arguing anything. Your first comment is an ad hominem at best, and your second is an appeal to the unknown. You haven’t argued anything.
Here, here’s my argument. LLMs are built on neutral networks. They use a ridiculous amount of compute to “simulate neurons”, but their neurons are not even a 100th of the power of an actual neuron, which is why this is a dead-end for AGI. You’ve probably seen this diagram before
And had it explained that each of the nodes in this image represents a virtual neuron. Assuming that AGI is achieved via the means we know right now, namely being human-like, then you’d probably expect creating a neural net with the complexity of a human brain, minus all the stuff needed for steering a meat-suit, would get us there.
The problem is that those little nodes are input-output machines that are ridiculously simple compared to real neurons. They take in inputs from every connection they have, based on internal weights and measures, and then they output a single value to all their output neurons. That value is a float between 0 and 1, representing some percentage of activation.
Real neurons, meanwhile, use neurotransmitters to communicate to their neighbors, not simple electrical signals. There are hundreds of those, but even if there were 10, then neural networks would be completely incapable of modeling a simple brain, let alone a human one.
Why? Because that means each of those nodes in the neural net would need 10 separate inputs and outputs connected to all their existing nodes to even come close. That’s a 100x more connections for every part of the node. And we’re currently consuming basically every chip on earth to feed to this thing, and it’s not even 1/100th the complexity needed for even a dumbed down neuron, let alone a realistic one.
15 - 30 years of pain followed by the end of life as we know it those who adapt may thrive but will continue to be exploited.
Tbf to LLM manufacturers, the end of life as we know it was coming either way.
