I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
As this article points out, an LLM can spit out chunks of regurgitated code that it scraped from the internet, but that does not make the LLM a programmer. The resulting output is an attempt to find an existing pattern in the database which fits with what the user has asked for, but it is not a product of actually understanding the use case for the code. It is just statistical correlation.
So, sure, an LLM can be set up to generate output related to X task. If you can collect and clean data that can be used to train the kind of output you want, it should be able to produce an approximate facsimile of the results you want. Is that valuable for your use case? Maybe.
We’re still just talking about what is essentially a complex search function. The statistical model returns results from its database that correlate most closely to your input. That does not mean it returns the right answer. If there is no good correlation, it will still return a result.
As long as you understand that the result you get is just a correlation based on your input and may or may not be relevant to your specific problem, and you are not fooled into believing that the LLM actually understands what you’re asking and produced a result by “thinking” about it, then you might be able to use an LLM as an effective tool - to search a large collection of information for something that is relevant(ish) to what you’re asking for.
The real mistake has been broad misunderstanding of what LLMs actually do, and trying to use them as general-purpose problem solving tools (or worse, as accurate and reliable sources of information).
Language models are not databases and they are not markov bots (similar function but work directly using statistical word association maps). The big difference is that those things are algorithms someone wrote and can fully comprehend what they do, but machine learning models are large algorithms built by another algorithm processing training data. There is much more uncertainty about what is going on under the hood.
There is also great uncertainty about what concepts like understanding or thinking might mean in computer science terms. The main thing we can really know is that ultimately a human mind is a computer, which means that understanding and thinking have some yet unknown mathematical representation, and therefore a comparison can be made. We should eventually be able to quantify whether or to what extent a given algorithm thinks. But you said in another comment that you don’t believe minds can be represented mathematically; this should mean that such comparisons would be apples to oranges, but you’re making them anyway for some reason, and implying they have predictive power for the limitations of LLMs.
Certainly they do have limitations, at least individually and possibly as a technology. There are things given models are bad at, there are things they initially seem to be able to do well as humans but fail in different ways that suggest over-reliance on pattern matching. But these have been determined empirically through testing. The idea that they are “just statistical models” and this knowledge can be used to say what is impossible for them from philosophical first principles keeps getting repeated but has never worked in practice. The reality is that no one knows enough to say for sure where the line is.
Except that it’s been demonstrated multiple times that original training data can be extracted from a language model, so it is completely valid to talk about the model as a database, because the training data is stored within it.
Here’s a broad survey of more than 100 research papers demonstrating this: Training Data Extraction From Pre-trained Language Models: A Survey
So, this is a good anology in this case.
See, I know how an internal combustion engine works. I don’t know, by looking at the hood of a particular vehicle, how exactly a specific car’s engine operates (maybe it has 4 cylinders, or 6 or 8, maybe it has fuel injectors, maybe it has a carburetor, etc). However, I do know that the principles are the same for all internal combustion engines, and that just because I don’t know the details of how a particular engine operates, that does not mean that its operation is beyond my understanding.
The same is true for machine learning models. There may be uncertainty as to how a particular model operates “under the hood”, but the principles of operation are the same for all, and are not incomprehensible.
We actually don’t know this. This is called computationalism. It is speculative, there are several alternative theories, and little in the way of experimental evidence supporting any particular theory.
You have to understand, the current branch of machine learning models grew out of algorithms whose purpose was processing large data sets with thousands or millions of variables and optimizing for areas in the data set where many of those variables were maximized (or minimized). Here’s a better explanation:
Hill Climbing Algorithm & Artificial Intelligence - Computerphile
How these tools perform their optimization, and what they optimize for, has been recombined in different ways to produce different types of models, and the search space of variables has been expanded with increased computing power, but the underlying operating principles are still the same. This is not a tool that can comprehend what it is doing, it can’t be self-aware. It can only process large amounts of input data and attempt to maximize for particular dimensions. This seems vague to humans because the amount of variables being handled at any given time is far more than a human mind can focus on, but that doesn’t make the optimization routine intelligent or conscious. It’s just doing a lot of number crunching really fast, optimizing for specific aspects as directed by its developers.
It’s been demonstrated that some more prominent pieces of training data can be reproduced, the majority of it cannot. This shows that those particular pieces of data are represented in some form within the model, it does not show that the way it works is equivalent to database lookups. If I can write down the lyrics of a song from memory, it shows that those lyrics are encoded as data in some form in my brain, but that doesn’t mean it’s valid to talk about my brain as a literal database, especially not in the sense that the limitations in the capabilities of a database can be ascribed to me (or its strengths, I cannot remember the exact lyrics of most songs I’ve heard, even if I can remember some).
This video literally starts out by describing evolution as a similar optimization algorithm. If you know the basic mechanism of evolution, does that mean you can use that to then say with certainty and specificity what biological life in its vast diversity of techniques is not capable of? The “underlying operating principles” of evolution don’t “understand” chemistry or deception, but they still produce organisms capable of photosynthesis and camouflage. It’s an algorithm that produces other algorithms, which is what puts those resulting algorithms in a different category of comprehensibility than fixed algorithms that were explicitly written by someone. We are very far from having a comprehensive understanding of biological systems, despite knowing how evolution works.
This is like saying evolution is only a simple mechanism taking in the world as data, which, yeah, obviously, but that property doesn’t carry forward to what it produces. The bigger problem here though is, again, concepts like comprehension, consciousness, and intelligence are not well defined in computational terms, and it is unclear what statements involving them mean in any practical sense. These sorts of claims are non-falsifiable and don’t make testable predictions about the boundaries of AI capability.