I am a programmer with 25 years of professional experience, mostly embedded C/C++, with a decent amount of Python and the last year has been mostly Rust with no prior experience.
Current LLMs write both worse and better code than me, often at the same time. Let me explain.
LLMs are very good at syntax, with an encyclopedic knowledge of libraries, languages, and algorithms that is a few years out of date. I particularly notice this in Rust, which has a lot of syntax and features, like the 482ish ways to chain Options and Results that I still haven’t memorized all of.
So for a relatively simple function in a language that I’m not very famialiar with they will often produce a bit tighter code that makes better use of the language features.
They will also sometimes come up with an obscure algorithm that I haven’t though about in decades or even heard of that can solve the problem better than my first attempt.
On the other hand, they are terrible at larger scale work, generally building fragile code while protecting against things that can’t actually happen, using overly complex architecture, organizing code in dumb ways, missing obvious opportunities for simplicity, and relying too much on existing bad patterns instead of fixing them along the way, etc. They also tend to be a terrible judge of when to use a library vs roll their own, leaning heavily towards rolling their own instead of just adding a single include, which means more new untested code.
On the third hand, I do find that they can be pretty good at reviewing code, and it is nice to have them write tests, since we all hate writing tests and tend to write as little as we can get away with.
Re: bad with large code – that’s been something I’ve been noticing as well. A project I’ve been working on for work has gotten larger now and it seems more often that it makes goofy mistakes. Like removing something somewhere but leaving something related somewhere else. My best guess is that it’s all context window stuff. As humans, we can learn a lot and keep it all in our heads. But an AI model, if it’s not something from its training set and it’s not in its context window, then it just straight up does not know it exists. I’m guessing because context windows have gotten bigger that ai coding has gotten more impressive. But we’ll see if they’re able to keep scaling that. I imagine at some point it’s gonna hit a soft wall (or at least a price wall lol).
I am a programmer with 25 years of professional experience, mostly embedded C/C++, with a decent amount of Python and the last year has been mostly Rust with no prior experience.
Current LLMs write both worse and better code than me, often at the same time. Let me explain.
LLMs are very good at syntax, with an encyclopedic knowledge of libraries, languages, and algorithms that is a few years out of date. I particularly notice this in Rust, which has a lot of syntax and features, like the 482ish ways to chain Options and Results that I still haven’t memorized all of.
So for a relatively simple function in a language that I’m not very famialiar with they will often produce a bit tighter code that makes better use of the language features. They will also sometimes come up with an obscure algorithm that I haven’t though about in decades or even heard of that can solve the problem better than my first attempt.
On the other hand, they are terrible at larger scale work, generally building fragile code while protecting against things that can’t actually happen, using overly complex architecture, organizing code in dumb ways, missing obvious opportunities for simplicity, and relying too much on existing bad patterns instead of fixing them along the way, etc. They also tend to be a terrible judge of when to use a library vs roll their own, leaning heavily towards rolling their own instead of just adding a single include, which means more new untested code.
On the third hand, I do find that they can be pretty good at reviewing code, and it is nice to have them write tests, since we all hate writing tests and tend to write as little as we can get away with.
Re: bad with large code – that’s been something I’ve been noticing as well. A project I’ve been working on for work has gotten larger now and it seems more often that it makes goofy mistakes. Like removing something somewhere but leaving something related somewhere else. My best guess is that it’s all context window stuff. As humans, we can learn a lot and keep it all in our heads. But an AI model, if it’s not something from its training set and it’s not in its context window, then it just straight up does not know it exists. I’m guessing because context windows have gotten bigger that ai coding has gotten more impressive. But we’ll see if they’re able to keep scaling that. I imagine at some point it’s gonna hit a soft wall (or at least a price wall lol).