Using LLMs to analyze code for problems makes a lot of sense. It is just an extension of the rule based analysis we already do. It is very much a better hammer rather than a replacement of labour/thinking/taste. The programming languages we use are often difficult for humans to reason about reliably and hide a lot of errors. Our programming languages still suck (yes even Rust). If one day clang of gcc ships with a local LLM to lint code for security problems I would probably run it occasionally even on my hobby code despite being an AI hater.
If people are identifying errors using LLMs and fixing them by hand I am fine with that. It seems like a good use of the technology. If they start vibe coding drivers I want it declared so I can disable them or move to BSD.
You know, compiler based AI (not just LLMs) seem like an idea that might be pretty interesting. The option to disable or completely omit them from the toolchain would be mandatory however. But, even then we would have to consider how many pieces of software that then rely on this proverbial black box built into compilers would then get integrated into critical supply chain? We soon would have to start running behavior based “background checks” (just an analogy) on our software much like we do with people due to the unpredictability of such complexity.
Side note, I can’t wait to see just how complex the field of game theory gets with AI. I would love to see any video essays on the topic if they do exist already.
Many projects require replicatable builds. Generative AI is driven by noise so I am not sure if it has any role in optimization or code gen. I guess we could share an RNG seed with source but honestly current tool chains are too slow as it is and ML would likely make it much worse. I want faster edit/compile/debug cycles and compilers already do insane optimisations. Ken Thompson’s Reflections on Trusting Trust becomes much scarier if your code generation is being driven by a black box of billions or parameters nobody understands. An adversary could embed anything in there.
The machine translation origins of LLMs make them really good at inspecting text and looking for things that look like known bugs. It must be a game changer for security researchers who already know what sort of problems to look for but don’t have enough hours in the day to search the haystack manually. If some of that knowledge was available in a good quality FOSS local code analysis tool it might be irresponsible not to use it. We can’t be dependent on commercial vendors and closed models.
I do wonder if Ai driven reverse engineering could have reproducible builds based on a similar principle so as to avoid needing to distribute any part of the original program
Using LLMs to analyze code for problems makes a lot of sense. It is just an extension of the rule based analysis we already do. It is very much a better hammer rather than a replacement of labour/thinking/taste. The programming languages we use are often difficult for humans to reason about reliably and hide a lot of errors. Our programming languages still suck (yes even Rust). If one day clang of gcc ships with a local LLM to lint code for security problems I would probably run it occasionally even on my hobby code despite being an AI hater.
If people are identifying errors using LLMs and fixing them by hand I am fine with that. It seems like a good use of the technology. If they start vibe coding drivers I want it declared so I can disable them or move to BSD.
I agree that finding bugs is probably one of the most useful usages of LLMs ever.
But avoiding anything written with AI will let you few options in the future.
Agreed, this is one way that LLMs can increase the quality of code.
You know, compiler based AI (not just LLMs) seem like an idea that might be pretty interesting. The option to disable or completely omit them from the toolchain would be mandatory however. But, even then we would have to consider how many pieces of software that then rely on this proverbial black box built into compilers would then get integrated into critical supply chain? We soon would have to start running behavior based “background checks” (just an analogy) on our software much like we do with people due to the unpredictability of such complexity.
Side note, I can’t wait to see just how complex the field of game theory gets with AI. I would love to see any video essays on the topic if they do exist already.
🧐
Many projects require replicatable builds. Generative AI is driven by noise so I am not sure if it has any role in optimization or code gen. I guess we could share an RNG seed with source but honestly current tool chains are too slow as it is and ML would likely make it much worse. I want faster edit/compile/debug cycles and compilers already do insane optimisations. Ken Thompson’s Reflections on Trusting Trust becomes much scarier if your code generation is being driven by a black box of billions or parameters nobody understands. An adversary could embed anything in there.
The machine translation origins of LLMs make them really good at inspecting text and looking for things that look like known bugs. It must be a game changer for security researchers who already know what sort of problems to look for but don’t have enough hours in the day to search the haystack manually. If some of that knowledge was available in a good quality FOSS local code analysis tool it might be irresponsible not to use it. We can’t be dependent on commercial vendors and closed models.
I do wonder if Ai driven reverse engineering could have reproducible builds based on a similar principle so as to avoid needing to distribute any part of the original program