Defenders finally have a chance to win, decisively
I’m curious how it will turn out to be in a long term. Are we going to have safer software? Because not only defenders will have a powerful tool, but attackers too. But at the same time, number of bugs is finite… Can we in theory one day achieve literally zero bugs in codebase?
Another factor Mozilla didn’t mention (and that Anthropic wouldn’t like to emphasize) is that major LLMs are pretty similar. And their development is way more conservative than you’d think. They use similar architectures and formats, train from the same data, distill each other, further pollute the internet with the same output and so on. So if (for example) Mozilla red teams with Mythos, I’d posit it’s likely that attacker LLMs would find the same already-patched bugs, instead of something new.
…So yeah. I’d wager Mozilla’s sentiment is correct.
Add to that that AI is pretty good at copying from pre-existing knowledge (like a database of known vulnerabilities) and not good at generating novel ideas (like discovering a new vulnerability), and the scales are further tilted in the defenders’ favor.
Are we going to have safer software? Because not only defenders will have a powerful tool, but attackers too.
Probably not safer software, but the window of time for a bug being known and exploitable will be shortened greatly. Instead of 0-days, we might have 0-minutes.
That’s assuming these ridiculous AI systems are rolling deployments that fast, so maybe that idea’s nonsense.
Cyber security in general is going to get interesting. Breaking into protected systems often requires more patience than expertise. Attackers often get detected when they take short cuts because of laziness and overconfidence. AI agents have unfathomable patience and attention to detail.l
I don’t really agree with the attention to detail part from my experience. AI agents love to take shortcuts from what I’ve seen, and you have to pay a lot of attention to what they’re doing to make sure they do the right thing.
AI will be good at scaning for known vulnerabilities, but patience and attention to detail? Not in my experience. I use agentic coding agents for work and they are getting better, but they still will regularly get stuck in a loop of running into a bug when running tests, attempting to fix the bug in a stupid way, still erroring, trying another stupid fix, trying the first stupid fix, and so on until a human intervenes. They may be patient (as long as you pay for more tokens), but they aren’t using their time wisely.
AI tends to use the “throw shit at the wall and see what sticks” approach. It’s getting better at writing maintainable code, but it still will generate more-or-less spaghetti code with random unused or deprecated variables, crazy unnecessary functions, poor organization, etc… and requires lots of testing before producing something functional. Which is fine in an environment where you can iterate and clean things up. But as an attack vector, if you need 58 attempts to fully realize a vulnerability, in most secure environments you’re going to get detected and blocked before you finish.
I don’t disagree on the current state. However, it’s not hard to foresee that attack tools will be developed that can maintain “attention” on an attack for days or weeks at a time with privately run agents. I’m sure they are out there already to some degree.
Not zero bugs, but it should help. A benefit for defenders is that they can use AI review on code before they make it public or release it in a stable release
I’m curious how it will turn out to be in a long term. Are we going to have safer software? Because not only defenders will have a powerful tool, but attackers too. But at the same time, number of bugs is finite… Can we in theory one day achieve literally zero bugs in codebase?
It does seem advantageous to the defender.
Another factor Mozilla didn’t mention (and that Anthropic wouldn’t like to emphasize) is that major LLMs are pretty similar. And their development is way more conservative than you’d think. They use similar architectures and formats, train from the same data, distill each other, further pollute the internet with the same output and so on. So if (for example) Mozilla red teams with Mythos, I’d posit it’s likely that attacker LLMs would find the same already-patched bugs, instead of something new.
…So yeah. I’d wager Mozilla’s sentiment is correct.
Add to that that AI is pretty good at copying from pre-existing knowledge (like a database of known vulnerabilities) and not good at generating novel ideas (like discovering a new vulnerability), and the scales are further tilted in the defenders’ favor.
Eh, I don’t totally agree. AI can discover novel exploits that aren’t already in some database, and likely have in this case.
I’m just saying the operating patterns between different LLMs are more similar than you’d expect, like similar tools from the same factory.
You can achieve zero bugs through liberal use of rm.
You can achieve the same effect with a hammer
deleted by creator
Probably not safer software, but the window of time for a bug being known and exploitable will be shortened greatly. Instead of 0-days, we might have 0-minutes.
That’s assuming these ridiculous AI systems are rolling deployments that fast, so maybe that idea’s nonsense.
Cyber security in general is going to get interesting. Breaking into protected systems often requires more patience than expertise. Attackers often get detected when they take short cuts because of laziness and overconfidence. AI agents have unfathomable patience and attention to detail.l
I don’t really agree with the attention to detail part from my experience. AI agents love to take shortcuts from what I’ve seen, and you have to pay a lot of attention to what they’re doing to make sure they do the right thing.
AI will be good at scaning for known vulnerabilities, but patience and attention to detail? Not in my experience. I use agentic coding agents for work and they are getting better, but they still will regularly get stuck in a loop of running into a bug when running tests, attempting to fix the bug in a stupid way, still erroring, trying another stupid fix, trying the first stupid fix, and so on until a human intervenes. They may be patient (as long as you pay for more tokens), but they aren’t using their time wisely.
AI tends to use the “throw shit at the wall and see what sticks” approach. It’s getting better at writing maintainable code, but it still will generate more-or-less spaghetti code with random unused or deprecated variables, crazy unnecessary functions, poor organization, etc… and requires lots of testing before producing something functional. Which is fine in an environment where you can iterate and clean things up. But as an attack vector, if you need 58 attempts to fully realize a vulnerability, in most secure environments you’re going to get detected and blocked before you finish.
I don’t disagree on the current state. However, it’s not hard to foresee that attack tools will be developed that can maintain “attention” on an attack for days or weeks at a time with privately run agents. I’m sure they are out there already to some degree.
They have attention to detail, just not the right details. It’s super easy for them to get lost in a never ending train of tangents.
It is theoretically possible by using formal verification. Which is getting easier due to lean. But still impractical.
Not zero bugs, but it should help. A benefit for defenders is that they can use AI review on code before they make it public or release it in a stable release