the models only need to be right some of the time and they still constitute a threat.

Right!?!

And with how cheap and fast they are to run, they don’t have to be right that often as long as a separate run can verify/validate claims. In a lot of ways what the LLMs do should probably be thought of as just a very sophisticated sort of brute force approach. You’re effectively just searching the space of combinations of known vulnerabilities. It doesn’t require particular cleverness, just an encyclopedic knowledge, a reasonable way to prune the search space, and the ability to put in the compute time.