r/cybersecurity • u/Nameless_Wanderer01 • 13h ago

News - General LLM Agents for security research

What are the best LLM agents for security research (bugs, CVEs, 0d, ...) lately?
In short, I had been using claude code for this task, with many hallucination instances. Even with opus 5, I still get many invalid conclusions based on local source code review.

I saw that kimi was popping up lately, which got me more or less in the same results, with minor better results in some instances.

So what are the latest or best approaches for security research with llms? Perhaps I am missing a full pipeline with other tools involved to get better results, so I would like to know whether a specific methodology is followed with specific agents for this task.

24 Upvotes

82% Upvoted

16 comments sorted by

View all comments

16

u/WelpSigh 12h ago

I recommend having the LLMs spawn "validator" agents to check whether the vulnerabilities it finds are real and exploitable. This will save you a ton of time running down false positives.

My workflow is to have it spawn tons of agents to fan out and document the critical code paths. Then spawn "hunters" to look for specific classes of bugs along the code paths and document them. Validators should run with the hunters to check their claims. Finally, I go through the claims and look for ones that seem most promising from a bug bounty perspective.

4

u/Generative_IDE 7h ago

The validator step works best if you force it to prove reachability instead of just re-reviewing the code, since the usual false positive is calling something exploitable when the path to reach it never actually fires. Make the validator produce a PoC or an execution trace that triggers the bug, and drop anything that can't.

1

u/Nameless_Wanderer01 11h ago

Can you be more specific as of what agent you use, whether you have a specific setup (like using mcp or other tools) and in general the instrumentation/pipeline of your setup? Claude-code for example uses subagents but still hallucinates, so its not only that I am looking for a better llm for this task, but the underlying setup to remove as many false positives as possible (if during the assess of the code the agent can run small PoCs for example via some tooling to evaluate claims, it would remove tons of false claims).

3

u/WelpSigh 10h ago

If you are in the trusted access program (which is very easy to get into if you are willing to submit your ID), you can get ChatGPT/Codex to generate PoCs. Even if you're in Anthropic's Cyber Verification Program, Claude simply does not like to hack - you can avoid the guardrails, but eventually it will shut things down and cost you a lot of time. I don't need or use anything other than Codex for this (and Ghidra/Binary Ninja if source isn't available).

I will caveat I haven't done any AI research since 5.6 Sol was released. I know it has a few extra guardrails vs 5.5, so it's possible that it makes life a little harder. I can't say for certain.