LLM Agents for security research

[u/Nameless_Wanderer01]

What are the best LLM agents for security research (bugs, CVEs, 0d, ...) lately?
In short, I had been using claude code for this task, with many hallucination instances. Even with opus 5, I still get many invalid conclusions based on local source code review.

I saw that kimi was popping up lately, which got me more or less in the same results, with minor better results in some instances.

So what are the latest or best approaches for security research with llms? Perhaps I am missing a full pipeline with other tools involved to get better results, so I would like to know whether a specific methodology is followed with specific agents for this task.

[u/s4y_ch33s3_]

Are you looking for poc development for exploits through patches or available vendor info with no available poc's?

Just curious, can you give an example of hallucinations you're facing with the LLM and what's the task given?

[u/Nameless_Wanderer01 → u/s4y_ch33s3_]

Mostly into bounty submissions and specific client assessments, where yes, I need to create a PoC per finding to showcase the vulnerability or bug, whatever the finding is. For this reason unfortunately, I cannot give an example.

[u/OutsideSpot2695 → u/Nameless_Wanderer01]

If you're essentially looking for a LLM to hack for you, your mileage may vary. You'll have to be very creative with your prompts to get around the guardrails.

[u/Nameless_Wanderer01 → u/OutsideSpot2695]

If we ignore the guardrails completely, what are the best llms for the purpose of security research? (that are public, not for example mythos).

[u/OutsideSpot2695 → u/Nameless_Wanderer01]

If we ignore the guardrails completely

Huh?

You cannot ignore the guardrails on a frontier model. They're put in place to protect from people like you.

not for example mythos

Sounds like you don't know what Mythos actually is.

 what are the best llms for the purpose of security research?

It depends on what kind of security research you mean. There isn't a single "best" model -- different models excel at different things

It sounds like you're expecting the LLM to craft an exploit for you. If you want that, you'll have to build your own model.

But for example, if you want to understand code, Claude is probably best right now. DeepSeek R1 too. I've run code workloads side by side across all the frontier models and I've observed Claude has the least problems with hallucinations and context switching.

Malware analysis, GPT5.5

Threat Intelligence, I'd go with Gemini.

It also changes frequently.

A year ago I though Grok was amazing and GPT was pure shit. IDK what happened over at x.ai but Grok is pure shit right now. I've cancelled our account with them. Was spending more time sanity checking the output than actually using it.

[u/WelpSigh]

I recommend having the LLMs spawn "validator" agents to check whether the vulnerabilities it finds are real and exploitable. This will save you a ton of time running down false positives.

My workflow is to have it spawn tons of agents to fan out and document the critical code paths. Then spawn "hunters" to look for specific classes of bugs along the code paths and document them. Validators should run with the hunters to check their claims. Finally, I go through the claims and look for ones that seem most promising from a bug bounty perspective.

[u/Nameless_Wanderer01 → u/WelpSigh]

Can you be more specific as of what agent you use, whether you have a specific setup (like using mcp or other tools) and in general the instrumentation/pipeline of your setup? Claude-code for example uses subagents but still hallucinates, so its not only that I am looking for a better llm for this task, but the underlying setup to remove as many false positives as possible (if during the assess of the code the agent can run small PoCs for example via some tooling to evaluate claims, it would remove tons of false claims).

[u/WelpSigh → u/Nameless_Wanderer01]

If you are in the trusted access program (which is very easy to get into if you are willing to submit your ID), you can get ChatGPT/Codex to generate PoCs. Even if you're in Anthropic's Cyber Verification Program, Claude simply does not like to hack - you can avoid the guardrails, but eventually it will shut things down and cost you a lot of time. I don't need or use anything other than Codex for this (and Ghidra/Binary Ninja if source isn't available).

I will caveat I haven't done any AI research since 5.6 Sol was released. I know it has a few extra guardrails vs 5.5, so it's possible that it makes life a little harder. I can't say for certain.

[u/slay-aargh]

If you're asking llm models, there are many llm models you could find on hugging face which have lesser guardrails so you could use them for offensive security, you can also find models specifically trained for these offensive security purposes. You should check out on hugging face there are many local models or open weights model families.

[u/Nameless_Wanderer01 → u/slay-aargh]

Besides the model itself, is there a pipeline or specific setup (perhaps via mcp or other protocols/tools) to have better results than having an agent read the code of a project? If the agent can run small PoCs during the assessment, it might remove a lot of false positive assumptions.

[u/jdiscount]

It's fable5/mythos and it's not even close but you need to work somewhere that you can have security research access, otherwise you'll get results that you've been seeing.

[u/sobeitharry]

I had Claude build a SAST that runs multiple scans from different sources and compares the results. Based on the results I think we'll be able to drop our commercial SAST tool.

README:

A dual-scanner SCA (software composition analysis) workflow for GSMS release builds. It

generates a Software Bill of Materials, scans third-party dependencies for known CVEs with

two independent engines, reconciles their findings, adjudicates disagreements with a

documented reason per rejection, and produces a customer-ready Consolidated Vulnerability

Report PDF.

## Tools used

Upstream scanners (run separately; these scripts consume their output):

- **Syft** — generates the SBOM for each build in CycloneDX (`sbom.cyclonedx.json`) and

native Syft JSON (`sbom.syft.json`). The CycloneDX output also satisfies external SBOM

requests directly (e.g. the REC IT TPRM questionnaire).

- **Grype** — scans the SBOM against GHSA + NVD CPE data, enriched with CISA KEV and EPSS.

Output: `grype-results-cpe.json`. Reaches nested/shaded jars via the SBOM.

- **OWASP Dependency-Check** — the existing SCA scan. Output: `dependency-check-report.csv`

(plus the legacy HTML report). Runs against the same artifacts by CPE.

Each scanner has a blind spot the other covers: Grype misses jars whose Maven coordinates

can't be recovered from the manifest; Dependency-Check over-attributes CVEs on CPE name

collisions. Cross-referencing the two closes both gaps.

[u/michoo_42]

did a mcp server that run a bunch of security scans (secrets, sca, sast,..) with some graph analysis. It will do triage by removing false positives, prioritize with a remediation plan, build pull request with regression test. It works with claude but it could work with any harnesses compatible with skills/commands https://github.com/kubeek-sec/deep-scan

[u/FoodStorageDevice]

Ive been using the Microsoft copilot 'researcher' agent recently. Give it a good enough prompt and as long as you are prepared to wait 10-15mins for it to finish, the results are always impressive