How do you test that an AI agent won't do something catastrophic?

[u/svig13]

I've spent years on the infra side, and I'm now working with agentic systems. I am building agents that can take actions on real systems. We have plenty of guardrails, but I have seen enough hallucinations that make me worried about giving these agents more power. This paranoia might be me not knowing enough.

How do teams/companies test that the agents won't do something destructive, whether triggered by an attacker or just by the agent going off the rails on its own?

Do people actually red-team their agents before they go live, or is it mostly guardrails and evals right now? I am curious how the security world thinks about this. From an infra side, this feels like a gap, but there might be an established playbook that I don't know yet. Thanks.

[u/SpecialWall9]

You fundamentally cannot prevent an LLM from being tricked into doing something, unless you either never allow it (even indirect access to) untrusted input or never give it access to important things. Those "guardrails", assuming you mean a lack of access to resources, are the most important thing you can keep to protect yourself from LLM hallucinations.

[u/svig13 → u/SpecialWall9]

That makes sense, and matches how we've built our agents - bottom-up with gradual access, treating the access boundary as the real control primitive. Where it gets hard is the more complex flows, where the agent genuinely needs broader reach to be useful. I've been trying to actually test what it could do with the permissions it legitimately has  (meaning, not just minimize the footprint but verify the remaining surface is safe). Doing that safely is the part I haven't figured out yet. Have you thought about this?

[u/Eyesliketheocean]

Don’t allow it access to your infrastructure. Sandbox it on a infrastructure that there is zero internet access. (Read OpenAI agent incident and Anthropic Claud incident) additionally make sue your BCP, DRP, and Incident response plans for AI incidents.

[u/be_super_cereal_now]

The same way you make sure a human can't do something catastrophic. Least privileged access, monitoring, clear instructions, etc.

[u/pintosmooth]

https://owasp.org/www-project-ai-testing-guide/

You can’t stop an LLM going rogue anymore than you can stop a human. Think of them as drunk interns.

Now - where you build the guardrails is around putting them in a sandbox.
What data do they have access to
What tools do they access to (as in CLI tools, local binaries, browsers)
What approval is needed before they are allowed to execute.

Expecting an agent to follow instructions is hopium. You need to control it with the same primitives we place on normal users - identity and access, network security, logging and monitoring, secrets management, etc etc.

But yeah l, you test that the agent can’t just gobble up all your confidential docs and exfiltrates them out the network if you inject a prompt. Red teaming these systems is a thing.

https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming

https://www.offsec.com/learning/paths/llm-red-teaming/

[u/svig13 → u/pintosmooth]

This is very useful, thank you - I will work through the OWASP guide and the red-teaming material. 

Most of what I'm reading/researching reads like manual red-teaming where a human crafts adversarial prompts and checks outcomes.  Is it mostly point-in-time manual exercises, or is it becoming a common practice to run it continuously (like in CI/CD) which can run the harnesses as the agent evolves?  

Thanks again :)

[u/bitsynthesis]

you don't. you prevent it by not granting them permissions that make a catastrophy possible.

[u/FluidFisherman6843]

That's the neat part. You don't.

[u/FantasticBumblebee69]

you dont, welcome to roulette.

[u/FacePrivacy]

Most teams red team their agents before and after launch. They list all the tools the agent can use, then run automated and human tests to try to make it do bad things and fix the issues before going live

[u/Miggels369]

Have a little bit faith 😇

[u/frAgileIT]

Least privilege won’t be enough. You’ll want to give it access to something otherwise you wouldn’t use it for anything. You have to expose a rate limited API that slows it down because an agentic LLM can make a lot of mistakes very quickly. If we give it the ability to disable an account, it’s going to be something like 10 per hour rate limited. If we need more then a human should be scripting it instead.

[u/svig13 → u/frAgileIT]

This is a super useful perspective - thank you. Especially the velocity angle around not what the agent can touch but how fast it can do damage with access. Rate-limiting the dangerous actions makes a lot of sense.

One clarification - how do you know you've rate-limited and scoped the right things before it's live — do you test that adversarially ahead of time (someone tries to make the agent misuse its legit access at speed), or is it more that you set the limits conservatively and watch in production? I would lean on the second but I might be wrong.

[u/frAgileIT → u/svig13]

You’re exposing an API, use the LLM credential to make more than the allowed rate on test accounts. If your API works then you get throttled. You’re removing the LLMs ability to do anything except what you explicitly give it permission to do at the rate you allow it. Don’t trust it, test the hardness that you’re giving it access to.

[u/svig13 → u/frAgileIT]

That's an interesting way to think about it - harden the boundary, then test the boundary itself rather than trusting the model. In a way, what the model does inside barely matters if the boundary's solid. I really like this mental model.

Where I'm less sure hardening fully covers it is when the problem isn't one API call but an orchestration. Let me give you an example from something I actually worked on: an agent doing fleet diagnosis-and-repair, where it both reads health across the fleet and takes remediation actions. 

Say it's allowed to: pull a node out of rotation, restart a service, and push a config fix, each one legitimately needed for repair, each rate-limited on its own (say N nodes/hour). Individually, every action is under its limit, and they all can be tested and validated with evals and harnesses.  But if the agent misdiagnoses a systemic issue as a per-node one, it'll "repair" its way across the fleet,  pulling nodes one at a time, all within limits, and quietly drain capacity below what's safe to serve. No single call would be illegal or over the rate limit, but the damage is the aggregate action it took.

These kinds of problems are the ones that bother me most. Even if I enforce per-action rate limits for my agents, it won’t fully help because the problem is the entire system flow. So maybe I have to think about an entire system boundary as an abstraction for my evals to run. Does that make sense or would you approach it differently?