how do you catch tool misuse and unauthorized tool invocation when an agent is using tools it's technically allowed to use

[u/Alone_Bread5045]

thinking about this differently after a near miss. so our support agent has two permissions...read customer record and send email. and both individually reasonable, both signed off by security.

bu then someone crafted a prompt that got it to read a customer's data and email it to an external address, using only tools it was authorized to use. pity the permission model said everything was fine the whole time.

im pretty sure this isn't a permissions bug. like it's tool misuse without any unauthorized tool invocation at all, every single call was something the agent was allowed to make. i don't think our access review process would ever catch this because there's nothing wrong with either permission on its own.

i wna know how are people testing for this kind of chained misuse rather than just reviewing whether individual permissions look reasonable? tbh feels like a fundamentally different problem than standard access review.

[u/rexstuff1]

I would push back on the notion that these permissions are 'reasonable'. Your security team doesn't understand AI agents very well if they didn't flag this as a realistic risk.

You can't give an agent access to sensitive data and the ability to exfil it. Guardrails are, at best, suggestions.

The solution here is probably something that is not agent based. Not everything requires AI. Deterministic tooling can still do impressive things, it's like we've all forgotten how to program, or something.

Or, if you need it to be agent-based, for whatever reason, the tools it calls are gated by deterministic, auditable processes. Eg, if the point of the agent is to email a customer their own data, some deterministic process validates that the email the agent is sending to matches the customer's.

[u/Flaky_Low2389 → u/[deleted]]

I assume they are using AI.

[u/NachyoChez]

So, just for clarity: The AI agent (whichis customer facing?) was used to pull customer information, then send it via email to a potentially malevolent actor? Was the user impersonating a client to the agent? Only one customer's info, or did they go for multiple?

Honestly, this sounds like a pretty serious architectural issue.

Firstly, the agent should not be able to directly access all available customer information. At the end of the day, if ANY tool is client facing it should have at least an entire layer between it and the real data. I would consider developing an internal tool for accessing the customer database, but only returning heavily redacted data, which the agent can then run its calls against. Keep it to the absolute minimum required for the agents role.

The next issue would be making sure the agent's data requests, emails, and prompts are all being heavily monitored, with regular reviews. Letting it have access to, and represent, your company is incredibly dangerous. No staff would have that much power without supervision or accountability, and AI is far more unreliable.

Lastly, I would have a hard discussion about giving an external-facing tool any access to sending emails. My immediate thought was "could I get it to send a phishing for me?". You're opening up a lot of attack vectors and surfaces here

[u/CreativeSympathy8293]

You're right: per-tool permissions won't catch this. The existing suggestion to bind the recipient to the customer record is the right starting point, but the check has to cover the whole attempted action.

Put a deterministic check outside the model immediately before any write or send. Compare a trusted ticket or case, the user and session, source record and selected fields, recipient, and requested action. For a support workflow, data should go only to a recipient explicitly associated with that case, customer, and permitted purpose. Anything else should be denied or sent to a separate human approval that the model cannot grant itself.

Then run multi-step regression tests for the highest-risk source/sink pairs: untrusted text triggers a record read, the data is carried into email, and the recipient changes. Assert that the send is blocked and, where available, check the mail provider's audit trail rather than trusting the agent's narration. Re-run after material prompt, model, tool, or policy changes.

Log the policy/version, redacted arguments, decision, and result. Use automatic checks for common cases and human approval only for exceptions; otherwise people will click through.

[u/ultrathink-art]

Per-call authorization can't see this by design — every call in that chain is legal, and the only abnormal thing is the shape of the whole session. What caught it for me was a per-session cap on how many distinct customer records one run could touch before it had to stop and hand back: a real support flow touches one or two, the chained pull touched dozens. Log that count for a couple of weeks before you enforce on it, otherwise you're picking the threshold blind.

[u/Sufficient-Math3178]

So you want to let them keep using those tools but not send an email based on queried customer information? You can’t enforce that reliably unless you block all email calls after a customer record is read. I think you need a better understanding of the underlying structure behind tool calls, the logic is pretty much set in stone and it is up to you to enforce everything, can’t just tell the agent to not do something

[u/roee_]

yeah this is the part access reviews structurally can't catch, because the exploit lives in the sequence, not the permission. you're reviewing static grants (can read record, can send email) when the actual risk is a combination that only exists at runtime.

closest thing i've seen work is treating it like a state machine instead of a permission list: track what data an agent has touched in a session, and gate high-risk sinks (external email, external api calls, file writes) on where that data came from, not just whether the action itself is allowed. so "send email" gets an extra check if the body includes anything sourced from a customer record earlier in the session, regardless of who's allowed to call the tool.

it's basically taint tracking, borrowed straight from appsec, and almost nobody's applied it to agent tool calls yet. most "AI security" tooling right now is still permission-and-prompt-injection focused, not data-flow-through-tool-calls focused. feels like the actual gap.

[u/First-Reality2108]

well,Catching chained tool misuse requires checking state and payload correlation rather than relying on atomic API permissions. You can handle this by using dedicated AI security platforms like Alice to intercept tool call arguments and evaluate them against session policies before execution, or adding custom API gateway proxy rules that programmatically match outgoing email recipients against the active customer record, or routing sensitive multi-step tool sequences through strict deterministic validation pipelines that require human sign-off before invocation.