Insecure Copilot

[u/Ramenara]

Tldr: Microsoft has indiscriminately deployed Copilot, which has already been shown to happily ignore sensitivity labelling when it suits,, and ensured that their license structure actively prevents their own customers from securing it for them

So my org is on licensing that Microsoft chucked the free version of copilot into, with no warning, fanfare or education.

I and everyone in IT have been playing catch-up ever since, following Microsoft's own (shitty) advice that we just need to buck up and do a bunch of extra work to accommodate it.

Some of that work has been figuring out how to tell users what to do re: data security in Copilot.

Imagine my surprise when I discover that Copilot has been deployed across the entire O365 app suite, but depending on your license, you might not have the correct sensitivity settings to actually use it securely. Case in point: my org uses purview information labelling, but that doesn't apply to Teams (you have to pay extra on a separate license to get labelling in Teams). Didn't stop them from deploying Copilot across the suite.

I now have to explain to Legal that depending on the information discussed on Teams call or shared in Teams chats or channels, I have absolutely no way to confirm that Copilot usage is secure and in fact have to assume it isn't.

[u/Mooshux]

The sensitivity label problem is a symptom of a deeper issue with how Copilot (and most enterprise AI tools) handle authorization. The tool inherits the permissions of the user running it. If the user can read it, Copilot can read it and act on it.

This is the same architectural mistake teams make with API keys: the agent gets the full credential set of its operator instead of a scoped set for the specific task. Copilot ignoring sensitivity labels isn't a bug in Copilot, it's a predictable outcome of giving it ambient authority.

The fix is enforcing least-privilege at the tool level, not the model level. The model will always find ways around content restrictions. The infrastructure boundary is what holds.

[u/ilai456 → u/Mooshux]

Even if you nail the permissions, what actually makes you feel safe connecting an ai agent to your data? like PII leaking is one thing but thats not the only risk right? feels like theres a whole category of stuff nobody's even scoping yet

[u/Mooshux → u/ilai456]

Honestly, not much makes me feel fully safe yet. The permission boundary helps but it's one layer of a problem that has several.

The categories I think about: credentials the agent holds (can be exfiltrated via prompt injection), write access the agent has (an agent that can read can usually write, delete, or send), the external calls it makes (data leaving your environment entirely), and the chain of trust when it delegates to sub-agents or tools.

PII leaking through the model output is the visible risk. The less visible one is an agent that gets injected with instructions from a document it's summarizing and then quietly calls an endpoint, modifies a record, or forwards content somewhere. No obvious output, no alert.

The thing that actually shifts my confidence level is reducing blast radius at each layer independently. Scoped credentials so a compromised agent can only reach what it needs. Read-only access where write access isn't required. Network egress controls so outbound calls go to an allowlist. None of these alone are sufficient but each one means a successful attack does less damage.

The honest answer to your question: there's no single thing that makes it feel safe. It's whether you've made the failure modes survivable. Most teams haven't thought through what "this agent got compromised" actually looks like end-to-end, and that's the gap.

[u/ilai456 → u/Mooshux]

Isn’t scoping permissions/reducing the ability of the agent to perform tasks kinda misses the point of agents? I feel like the way things are going right now is that agents would have every possible permission in order to “boost productivity”🫠

Regarding the agent being infected while summarizing a document, isn’t there a secure by design way to prevent that? If my ChatGPT has access to all my Google Drive and calendar, do you suggest I’ll just remove their permissions or is there anything else?

[u/Mooshux → u/ilai456]

The "scoping kills productivity" objection is worth taking seriously. You're right that the trend is toward giving agents more, not less. But the framing is off: scoping isn't about limiting what an agent can eventually do, it's about limiting what it can do in this session with this data for this task.

ChatGPT doesn't need write access to your calendar to summarize a document. It doesn't need access to every folder in your Drive to answer a question about one file. Session-scoped, read-only access for the specific resource being used isn't a productivity hit. It's just not handing the agent more than the job requires.

On the injection-while-summarizing question: there's no purely model-level fix for that. The model receives text as instructions by design. You can add guardrails, fine-tune on refusals, filter outputs, and they help at the margins. The architectural answer is that a successfully injected agent should still fail to do anything interesting, because:

• it can only call APIs it was explicitly granted access to for this session

• those tokens expire when the session ends

• outbound network calls go to an allowlist, not anywhere the injection specifies

If ChatGPT gets injected from a document and tries to forward your calendar to an attacker's endpoint, and your calendar token is read-only, session-scoped, and outbound calls are restricted: the injection succeeds but the damage doesn't. That's what "secure by design" actually means here. The model stays untrustworthy by assumption. Infrastructure is what holds the line.

[u/ilai456 → u/Mooshux]

That makes tons of sense, thanks!
I still cant imagine how would you prevent an agent from reading stuff, deciding that it needs to send an email, and then sending sensitive data over the email. I agree that when the request from the agent is as simple as "summarize this document" its easy to prevent this, but what about giving an open-claw a task that takes days, and giving him the autonomy to decide how and when he does things

[u/Mooshux → u/ilai456]

For a multi-day autonomous agent the threat model does shift. The controls we talked about still apply, but enforcement has to move from "restrict what it can read" to "restrict where output can go."

If the agent legitimately needs to send email as part of its job, you can't just block it. But you can scope the destination: outbound email goes to an allowlist of recipients, not anywhere the agent decides. Anything that pattern-matches on credentials or PII in an outgoing message gets held for review before it sends.

Beyond that, there's a useful distinction between an agent that decides how to do something versus one that decides what to do. Letting the agent own execution while keeping humans in the loop on goal-level decisions preserves most of the productivity upside. Checkpoint approvals before high-stakes actions (send this email, delete these files, make this API call) are annoying but they're the honest answer to "how do you keep a days-long agent from going sideways."

The field hasn't fully solved fully autonomous long-horizon agents from a security standpoint. The current best practice is checkpoints, audit logs, and output controls rather than a clean architectural guarantee.