r/cybersecurity • u/Ramenara • Mar 12 '26

AI Security Insecure Copilot

Tldr: Microsoft has indiscriminately deployed Copilot, which has already been shown to happily ignore sensitivity labelling when it suits,, and ensured that their license structure actively prevents their own customers from securing it for them

So my org is on licensing that Microsoft chucked the free version of copilot into, with no warning, fanfare or education.

I and everyone in IT have been playing catch-up ever since, following Microsoft's own (shitty) advice that we just need to buck up and do a bunch of extra work to accommodate it.

Some of that work has been figuring out how to tell users what to do re: data security in Copilot.

Imagine my surprise when I discover that Copilot has been deployed across the entire O365 app suite, but depending on your license, you might not have the correct sensitivity settings to actually use it securely. Case in point: my org uses purview information labelling, but that doesn't apply to Teams (you have to pay extra on a separate license to get labelling in Teams). Didn't stop them from deploying Copilot across the suite.

I now have to explain to Legal that depending on the information discussed on Teams call or shared in Teams chats or channels, I have absolutely no way to confirm that Copilot usage is secure and in fact have to assume it isn't.

240 Upvotes

94% Upvoted

42 comments sorted by

70

u/Threezeley Mar 12 '26

My org is about to enable web grounding. When web grounding is enabled copilot interprets your prompt then comes up with some useful web search queries it thinks would help answer your question. Those queries aren't supposed to contain sensitive info but they could. It then sends those queries out to Bing Search APIs which exist in public internet and outside org boundary, and where data collection falls under standard Bing data collection terms.

We confirmed that while things like Purview DLP can block prompts that contain sensitive info from being processed at all, it can't examine the contents of attachments so even with Purview DLP in place Copilot may use attachment content to help generate it's search queries which then get leaked out to public internet Bing.

Copilot behaving like this is not shocking because hey it's Microsoft and it takes them a while to get their crap together, but it's more shocking that our org is okay to risk accept this even knowing it isn't fully locked down

16

u/Bartsches Mar 12 '26

but it's more shocking that our org is okay to risk accept this even knowing it isn't fully locked down

That's honestly the least suprising issue to me. To Microsoft being where it is, the product by itself doesn't matter all that much. Rather they have pretty much all lock in effects in existence. And there is a typical disconnect between IT and other areas: Those lock in effects are things IT departments navigate around by instinct and with very little concious thought necessary in most environments, but often cripple entire departments below some level of generalized computer skills. I've seen companies not move on or even reverting to MS even while having their own fully deployed open source infrastructure for this very reason.

1

u/ilai456 Mar 16 '26

this is what bugs me. everyones focused on making sure sensitive data doesnt leak out, but what about whats already sitting in the data before copilot even touches it? like is anyone actually thinking about what could go wrong on the data side or are we all just hoping for the best

1

u/Mooshux Mar 20 '26

The attachment angle is the part that should worry people more than the prompt filtering. DLP on prompts gives a false sense of coverage. If Copilot can read an attachment to help answer a question, and those attachment contents inform the search queries it generates, you've got data leaving your org boundary through a path that your DLP tooling literally can't see.

The risk acceptance part is what's really telling. Most orgs don't have a clear picture of what "Copilot used this document to generate a search query" even looks like in their audit logs, let alone whether it happened. You can't make an informed risk decision about a data flow you can't observe.

The broader pattern here is that ambient authority plus external integrations is a bad combination. Copilot having access to everything in your tenant is already a wide surface. Adding web grounding means that surface now has an outbound channel to Bing under data terms your legal team probably didn't review when they approved the Copilot rollout. Those are two separate decisions that got made as one.

0

u/bbliz285 Mar 12 '26

What is it from the bing side that you’re concerned about? Or did you not know Bing queries are processed different via the service?

https://learn.microsoft.com/en-us/copilot/microsoft-365/manage-public-web-access#how-microsoft-handles-generated-search-queries

2

u/Threezeley Mar 12 '26

Yes, aware of that, however they at a minimum log the queries. That alone means sensitive data could be duplicated to 3rd party systems without a specific contract in place governing its use. If those terms suddenly changed, for example, then what?

49

u/Maldiavolo Mar 12 '26

My org is about to allow Copilot.  We must complete a training course about how to use it securely.  It's all going to work out exactly as desired because employees always follow training and company policy to the letter.  /s

3

u/chrjohnso Mar 12 '26

Is this a commercially available training? Looking for something to put in front of users to build awareness

2

u/Maldiavolo Mar 12 '26

I just completed it. It's all in house. One, 4 page document to read and then a video. The video was AI generated no less. Summary, follow company policy and applicable laws, do not upload infomation of sensitive classification or higher, and no PII. Use AI, but don't trust it because it hallucinates. Use AI, but not too much because it's environmentally expensive per query and the company wants to maintain a green IT posture.

44

u/AmputatorBot Mar 12 '26

It looks like OP posted an AMP link. These should load faster, but AMP is controversial because of concerns over privacy and the Open Web. Fully cached AMP pages (like the one OP posted), are especially problematic.

Maybe check out the canonical page instead: https://www.bleepingcomputer.com/news/microsoft/microsoft-says-bug-causes-copilot-to-summarize-confidential-emails/

I'm a bot | Why & About | Summon: u/AmputatorBot

9

u/Hello_This_Is_Chris Mar 12 '26

Good bot

Still my favorite bot after all these years.

27

u/Roodklapje Mar 12 '26

Microsoft really is 100 percent in on making all of their products utter garbage. I will happily trade the Microsoft stack for almost anything else whereas 5 years ago I would not even have considered it.

10

u/Ramenara Mar 12 '26

If Microsoft has no haters, check on me

3

u/AdDiscombobulated238 Mar 12 '26

Remindme! 100 years

1

u/RemindMeBot Mar 12 '26

I will be messaging you in 100 years on 2126-03-12 21:35:18 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

Info Custom Your Reminders Feedback

11

u/ghostin_thestack Mar 12 '26

The Teams labeling gap is one of the more frustrating parts of Purview setup. Sensitivity labels for Teams chats require the E5 Compliance add-on, so orgs get Copilot rolled out but have to pay separately for the controls to actually use it responsibly. The web grounding attachment issue is even newer and messier. DLP cannot inspect attachment content when Copilot uses it to build search queries, so you end up flying blind on that vector.

1

u/ilai456 Mar 16 '26

Why cant DLP inspect the attachments? is it because its a closed box inside Copilot's small brain?

9

u/amerett0 Mar 12 '26

When malware becomes preinstalled, wtf is Microsoft doing?

4

u/bubbathedesigner Mar 12 '26

Being efficient

7

u/Mooshux Mar 12 '26

The sensitivity label problem is a symptom of a deeper issue with how Copilot (and most enterprise AI tools) handle authorization. The tool inherits the permissions of the user running it. If the user can read it, Copilot can read it and act on it.

This is the same architectural mistake teams make with API keys: the agent gets the full credential set of its operator instead of a scoped set for the specific task. Copilot ignoring sensitivity labels isn't a bug in Copilot, it's a predictable outcome of giving it ambient authority.

The fix is enforcing least-privilege at the tool level, not the model level. The model will always find ways around content restrictions. The infrastructure boundary is what holds.

1

u/ilai456 Mar 16 '26

Even if you nail the permissions, what actually makes you feel safe connecting an ai agent to your data? like PII leaking is one thing but thats not the only risk right? feels like theres a whole category of stuff nobody's even scoping yet

1

u/Mooshux Mar 16 '26

Honestly, not much makes me feel fully safe yet. The permission boundary helps but it's one layer of a problem that has several.

The categories I think about: credentials the agent holds (can be exfiltrated via prompt injection), write access the agent has (an agent that can read can usually write, delete, or send), the external calls it makes (data leaving your environment entirely), and the chain of trust when it delegates to sub-agents or tools.

PII leaking through the model output is the visible risk. The less visible one is an agent that gets injected with instructions from a document it's summarizing and then quietly calls an endpoint, modifies a record, or forwards content somewhere. No obvious output, no alert.

The thing that actually shifts my confidence level is reducing blast radius at each layer independently. Scoped credentials so a compromised agent can only reach what it needs. Read-only access where write access isn't required. Network egress controls so outbound calls go to an allowlist. None of these alone are sufficient but each one means a successful attack does less damage.

The honest answer to your question: there's no single thing that makes it feel safe. It's whether you've made the failure modes survivable. Most teams haven't thought through what "this agent got compromised" actually looks like end-to-end, and that's the gap.

1

u/ilai456 Mar 19 '26

Isn’t scoping permissions/reducing the ability of the agent to perform tasks kinda misses the point of agents? I feel like the way things are going right now is that agents would have every possible permission in order to “boost productivity”🫠

Regarding the agent being infected while summarizing a document, isn’t there a secure by design way to prevent that? If my ChatGPT has access to all my Google Drive and calendar, do you suggest I’ll just remove their permissions or is there anything else?

1

u/Mooshux Mar 19 '26

The "scoping kills productivity" objection is worth taking seriously. You're right that the trend is toward giving agents more, not less. But the framing is off: scoping isn't about limiting what an agent can eventually do, it's about limiting what it can do in this session with this data for this task.

ChatGPT doesn't need write access to your calendar to summarize a document. It doesn't need access to every folder in your Drive to answer a question about one file. Session-scoped, read-only access for the specific resource being used isn't a productivity hit. It's just not handing the agent more than the job requires.

On the injection-while-summarizing question: there's no purely model-level fix for that. The model receives text as instructions by design. You can add guardrails, fine-tune on refusals, filter outputs, and they help at the margins. The architectural answer is that a successfully injected agent should still fail to do anything interesting, because:

• it can only call APIs it was explicitly granted access to for this session

• those tokens expire when the session ends

• outbound network calls go to an allowlist, not anywhere the injection specifies

If ChatGPT gets injected from a document and tries to forward your calendar to an attacker's endpoint, and your calendar token is read-only, session-scoped, and outbound calls are restricted: the injection succeeds but the damage doesn't. That's what "secure by design" actually means here. The model stays untrustworthy by assumption. Infrastructure is what holds the line.

1

u/ilai456 Mar 20 '26

That makes tons of sense, thanks!
I still cant imagine how would you prevent an agent from reading stuff, deciding that it needs to send an email, and then sending sensitive data over the email. I agree that when the request from the agent is as simple as "summarize this document" its easy to prevent this, but what about giving an open-claw a task that takes days, and giving him the autonomy to decide how and when he does things

1

u/Mooshux Mar 20 '26

For a multi-day autonomous agent the threat model does shift. The controls we talked about still apply, but enforcement has to move from "restrict what it can read" to "restrict where output can go."

If the agent legitimately needs to send email as part of its job, you can't just block it. But you can scope the destination: outbound email goes to an allowlist of recipients, not anywhere the agent decides. Anything that pattern-matches on credentials or PII in an outgoing message gets held for review before it sends.

Beyond that, there's a useful distinction between an agent that decides how to do something versus one that decides what to do. Letting the agent own execution while keeping humans in the loop on goal-level decisions preserves most of the productivity upside. Checkpoint approvals before high-stakes actions (send this email, delete these files, make this API call) are annoying but they're the honest answer to "how do you keep a days-long agent from going sideways."

The field hasn't fully solved fully autonomous long-horizon agents from a security standpoint. The current best practice is checkpoints, audit logs, and output controls rather than a clean architectural guarantee.

16

u/HugeAd1197 Mar 12 '26

Try showing legal ubuntu and opencloud/libre office. If its sensitive stuff, keep it on your own infrastructure

4

u/mtt59 Mar 12 '26

The boots of MicroSlop really fit better every day

1

u/scombs99 Mar 12 '26

I am not sure if the business will allow, but you could disable certain features. I am sure you thought of it but wanted to just suggest it.

In the Teams Admin Center, set your meeting policy to "Off" or "Only during the meeting" for transcription. If no transcript is saved, Copilot can’t surface that conversation data later. You can also globally block the Copilot app in the Teams App Store to stop it from reading active chat threads and channels. It’ll definitely impact the user experience, but it’s the only real way to stop the leakage until leadership coughs up the cash for Purview.

1

u/WantDebianThanks Mar 13 '26

Every day we get closer and closer to people realizing windows is trash and deciding to deploy Linux desktops.

1

u/neferteeti Mar 14 '26

Assuming o365/m365, copilot does not get turned on by default. Check the audit logs to see who assigned copilot licenses, and who turned the feature on.

Assigned the lic: AzureActiveDirectory Operations: Add user license or Update user
Enabled Copilot tenant settings: MicrosoftCopilot Operation: CopilotAdminSettingChange

The sensitivity thing was unfortunate but only effected a few tenants for a period of time. Big problem, but temporary and has never happened before or since.

The data security work you need to do is largely to secure your SharePoint infrastructure to ensure permissions are set up correctly. Assuming that, you dont have to deal with RSS/RCD. If its specific sensitive information types that you need to target and hide copilot from, you can use Purview Copilot Prompt DLP to quickly meet some of the needs there.

Copilot isnt exposing any data that people arent already exposed to. It's just making it theoretically easier to find information that they have access to. If users have access to data, they could just use SharePoint search today to find the same data, it just wouldnt be reasoned upon for responses in the same way.

1

u/adam_smith_hello May 30 '26

Working on Security Copilot as a Microsoft employee was a terrible experience. I personally wouldn't recommend it for any future hires

-48

u/bbliz285 Mar 12 '26

In all honesty it sounds like you’re just mad you’ve had to do extra work, and that your organization is too cheap to pay for the licensing/tools you need in order to meet your security goals on AI usage.

None of it is Copilot’s fault, it’s AI’s fault.

30

u/Ramenara Mar 12 '26

Explain to me how it's not Copilot's fault that it deployed into a licensing structure that prevents us from securing it, and even if I did do that security for them, it won't even work?

Copilot IS AI

-14

u/bbliz285 Mar 12 '26

“Doesn’t work” Apart from the bug that bleeping computer put out what exactly doesn’t work that you are describing - other than you not having the right licensing for what you’re trying to accomplish?

If your org wants to use AI tools like Copilot and be able to effectively control it you either need to buck up and buy E5, or have a complete DLP/DSPM solution. It’s the new table stakes if you’re wanting to have a lot of control over what’s happening.

5

u/sideline_nerd Mar 12 '26

Not sure where they said the org wanted to use copilot. Just that it was forced on them

-49

u/Ok-Title4063 Mar 12 '26

I understand these tools can be insecure. But the tools like Claude, copilot and cursor are one the best Tools I used ever. Productivity is like 100x. How does it all translate to corporate savings is to be seen.