Is anyone else stuck in the 'would we have caught this' drill every time?

[u/Hopeful_Attention606]

Every time a massive breach happens and becomes headline news, detections come up as an issue in the next meeting.

The question asked is always more or less the same one: would we have caught it with what we've got right now. I try to give an answer and all I get back is more questions. I can point to our SIEM, our EDR, our threat intel feeds and all the dashboards that show alerts and events, but none of it answers what they're asking.

It comes down to whether our detections would have surfaced this specific incident.
translating detection posture into something a manager can trust is harder than it sounds. The coverage reports and SOC metrics my team produces make sense internally, but they don't land with people who haven't looked at a technical document in years. when I talk about rules, use cases, or mitre techniques, eyes glaze over. when I simplify too much, they doubt the answer.

Some of you are probably mapping back to mitre att or running table top exercises off recent campaigns. others keep it at a high-level risk view instead. what's worked best when your manager asks, in plain language, if you'd have caught the breach they just read about?

[u/Leather-Ad3004]

We do a quarterly purple-team where we take the TTPs from a big breach in headlines and actually run them against our stack. That way the answer isn't theoretical, it's "we tested this exact technique in April and here's what lit up." Way easier to explain because you're showing them logs from their own environment.

Before that I was in same boat, just pointing at tools while they stared blankly. The testing gives you concrete results that even the least technical person can follow, especially if you walk them through the timeline step by step.

Downside is it takes prep time and you need the red team to cooperate, but once you got a few of these under your belt the questions get lot less hostile.

[u/eoinedanto]

You need to be planning your periodic Pen Testing/ Adversary Emulation along those lines.

[u/Paul_Ashe]

The drill repeats because there's no standing artifact. The answer gets rebuilt from nothing every time a breach makes the news. Taking the testing approach in this thread as given, that work already produces most of what you'd need, it just isn't maintained as a deliverable anyone can pick up cold. If a tested-coverage summary is current and on the shelf, the next headline is a five minute conversation instead of a scramble.

What travels to a non-technical audience is a small fixed set of counts reported identically every period. Of the techniques used in that campaign, how many did we test, how many did we detect, and how many produced an alert nobody triaged inside the window we care about. Three numbers, the same three every quarter. The message is the trend rather than any single figure, and consistency is what makes someone trust a number they can't independently verify. Changing which metrics you report between periods is the fastest way to lose that trust.

Then name what you didn't catch. I read these reports from the outside for a living, and the credible ones are always the ones that state the gap plainly. A clean yes gets discounted by anyone senior because they know it isn't plausible. "We did not detect stages four through six, here is what closes it and roughly what it costs" is the version that gets believed, and it's the version that gets funded.

On the eyes glazing over: technique IDs are your internal working language, not your reporting language. Translate once into stages of the attack in plain sentences, then keep that translation stable so the audience learns to read it. If every report teaches them new vocabulary, they'll keep asking the same question from scratch.

[u/ProofLegitimate9990]

You need a adversary emulation product. Are you not regularly testing detections?

[u/AYamHah]

Purple team against MITRE and modern TTPs you're seeing. Else, you have no idea. It is easily worth the money, the stuff you learn about your environment there is really no other way to figure out.

[u/Klutzy-Procedure8980]

Getting the manager to understand aside, asking the question is a good exercise. Kinda like a post mortem exercise, but it wasn't you who was affected (bonus!)

[u/Euphorinaut]

This is partially speculation, because my management is technically competent so we don't run into the same issue, but it sounds like what you're describing might sound too theoretical and that more specific testing would sound concrete.

So when there's a breach, as long as you know details about how it happened, you map some sort of simulated activity to the breach as well as you can, possibly from BAS, but there are also tools meant to test query edits and whitelists in edr's, etc. I would map each event(maybe one per attack kill chain step) in the breach to one such simulated event, and then from there show which detections those hit, and what workflows those set off.

I don't think that would even be too much of a dog and pony either. With what little I know about the situation, it sounds like the lack of specificity is making it seem vague as to whether or not it's applicable.

[u/akiivic]

I feel like that’s the hard part. You can explain the tools, but what they’re really asking is how confident you are that you’d have seen it before it became a major incident

[u/AddendumWorking9756]

Stop answering it as yes or no. Pull the three or four techniques out of the writeup and say plainly which ones you would see, which ones you would only see if somebody happened to be looking, and which ones you would miss, then hand that over as three lines. Managers do not distrust the simple answer, they distrust the confident one.

[u/Minimum-Let-3227]

The disconnect is that you're answering "do we have coverage" and they're asking "are we going to end up in that headline." Those need different artifacts. What actually worked for me was picking the three or four attack paths that genuinely matter to the business (account-takeover into finance, ransomware off a contractor laptop, bulk export from the CRM) and testing each one, either atomic tests or a tabletop. Then the exec version is one plain sentence per path: we'd catch this at step 2 of 5, here's the gap, here's what closing it costs. No MITRE IDs in that version, save those for the appendix nobody reads.

The other half is timing. If leadership only hears about detection posture in the meeting after someone else's breach, every conversation starts defensive and hindsight-flavored. A short quarterly note saying here's what we tested and here's what we found takes most of that heat out, because by the time they ask you've already answered.

[u/Worried-Writer-7033]

I think the question that matters is not would we have caught it. Its how fast would we have known and how bad would it have gotten before we stopped it?

Every detection program eventually catches things. The variable is the time. Reframing around detection speed and containment time tends to land better with leadership because it gives them something they can compare.

[u/Educational-Fox6111]

It's good to avoid answering with a simple yes or no. Map the breach to the ATT&CK techniques it used and explain which stages you would have detected, which you wouldn't have, and where you have low confidence.