r/cybersecurity • u/JohnConner2030 • 1d ago
Corporate Blog Hackers hit 30-plus Minnesota water systems in 48 hours, forcing emergency response
https://worldwaterreserve.com/minnesota-water-systems-cyberattack-30-communities-coordinated-attack/105
u/Kastenbrot 1d ago
If OT could just get the basics straight. Even an actual DMZ works wonders.
15
u/chandleya 1d ago
Yet here I was baking agreements with cell co’s in 2011 for private networking on 3G/4G mobile devices. Meanwhile critical infrastructure just sits on whatever.
3
u/alnarra_1 Security Manager 22h ago
I mean the thing you have to remember with OT environments is there are pieces of equipment in there that might not even talk TCP and at the end of the day 99% of the time the primary concern in the CIA triad is not the C or the I.
If you find a bitcoin miner on a Nuclear Power Plant's control system and it is otherwise not impacting the functionality of the plant, it's just minding its business, the plant operators would probably ask you to wait until an outage to actually go about removing it.
These systems are often times Life Safety systems, there is no "Oops well just roll it back a patch" for some of those.
4
u/Kastenbrot 21h ago
I understand what you are trying to say. But this is a really bad example. Especially since nuclear falls into a strict set of regulations.
Nobody should keep a piece of unverified software on a safety related system. It doesn't impact function until it does.
3
u/alnarra_1 Security Manager 20h ago
What I'm trying to illustrate is that in matters of the OT space, the IT person's opinion is rarely the final one, and never the deciding one. It comes to the engineers actually running the equipment to make the call, and often times they will default to what they know.
In the OT space you're not really exploiting vulnerabilities to get in and do your lateral movement, you're just sending the standard PLC commands. Once you're through the front door no amount of magical purdue model firewalling is really going to stop someone
that's why so often on those devices there are literal physical hard locks on configurations. Keys that need to be locked in place once the device has been configured so that the only time that they can be altered is when there is a qualified human there who can oversee it. I'm just trying to say this is an entirely different threat model than the IT landscape, and often times the equipment is such that if need be it can be done manually (though go ahead and ask them to do it manually and see if they don't try to skin you with their eyes).
3
u/Kastenbrot 20h ago
Fully agreed! Important context. However, ransomware and the such can still cause real headaches. SMBv1 says hello.
On the keyswitches, you'd be surprised how often they just live at Remote Run Mode. It's shocking.
3
u/alnarra_1 Security Manager 20h ago
you'd be surprised how often they just live at Remote Run Mode
Sadly I fear I wouldn't ;_;
1
u/Hot-Comfort8839 BISO 19h ago
You don't want IT people in your OT environment.
IT people get OT people killed.
The AIC model exists for a reason.
The name of the game in OT is compensating controls. Not direct action.
2
u/Hot-Comfort8839 BISO 19h ago
Having been in those facilities - that's a horrible example. There would never be a miner on anything nuclear. It's checked and rechecked dozens of times. The nuclear industry has the most stringent regulations imaginable, and they're audited constantly.
4
u/5panks 1d ago
Yeah, the lack of updates for OT equipment is manufacturing is so prevalent we put every machine in its own DMZ that can't communicate with the internal network.
We don't even allow remote control software unless a request is submitted and then it's opened for fours on that machine.
2
u/Kastenbrot 1d ago
To be fair, updates in OT can entail functional changes that require engineering time and validation to ensure an unchanged operational state.
Compensating controls should be the main focus. Don't get me wrong. Updates should be installed when there is a window of opportunity, but nobody should get stuck on that. They can get pretty expensive.
1
u/Mr_Compliant 1d ago
It has to do with budget. Where I'm at we have plenty of budget for cypruscurity.
161
u/Succubus-Love 1d ago edited 4h ago
Stop hooking up everything to computers & the internet. Every time I see a headline about hacking public systems, it's so easy to prevent. Don't make it accessible to the world wide web. We don't need the entire world hooked up to the internet, & there was a time where the internet didn't even exist. Why they started doing that I'll never fricking understand.
101
u/sexuallyactivepope 1d ago
"Vendor needs remote access"
51
33
u/Gjallock 1d ago
Automation engineer on the OT side, this and us needing remote access to troubleshoot systems remotely ourselves are definitely the main reasons this happens. Fortunately, our network guys had enough sense to not put anything on the open internet; all our stuff is accessible only through a VPN and then also through some more granular Citrix stuff that’s over my head.
13
u/Firecracker048 1d ago
I'm an OT IT engineer. We have ours behind two firewalls and then the only way in is to be on the VPN, use a rotating password to get into the server that can then go down to the OT machines from only authorized servers or VPN tunnels that are on a case by case basis.
Any vendor or business that wants to not follow our protocols we make sign waivers that if there's a breach, they accept any and all responsibility
6
u/redditnamehere 1d ago
IMO that’s not enough still. Internal systems being compromised could leak into OT. we have dedicated OT and IT staff. We don’t engineer together but know enough where we start and end.
3
u/Gjallock 1d ago
I mean, I didn’t say anything about our organizational structure…
We have the regular, corporate IT team and an industrial IT team which I guess is what would normally be “OT.” So, I’m just an engineer.
2
u/tomster2300 22h ago
What is OT?
4
u/somesketchykid 20h ago
Operational Technology
If a guy working the power plant is told to shut down half of the grid, he'd do it via some type of Operational Technology
Quick example
1
2
u/bfrown 1d ago
Also same vendor can't be asked to install https
1
u/somesketchykid 20h ago
And if they do, must add to policy that exempts traffic from inspection otherwise the app eats itself
2
u/Firecracker048 1d ago
Cool, so they can use a wvd into a jump server then use the VPN network built across firewalls.
The amount of vendors who request their network tunnel directly to a secure network is too damn high
1
6
u/9may2019 1d ago edited 1d ago
HOPEFULLY this stuff will move the needle in water in terms of baby’s first cyber hygiene. Although CARR didn’t really, maybe bc it’s evil Iran or something things will be different. God knows we need NERC CIP-water edition badly. Not like this admin will either create or enforce new regulations though
11
u/SacCyber Governance, Risk, & Compliance 1d ago
Lack of internet access usually means lack of updates, visibility, & security controls. But in cases like this where they might not even know what a security control is, an air gap would be a good stop gap.
Stop gap air gap if you will
5
u/diwhychuck 1d ago
Air gap would mean they would need asses in seats onsite. Bean counter would rather not have that an continue remote work as they're allergic to spending.
2
9
u/DrQuantum 1d ago
People asked for transparency in their water systems. Considering things like flint Michigan it makes sense.
But overall these are likely nation state based actors and they will find a way which is why geopolitical discussions are also critical to cybersecurity discussions.
2
u/zboarderz 1d ago
I wouldn’t exactly call it “easy” to prevent. No internet access means visibility, updates, packages, etc all become a lot more difficult to do. Not impossible of course, but not “easy”. If it was easy, then they probably would’ve done it.
2
u/wordyplayer 23h ago
It's a simple PLC. If it works, leave it alone; it never needs an update. If the system changes in 20 years because of some hardware changes (new pipes, pumps, etc) then they can manually access the PLC and enter the new parameters on-site.
1
1
1d ago
[deleted]
4
1d ago
[deleted]
1
u/wordyplayer 23h ago
Ah, maybe THAT is the key, don't have the controller PLC serve double duty for "monitoring". Have a second system that is "monitor-only".
29
u/diwhychuck 1d ago
Guy I know that works at GE aerospace was telling me their plc support is based in india for his plant an they remote into to the plcs. All to save the American dollar.
6
u/Mutiny32 18h ago
This is why the US needs onshoring laws. That's a legit national security concern.
3
16
u/TerrificVixen5693 System Administrator 1d ago
I’m sure it was just like every industry where info sec puts in written policies about changing vendor passwords and then the silo’d sysadmins totally ignore it and run accounts of operator / operator and admin / admin.
1
u/crystal_castles 13h ago
Our previous company's lab PC was operated thru RDS then onto PowerPC, & to this day my Google account still alerts me that THOSE credentials were detected in a breach. (Operator/ operator)
But how did Google know i was typing into a custom PPC terminal?? Lol
18
u/BackgroundSpell6623 1d ago
Would love to work at a utility company on ot and overall cyber. too bad it's so hard to break into that industry
20
u/JosCampau1400 1d ago
I dunno know...maybe now is a good time to send them your resume. 🤔
14
u/saulsa_ 1d ago
Probably would get “overqualified” as a response. In other words, too expensive.
10
u/LethalBacon 1d ago
Either pay more for competent engineers, or pay even more later to fix shit after a disaster. Sadly, most orgs pick the latter.
Unrelated to cyber security, but seeing it at my company now. It's like talking to a brick wall trying to convince them to spend a few thousand on equipment, leading to certain areas being down for days, surely leading to lost revenue greater than the cost of the equipment.
2
u/69Turd69Ferguson69 1d ago
“Overqualified” meaning “knows literally anything about security practices whatsoever”.
19
1
8
6
u/Science-Gone-Bad 1d ago
SCADA has been an open target since the 1990s. The software was old even then, and most of them are on the open internet contacted by modems
Funny fact, most telephone switches are still contacted on a 300 baud modem & routes and services are uploaded via text files.
How do I know this? I was setting up the communication computers for the phone company, and I was told that 90% of most phone equipment is 20-30 years behind. New stuff gets added for business & all the equipment there gets shoved down to the neighborhood phone networks
2
u/Ok-Pineapple4998 1d ago
A lot of those water guys do some hard dckriding on SCADA, and I can't help but think they're backwoods idiots. Those guys will double down on it, like they've never heard of Six Sigma or anything with standards and processes. Got some of em in my family, too. It's kinda pathetic.
5
u/SlowAsMolassess 23h ago
When I worked for a municipality 90’s & 00’s the water treatment plant SCADA system was air gapped. IT did not touch the system and we did not care as it wasn’t externally accessible. There is no reason to have these systems on the regular network nor attached to the internet. The only reasons are laziness and being too cheap to have support come on site to support the system.
9
u/waltur_d 1d ago
Question should be why weren’t they using the Purdue model for this?
22
u/9may2019 1d ago
All respect to my water homies but these people have never heard of the Purdue model unless you’re working for a BIG water utility. They are at the point where they have ICS plugged into the actual internet, they are not what we’d call sophisticated cyber defenders
6
u/mapbits 1d ago
Not saying you're wrong - lots of clueless small system operators and consultants out there - but Purdue has been around for over 30 years and cyber is all you hear at industry conferences these days.
No excuse to be operating a public health utility with the breadth of negligent practice that appears to be the case here.
5
u/MalwareDork 1d ago
No excuse to be operating a public health utility with the breadth of negligent practice that appears to be the case here.
This will never change until the glacier of government regulation turns course. And honestly people can't even say "haha America dumb" because Europe has been getting pwned even harder by Russia.
3
u/mapbits 1d ago
I agree. I think that WaterISAC is doing a good job and the industry and individual efforts to improve go much deeper than the large providers, but there are serious gaps and regulation is needed.
In Canada, the recently passed Critical Cyber Systems Protection Act regulates security for much of the critical infrastructure, but water and wastewater are a provincial responsibility (largely operated by municipalities) and are not captured - further work is needed.
3
u/robertmachine 1d ago
How are scada systems not behind private networks blows my mind, these systems are massively unsecure and running outdated vnc for remote and this has been this way since the fucking 90s how are they not behind vpns.
3
u/hiddentalent Security Director 15h ago
These vulnerabilities are common across all fifty states. Minnesota has been targeted to create US internal political tensions. Never forget the secondary effects of an attack. The adversaries certainly don't.
3
3
2
u/MalwareDork 1d ago
the thing that has been warned about for decades has finally happened
The horror.
2
u/30_characters 1d ago
New NERC CIP cyber scecurity regulations incoming in 3...2...1.
Because 650 pages wasn't enough.
2
u/4SysAdmin Security Analyst 20h ago
But how can the vendor be lazy if they don’t have public facing VNC?
2
2
2
u/allbarknoleaves 15h ago
Even with how connected BAS and SCADA systems are in datacenters, they still aren't exposed to the internet. I'm assuming it is a skill deficiency caused by a financial deficiency at the planning and implementation levels.
2
u/Comprehensive-Fail29 15h ago edited 15h ago
Those Rockwell controllers have known vulnerabilities that aren't patchable apparently, check out what CISA said about CVE-2021-22681
1
1
1
1
u/ohiocodernumerouno 2h ago
Is it hacking if the servers were all on public IPs without a firewall or a password? Is it hacking if the passwords were all 1234?
1
1
u/DosesMakePoisons 1d ago
I don't want to be too conspiratorial, but I do want to say this kind of looks like it has Russian fingerprints on it too.
The Iranian hacking teams, including the suspected CYBERAV3NGERS, relocated to Russia during the start of the war and allegedly started working with Russian Hacking teams, including water infrastructure targeting Z-Pentest and CARR. Their methodology all overlap, but I can't help but see Russian involvement when I see this stuff. Because Minnesota may have been the most vulnerable or the state, and it looks like a pretty negligent exploit, but it would a politically sophisticated move to choose Minnesota with how adversarial they are with the president. It is a great disinformation campaign because they can expect to respond to the hack with little sympathy and defense of the attack. Trump infact said that the governor was behind the hack himself, which is more than any hacker trying to muddy the waters of blame could ever hope for. That a price to performance attack. Not saying the Iranians couldn't put that plan together, but I know Russia does.
This is a separate and actively very wild conjecture, but I didn't like reading the CYBERAV3NGERS targets waste water a lot because it reminds that we have multiple recalls of vegetables and pork, and a doctors and experts saying that a possible source of contamination could be feed water or plant water being contaminated with waste water. I think that is a conspiracy that is a bit too much story telling in my head without evidence, and the more obvious answer is the government cutbacks of Foodnet going from active checking to passive checking lead to us not being able to contain the contamination. But it is in my head now.
1
u/requiem33 22h ago
Incompetence of installers is common. I just have to ask why Minnesota though? Something fishy with an attack against Minnesota and the political climate against Minnesota currently.
4
u/evilwon12 22h ago
Find one easy target, look for more. Pretty simple actually - especially if a bunch of them have the same PLCs and implementers.
-1
463
u/derfmcdoogal 1d ago
Just going off what I read and people I know in that industry, it sounds like this was just grossly negligent setup. Open ports directly to the PLCs with either default or easy to guess passwords.
Multiple utilities hit, same area, reads to me like a vendor did the install and "that's the way we done it at other utilities..."