CISA released an urgent message warning water utilities to disconnect their logic controllers from the internet in the face of rising cyberattacks.
The hacks target internet-facing programmable logic controllers (PLCs) that control equipment and allow machinery to communicate. They monitor and control the water pressure, chemical dosing, and other factors to ensure the water is safe.
Many of the PLCs are apparently open to the internet and use default credentials, allowing a remote attacker to easily take them over.
I assume it is this CISA: https://en.wikipedia.org/wiki/Cybersecurity_and_Infrastructure_Security_Agency
It usually happens in 2 steps.
First step is that everything gets connected to the LAN and you can only access the PLC network from within the building’s network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.
The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: “Just make it work”. So IT punches a hole in the firewall, adds a NAT rule, and job done.
Except you skipped the part where IT is ridiculously underfunded and treated like crap (“We’re a utility, not a tech company!”)
With constant turnover, the person receiving the “Just make it work” call is about 6 layers removed from the person who originally set up the infrastructure, and probably completely unaware of all the previous security holes put in “because it’s air-gapped, it doesn’t matter.”
Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.
“Well, we just bought this new monitoring platform from my cousin’s company and they need access to it. No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don’t know what port or protocol, it also runs out of my cousin’s house so it’s on a dynamic IP so you can’t create some type of access rule for it. Just put an any/any rule in and it’s good. What do you mean we already have a monitoring platform? Well, this one is better!”
They won’t know that much. They don’t understand that their job isn’t running a water department, it’s running the computers that maintain the water department and that includes security. For someone familiar with computer security it’s easy to recognize negligence, but until the law sees it as criminal negligence nothing’s going to change.
Small town municipal water departments aren’t going to be able to afford the guy holding the shield against international state actors.
Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.
What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.
The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.
I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.
This is all shit they should be doing now. They have to have a firewall to get to the net, so they should be maintaining and patching it. Managing users is standard AD shit, and password hygiene is t as easy, but setting reqs in AD for complexity goes a long way (as does not forcing passwords to rotate on an arbitrary basis).
And yes, I know it doesn’t happen in a lot of places, which is why I said ‘should’. Having been on the cleanup side of multiple security incidents, doing the basics is harder than you’d expect for a lot of places …
Fine. New rule, anything that’s your fault and costs a company money you get to pay for out of your golden parachute, your pension, your savings, anything. You caused a problem to save money this quarter forsaking the next quarter when you aren’t there? Screw off.
Even when a company is doing things pretty well with a VPN, all it takes is one mistake and the cat is out of the bag. At one of my previous employers, we had a fairly good setup with VPNs using single sign on via our IdM provider. IdM enforced SSO and the VPN had a number of policies setup which kept good control over what users could login. And then someone had the brilliant idea to take an old VPN appliance, reset it to factory defaults and configure it just enough to work on our network and then hung it out on the internet for some sort of test (our configuration management left a lot to be desired). Of course, that led to it being left there for a couple months and an attacker finally found it. They also had a valid username/password combination which got them in the door.
Security is hard and often expensive. But still, compared to “we put out OT devices on the internet” a basic VPN is a huge step up.