Today the news is all a’flutter about the massive malware/extortion attack on hospitals all over the place. The sky is falling!

What’s going on is that a lot of companies have really bad security, and have comfortably had it for a long time. Medical informatics is particularly hard because of a “these hands have been touched by god!” attitude from doctors (doctors are probably the second worst ‘users’ you can have, after politicians, in terms of blockheaded self-importance and risk profile) coupled with industry regulation that makes it harder to update software on devices. There’s a really bad negative synergy with device certification: to get lawyers to sign off that a device is suitable for hospital use it has to be in a specific, documented, configuration – which means that if your device is certified, you don’t patch it or update it. So there are plenty of patient monitors out there running old versions of Windows that a hacker can blow straight through. In principle, that would be OK except that those devices are not on isolated device networks, so if Dr Jones clicks on a PDF from patientrecords@quest_diagnostics.com that says “X-ray results for your patient” and gets their machine owned, now the hacker is on the backend network where everything is reachable.

There are a lot of obvious things that can and should have been being done, but the certification process, plus “oh, we have a firewall so it’s OK!” concept have act to block any demand for better security. There is still a great deal of “hard shell around a soft, chewy center” as Bill Cheswick described firewalls back in the mid 1980s. Some organizations realized that malware was going to be a problem, and others didn’t. The two that have had their heads most thoroughly in the sand are medical, and government. Often, you’ve got a situation where a hospital network, with thousands of authorized users, is going to be completely open shortly after one user clicks on the wrong attachment. Worse, some of the devices and internal servers are not treated as core infrastructure and are not under system administration.

My first encounter with computer security was when I was a young systems-pup at Johns Hopkins Hospital in 1987, managing a cluster of Sun-2/110s and a couple of Pyramid 9080s, and one of my Pyramids started getting login and FTP attempts from another Sun system that I recognized was one of the controllers for the research MRI machines. It was highly unusual for an MRI to be trying to log into things, so I picked up a phone…  It turned out later that someone had put a connection to the University, so as to be able to exchange email with the medical school faculty, and Johns Hopkins University was an early ARPAnet node because of the H-bomb designers in the Applied Physics Lab. (I was on the “internet” in 1981 as an undergrad) Someone had come in through the Universary, broken into a default account on the MRI, and was having a look around. We put some filtering rules in the router and changed a lot of passwords but generally nobody got very excited about security until the mid 1990s.

Excitement about security comes in waves. I’ve always thought it’s basic humdrum but that’s because I’ve always thought it was a matter for consistent effort not a panic/respond/panic/respond cycle. Right now, there’s a lot of panicking and responding going on. It’s an International Cyberattack.[nyt]  Actually, no, it’s just a large-scale extortion campaign in which a bunch of hackers have decided to act on a lot of targets at once. This is nothing new: the low-hanging fruit are getting hit. As I said [stderr] last summer: if you can get hit by cryptolocker, you’re going to. So be prepared or be prepared to suffer. If you’re an IT practitioner and your organization doesn’t do its backups, you’re a candidate for losing your job and having it migrate to the cloud. Not a candidate: it’s inevitably your future.

Edward Snowden, unfortunately, weighs in and he’s got an axe to grind:

This is really not the NSA’s fault. It’s a terrible idea that they maintain and build attack tools, and it’s really incredibly stupid that they can’t keep them secret. The real problem is not the specific vulnerability, it’s that hospitals have such lamentable endpoint security that people are falling for basic phishing exploits and their servers aren’t immediately recoverable.

Cloud systems have similar problems. I was talking to an organization the other day that had a lot of stuff out at AWS and was extorted for a lot of money by an attacker who discovered that they had all their system images and all the backups of their systems images in the cloud. So, the extortionist deleted the backups then asked money not to delete the remaining images. As I said last summer: unless you have three copies of your data and one is offline, and none of them are in the same place, you don’t really own your data you’re just borrowing it.

I don’t like that Snowden tried to make this about NSA. It’s more about mediocre system administration practices. And, if you’re a patient or you have medical records or are about to lie down on a table for some tele-operated surgery, you might want to think about that. Snowden asking about NSA building attacks against Windows XP is missing the point that: hospitals should not be running Windows XP. And doctors shouldn’t be reading their email with local adminstrator privileges, etc. There’s a whole great shovel-load of basic security precautions that are apparently not being taken at some very important hospitals around the world.

What’s the most boring topic in IT security? Configuration Management (CM). Good CM practices are what separate Amazon and Google and ADP and Facebook from the rest of us. When you hear about Google’s model for repairing a failing system (“unplug it and slap a new one in”) or you use Amazon’s elastic server cloud – that’s all driven by automated system administration. They have a systems administration fan-out where it’s not one administrator to a cluster of five machines (like I was back in my distant hospital days) it’s one administrator to five thousand machines. And the notion of “machine” has become a squishy concept – it’s more like “capabilities” in the cloud. Stuff to do things to gets routed to places where stuff gets done and it’s all dynamic.

Computer security, as a problem, exists largely because system administration is harder than it ought to be, and organizations have been cheap about system administration and configuration management – they’d done IT at an amateur level, to save costs – and they’ll do it as long as the perceived costs of being bad at IT don’t overwhelm their imagined cost-savings by moving from mainframes to departmental computing and local data centers. The cloud is mainframe 2.0: a massive well-managed computing infrastructure where the user can’t screw up the configuration. That we have this problem at all is because organizations have been able to cheap out about the costs, and continue to pass the costs of their failures on the customer.

This melt-down that the hospitals are experiencing is an outcome of bad cost/benefit analysis.

It would have been cheaper to take the downtime to update systems. It would have been cheaper to put critical systems on separate networks. It would have been cheaper to actually design your networks for performance and reliability instead of letting them evolve based on what’s easiest to get working. Network administration, like system administration, is not just plugging a wire into a switch and patting yourself on the back when the green LED comes on. Did you notice how this outbreak affected the cloud players? Yeah, me neither.

Either you will professionalize and automate your system administration and configuration management, or you will continue to suffer unpredictable periodic catastrophes, and your business will fail or your IT organization will get plowed under and moved to the cloud.

I think NSA and CIA and other intelligence agencies’ “HACK THE WHOLE WORLD” attitude is short-sighted and stupid because it does result in security being weakened somewhat. But worrying about weakening it somewhat is silly when you’re talking about organizations that don’t have good disaster recovery, backups, endpoint control, and configuration management. It’s like blaming the particular bridge pylon you slam into when you’re drunk driving.

NSA is one of the agencies that is partially responsible for helping improve US government security. Trying to hack the whole world certainly doesn’t do that. And, as I’ve argued elsewhere, the best defense is a strong defense. Why are government agencies’ (including the CIA and NSA!) security so lame? Why are hospital’s security at such a level of lame? Defending is harder than attacking, and configuration management is a critical defensive technique that everyone is unwilling to do because “it’s hard” or “the users will complain.”  Just remember that when you lie down for tele-operated surgery.

A tip ‘o the hat to Shiv, who triggered this rant.