Today we learn that Yahoo!’s user database appears to have been compromised: 500 million accounts plus associated information including (apparently) hashed passwords.
The Mike Tyson Effect
And, of course, it was “state sponsored” (cough) (cough)
I call this the “Mike Tyson Effect” – when you get beaten up, you don’t want to tell your friends you got beaten up by PeeWee Herman, so, by the time you’ve told the story, the person who beat you up was incredibly skilled – probably a professional boxer like Tyson, or maybe a ninja, or Chuck Norris. After all, a company that suffers a major information security breach can’t come out and say “we screwed up, oops.” It’s necessary to mis-represent the skill of the attacker in order to downplay their own failure.
And, we know it’s a pretty typical-sounding failure. Apparently the breach happened 2 years ago.* That is in line with industry norms. According to the Verizon Data Breach and Incident Report (DBIR)** the average time to discover a breach is on the order of 100 days (depending on a variety of factors, I’m simplifying) – which is a big problem because the usual time between a breach and data being exfiltrated is under a week, usually a matter of days. So, Yahoo! got hit in a fairly typical way, and reacted in a fairly typical way; i.e.: they didn’t know they were hit.
The industry’s response to breaches has been lame, but not as lame as the NSA/CIA/FBI’s. By 2000 it ought to have been obvious to anyone with a website that their customer database was a target. Instead, with vigorous poking from legislators (who, themselves, demonstrate terribly poor information security practices) they focused on establishing a whole system of breach notification and indemnification for failure. So, if a provider leaks your personal information due to their incompetence and negligence, their liability ends when they tell you our information was leaked and that you should put a fraud watch on your bank accounts, credit cards, experian, blah, blah, blah. I.e.: “we suck, but here’s all this extra work for you, and now we’re not responsible.” Want to guess who thought up that response regime? Yeah, the businesses lawyers had a little bit to do with it.
Record Count is A Poor Metric
The media, of course, do not help. Because they do not understand security well enough to report sensibly about it. For example, why is number of records leaked relevant?
Well, it’s not, really. But the media assume that if one instance of leaking is bad, forty million instances of leaking are worse. That’s not true, of course: leaking a few dozen administrative passwords is more severe than leaking every twitter account’s password – simply because nobody’d be able to exploit all the twitter accounts and it would only impact a small number of users, whereas exploiting administrative accounts compromises entire domains or meta-domains instantly. It’s frustrating for serious security practitioners, because the noise-level is high and the clue-level is low, so it’s hard to combat people’s tendency to focus on the wrong thing, “Well, Yahoo! lost 500 million users’ data, so we don’t have to worry, we’ve only got a million users, total, we can lose ’em all and nobody’ll notice!”
Security is a Sea Of Mediocrity
There are sad and disappointing things about all of these breaches. Ones like the Office of Personnel Management being a notorious examplar of clueless that makes Yahoo! look minor in comparison. First off, yes, Yahoo! should have (and did, I know their security people) (or used to, if they haven’t all gone elsewhere) known they were a target and had their systems set up to detect attacks against the user database. There are basic techniques that people like myself have been talking about for decades, which they should have done. Much like the NSA, CIA, State Department, Hillary Clinton, etc not having their file server logs turned on, they should have been looking at access logs against the database, using privileged access management and data isolation, as well as rule-based policy violation detection. Office of Personnel Management’s breach was worse because they had been getting warnings for years about vulnerabilities in their practices and apparently didn’t put two and two together and realize that the US Government’s employment database includes a lot of juicy stuff. Basically, it’s a list of everyone’s salary, job code, branch they work in, classification status, etc. It’s the “org chart” for the whole US Government, including the intelligence community. /facepalm
Now, I assume that any country spying on the US already has that information, on an agency-by-agency basis, but it sure was nice of OPM to put all that stuff in a single database so they can confirm it.***
In both cases, OPM, and Yahoo! – actually, all the cases I mentioned so far: NSA, CIA, State Department, Clinton, Democratic Party HQ … – the breach was not discovered as a result of competence within the organization. Some system administrator or security person didn’t say “Whoah!?” and look more closely at the logs. They learned they were breached when they woke up and read about it in the news, or on wikileaks, or some journalist called.
So, if you want to understand how badly security sucks, that’s the number right there: zero. That’s how many of the organizations I’ve mentioned detected the breaches in a timely manner. The figure above, from the DBIR (“Figure 9”) shows the scariest trend-line in security – the number of breaches detected by the victims themselves has been going steadily downward. So, while I place NSA, CIA, State Department, etc, in the Pit Of Suck, they’re going to have more and more company. And they will lie more and more egregiously about how they were hacked by cyberninjas from the 9th dimension, not drilled with a basic phishing attack by a bored sociopath.
I know a few heroes out there, that are locally reversing that trend. One CSO I know has the organization’s security practice aligned toward the objective of being able to catch a penetration-tester in real-time. Think about that: that’s actually a minimum level requirement. If you can’t catch your penetration-tester, who you hired to come try your security, you’re not going to have a ghost of a chance of catching a hacker. What’s crazy is that the organizations I know that do basic, solid security stuff? They have substantially fewer problems than the guys that don’t. Imagine.
As a joke, a few years ago, I totalled up all the credit card/SS# breaches and it surpassed the population of the US. Yahoo! did that single-handedly. So, even assuming a healthy overlap, we can probably assume that the only Americans who haven’t had their information leaked are the ones who were born 5 minutes ago. And I’m not even sure about them.
Last, But Not Least:
If Yahoo! didn’t know about the attack for 2 years, how in god’s rolling green acres of fuck can they say NOW “it was state-sponsored”????
The heading “Security is a Sea of Mediocrity” sounds like something Jean Meslier would write, doesn’t it?
(* One trick I used to teach people was to have a procedure that runs every night that updates the birthday on a ‘tombstone’ user, let’s say Fred Fishlips: if someone offers to sell your database, you can tell if its yours because Fred Fishlips is there, and what day it leaked by looking at Fred’s birthday. There are lots of ways of fingerprinting data like that.)
(** The awesome Jay Jacobs was the brains behind it for several years, now he’s gone on to do other things. I expect the DBIR to slowly become irrelevant. One issue already is that it’s a self-selected sample, representing incident reports of those organizations that were willing to report breaches and incidents to Verizon. We have no way of knowing how biased that sample is. I interview Jay on the DBIR and related topics here)
(*** That’s the real game. It’s one thing to think you know something about another country, but when they’re kind enough to have data that cross-confirms it, then you’ve got a measure of their IT competence as well.)
*waves hand* These are not the droids you’re looking for.
Well, that’s just fucking swell. Great. Fabulous. Please pardon my repetitive crudity, Marcus, but there are a whole bunch of d00d scrota that should be slammed in their own car doors. In my opinion. Self inflicted scrota slamming, of course. I would never advocate a violence-for-stupidity approach.
Did I understand that correctly: It is too late to worry about the account I used now?
Should I use another one, in the gmx-we.de-system, more?
I don’t know how they can claim this was just figured out, ffs, anyone with a Yahoo account knew about the hacking a very long time ago. Rick used to have a Yahoo account, and I had to nag him to dump it, because of the sheer volume of spam that was coming to me, supposedly from his email. That also happened with a number of friends of mine – I was getting spam from an address I knew and was familiar to me. Fell for it the first couple of times, too, until I got a grip and asked a friend if this was her sending me this crap.
People dumped their Yahoo accounts in droves.
Ok, Ok, Ok, I still got a Yahoo account. Two, actually. Thing is, they’re telling me now to change my password? Like I didn’t change it a couple of times within the last few years?
… of course, it was “state sponsored” (cough) (cough)
Here, have a eucalyptus lozenge.
Unless/til you see something you can uniquely identify as your data oozing out of a hole in the ground somewhere, is there (assuming minimal competence by the other guys) any sort of reliable fingerprints left after a deep breach?
… if one instance of leaking is bad, forty million instances of leaking are worse.
Doesn’t a Really Big Number® indicate that the watch-person/dog was sounder asleep for a longer period?
Pierce R. Butler@#5:
There are often reliable fingerprints but they are still hard to interpret, even if skilled in the art. There’s some old pieces I did on attribution here.
The problem gets complicated if the attacker deliberately throws mis-fingerprints. For example, the attributions the FBI throws about China hacking are really flimsy, though hopefully they are also getting data from NSA using methods they are unable to disclose. NSA wouldn’t tell FBI anything FBI could (and would!) leak to the press. It’s a matter of a a very small amount of work to appear to be from somewhere else, and it improves the attacker’s operational security. Most of the hackers I know who play in the deep end of the pool have personas that are fully realized and often include head-fakes as to their real location.
On the inside there are ways to attempt to detect leaks but it’s better to protect against them than to suffer them and detect them. Most of these breaches result from overly open data culture in the target: someone gets phished, an administrative exploitation is done on a penetrated desktop, and the database is downloaded.
The number of records means little – we’re talking about a relatively small amount of data – maybe 100mb or so. Collecting it was probably a single interactive sql command to a file which was then exfiltrated in seconds. You could quadruple or decimate the scope of the breach without impacting the time it took or its detectability.
Giliell@#4:
I encounter lots of people who use one password for everything and change it every few years. The state of security is very very poor.
Caine@#3:
One trick I used to teach was “canary” accounts. You create a user like [email protected] that never does anything. Then, if that account starts getting email you know you’ve had a user database breach and it’s been sold to a spammer. That’s a basic technique a couple of us cooked up back around 1997 – it’s sad when megacorps don’t do the basics.
(I learned that Ebay had some problems well before everyone on the street, when my dummy ebay account started getting Chinese-language spam…)
Raucous Indignation@#2:
The valuation of Yahoo! will have dropped $millions because of this breach and the timing of its disclosure. That’ll hurt more than anything else could.
Oh Marcus, how you underestimate the pain of a well lubricated car door!
Raucous Indignation@#11:
Actually, I have a big sister who once slammed my finger in a door; I have a pretty good imagination and I can extrapolate.
For now most of the business consequences of incompetence are going to be financial. That’s important because the reasons given for being incompetent are usually financial (or convenience or time to market, which are different coin) – making incompetence expensive rather than painful helps reset the economics of computing. Too many people see computing as a free lunch. It’s not.
Hi Marcus,
Do you have a list of best practices you can point to for a one-person (part-time) business providing a specialized web-based service who would like to concentrate on the business side of things, but has to worry about security?
I’ve off-loaded a lot of the security concerns (sql injection, etc.) to experts by using a standard webapp framework (in my case, Django: https://www.djangoproject.com/). I’m interested in what other steps I can take to increase security?
Isn’t it possible that somebody’s done a dictionary attack on the entire domain, looking for addresses that don’t bounce… Not sure how common that is, but I wouldn’t rule it out.
Doesn’t necessarily mean that it was related to his account – they might’ve got his email from your address book. You can put anything you like in the “from” field of an email.
I’m honestly surprised it’s even worth $millions in the first place. Hell, the biggest surprise for me in the news of this breach was that Yahoo! still existed. (I kid – I do still use Flickr for a couple of things. Apparently I even still have a Yahoo! email account, although I’ve never used it for anything…)
Rune Olwen@#3:
That’s part of what’s so irritating about this announcement: it’s way too late to matter to anyone.
Dunc@#14:
Isn’t it possible that somebody’s done a dictionary attack on the entire domain, looking for addresses that don’t bounce… Not sure how common that is, but I wouldn’t rule it out.
Spammers have been doing that since the 90s. Find an open relay and check to see what addresses are good and which aren’t. When you set up a tombstone address, it’s an actual address – you have it forwarded to the security operations center inbox so it comes up as a red-line item if someone starts sending messages to it.
I’m honestly surprised it’s even worth $millions in the first place.
It won’t be, once the internet banner ad market collapses. The only company in that space worth anything is google (and they’re stupid as a pile of rocks, so they may still screw up the goldmine they are sitting on)
Sunday Afternoon@#13:
Do you have a list of best practices you can point to for a one-person (part-time) business providing a specialized web-based service who would like to concentrate on the business side of things, but has to worry about security?
If you’re one person, it’s tough. If you’re hosting your server locally, (at your home internet or whatever) then you should keep a firewall in place so that all that comes in is web traffic, keep abreast of vulnerabilities in whatever server software you’re running (apache or whatever) on port 80/443. Then, if you’re storing user account information, it gets tricky. Your main most likely problem will be a flaw in server side scripts – let’s say you’re using wordpress: make sure you keep that religiously up to date. And do your backups, of course. Using django is good – now you just need to keep up with bugs in django. If there’s a way of setting it up to use a captcha on logins that’s also good: a lot of sites get brute force attacked by robots (just do that on admin pages – wordpress has that option) (wordpress attracts a lotta lot of robots)
The more advanced stuff like having watchdogs on your database and system is going to be more than a single person shop will have time for/want to do. That sort of thing involves having logs tailed off to another machine which sits and looks for patterns that aren’t usual, which means you’ve got to build a notion of “usual” (which is tricky!)
Hi Marcus,
Thanks for the comments and suggestions.
My business is largely seasonal – I’m in my development phase again and am re-prioritizing my “todo” list. There’s a lot to improve!