This may be a bit scatter-shot; there’s a lot to cover, and I’m going to try to fold in some answers to comments on my previous posting on the topic. [stderr] I also want to predict the future, so I can say “I told you so!” when it happens.

To be frank, cybersecurity pisses me off so much whenever I think about it, that it’s almost painful to write about. But the questions are interesting and worthy of respect.

Our first item, in line with my paper on The Anatomy of Security Disasters, is that disasters are almost always worse than they look, at first. And, when the disaster is discovered, people like me come out of the woodwork and say “I told you so!” Also, people who were involved in the gestation of the disaster dig up and hand out old memos that say, “this is going to be a disaster, don’t do that.” But, of course, that was exactly what happened and now we’re just arguing about how late “too late” is.

One of the other bits of information that came leaking out is, apparently, someone who was involved with reviewing SolarWinds’ security says that it was pretty bad all along. [bloomberg]

A former security adviser at the IT monitoring and network management company SolarWinds Corp. said he warned management of cybersecurity risks and laid out a plan to improve it that was ultimately ignored.

In a 23-page PowerPoint presentation reviewed by Bloomberg News, Ian Thornton-Trump recommended to company executives in 2017 that SolarWinds appoint a senior director of cybersecurity, and said he told them that “the survival of the company depends on an internal commitment to security.”

I have archives of every communication I ever had with a client (most security consultants do, for exactly this reason) including some presentations that say basically the same thing. I remember one client (I was on a technical advisory board) decided my participation was no longer valuable, after I pointed out that having the developers roll their own O/S releases using Kubernetes was “just one more thing for them to be bad at.” Maybe I should have been more diplomatic, but there’s an awareness that develops, which is “these guys aren’t going to listen to me no matter what I say.” So all you can is say “I told you so” (a process I used to sarcastically refer to as “preallocating blame”)

In fact it sounds like SolarWinds was a fairly typical software development shit-show. Developers sometimes feel that being smart is all that’s necessary to build secure, well-architected systems and networks. Too bad they’re wrong. I have heard development managers non-ironically say, “our guys are really on the ball and I know they monitor the code repository carefully” so that’s good enough – there’s no need to worry about someone putting code in some library that one of the developers just lifted from some open source software archive. Hint to would-be hackers: write a pretty graphing package and put a few extra nudge-nudge features in it and you, too, can pwn a ton of development shops.

Meanwhile, there was not just one bunch of hackers dropping malware into SolarWinds: there were multiple. I used to joke, sometimes, that software is buggy because the CIA backdoors step on the NSA backdoors, and the Mossad backdoors are incompatible with the FSB’s backdoors and some pieces of code are 80% backdoor with a user interface. Ha, ha, ha! Joke’s on me: [reuters]

(Reuters) – A second hacking group, different from the suspected Russian team now associated with the major SolarWinds data breach, also targeted the company’s products earlier this year, according to a security research blog by Microsoft.

“The investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor,” the blog said.

Security experts told Reuters this second effort is known as “SUPERNOVA.” It is a piece of malware that imitates SolarWinds’ Orion product but it is not “digitally signed” like the other attack, suggesting this second group of hackers did not share access to the network management company’s internal systems.

Ah, I see that the situation will require some improvement before we can properly describe it as a “shit show.” It’s not unexpected, though – it’s a great big target, and it looks like it was a soft target; did they expect hackers to ignore it out of courtesy or something? By the way, one piece of advice I used to give my clients was to ask companies that provided critical components about their software security development lifecycle and if the answer is “we patch stuff really fast” then try to get your money back. I don’t expect anyone ever actually asked, but it was good to have the suggestion on a memo under my name, so “I told you so” becomes an option for later.

The worst part is that someone probably convinced themself that SolarWinds was just a network management thing, not a security thing, so security doesn’t matter. And then the government agencies just used it because other people use it. If you’re caught being stupid, a stupid person’s standard defense is to say “I was just doing what Bob did” and meanwhile Bob is saying “I just did what Fred does.” Etc. It’s the de facto standard. Well, so was smoking cigarettes, at a certain time, before people figured out that inhaling combustion byproducts is a stupid idea. I’m amazed.

One of the biggest problems with security is that people just assume that someone else has got it. It’s what Douglas Adams referred to, in The Hitchhiker’s Guide to The Galaxy as an “SEP Field” – an invisibility field that makes something invisible by causing it to be Someone Else’s Problem. For example, Apple has done a pretty good job of making their app store sound like they do mumble mumble something scan something mumble security before an app is available for millions and millions of users. What do they do? They’re cagey about it, but the simple fact is that there is a huge limit to what can be done and, realistically, Apple doesn’t even do that. For example, in Marcus-Land, apps would go into a petri dish that jacked around the app’s time-sense and collected all the traffic the app generated (apps should not generate traffic to anywhere except: Apple, and whoever wrote the app, and there should be specific rules as to data visibility, etc.) And app behavior should be examined; apps shouldn’t access your contact list unless they are doing certain specific things that require a waiver from Apple. That’s all a fail-game, anyhow: you can never be truly safe when you’re running someone else’s code on your machine. And you can’t safely run your data on someone else’s machine, either. Basically, as Pvt Hudson says, “Game over, man.”

It doesn’t matter how hard app stores try – there’s so much money at stake that the people who want your stuff will try harder. And what’s crazy about how security works, is that they only have to try slightly harder. Once they’re over the hurdle, then that’s it. There’s a co-evolution of sorts, but the defense against predation in this world is, pretty much: just sit there. [wired]

Despite some recentpronounced lapses, the iPhone remains one of the most secure consumer devices you can buy, thanks in large part to the locked-down ecosystem of the iOS App Store. But things do slip through the cracks – including 18 apps that used evasive maneuvers to sneak past Apple’s defenses.

The malicious apps – 17 of which were discovered by mobile security company Wandera, all from the same developer, while Apple spotted another using the same technique – have already been taken down. While they were live, they didn’t steal data or gain control of a victim’s device, behavior that other recent iOS fumbles could have enabled. Instead, the apps, which ranged from a calculator to a yoga pose repository, ran invisible ads in the background of the device, generating phony website clicks to inflate ad revenues.

That sort of adware makes regular appearances on Android, in part because that platform’s third-party app stores are riddled with bad actors.

There are whole ecosystems of crapware. Some specialize in harvesting your location, which is then sold to a company that sells location data, and the FBI buys it. Some specialize in ad-click fraud, as above. Some specialize in harvesting your contact list, etc. Each one is just crappy enough that it mostly drains your battery and makes your system’s behavior just a little bit shittier. And there are ecological niches for each kind of crapware – some runs in your browser, some runs in an app, some tracks your clicks, some mines bitcoin. It’d be fascinating except that none of this shit is what you bought your phone or computer for. The same can be said of SolarWinds’ hack: the people who bought SolarWinds did not expect that they’d get a bunch of backdoors – they didn’t ask for that, they didn’t pay for that, make it go away! But, it won’t because it’s always easier to steal someone’s stuff than to earn it and it’s always easier to ask forgiveness than permission. Not that any of these people ask forgiveness or permission, they’re gonna do what they’re gonna do and you’re just prey.

Companies that sell software should try to do some basic common sense stuff to keep people from injecting malware into their code. Beyond that, there’s no percentage in doing that; ease of use and pretty interface drives purchasing decisions more than quality. At NFR we were pretty proud of the fact that our product pretty much never crashed (we had one customer figure out how to mis-configure it so it was a mess, and had to fix the next release so it would block their attempts to shoot themselves in the foot) – but it turned out that customer purchasing decisions were more strongly influenced by the size of the vendor. People preferred to buy from Cisco and IBM because I don’t know; apparently people think that stuff is better, somehow, because it comes from a huge company? I had one customer say “we want a big vendor who stands behind their product” so I asked “have you ever tried to get a refund from Cisco?”

brucegee1962 asks: [stderr]

So Marcus, if things are this bad, what is the worst-case scenario that we’re looking at?

With 1 being Bladerunner “gritty, but stylish” and 10 being The Postman “collapse of civilization,” how bleak an apocalyptic hellscape are we looking toward?

I think we are, currently, in the worst scenario, and it won’t get any better. It may get worse for some people, but it’ll be in isolated incidents, mostly. Back in 2000, I was talking with Nat Howard about this, and he said something profound and depressing: “security will always be as bad as it can possibly be without everything breaking, and no better.” That’s pretty much a perfect assessment of the future. Things will suck but the suck will be spotty, and it’ll get some duct tape and baling wire slammed onto it, and then everything will continue to limp along. Nothing will ever be good, nothing will ever be catastrophic, but it’ll all be more annoying, more expensive, less reliable, and generally a lot crappier than it could be. Cybersecurity is going to be like Dorian Gray’s mirror for our human failings – the internet is shitty because it reflects the shittiness of authoritarian governments, the police state, rapacious capitalists, sleazy marketers, cryptocurrency scammers, payment fraudsters, credit card fraudsters, stupid FBI cops who like to troll through people’s shit, and sociopathic hackers and spooks who are compelled to sniff the underwear in other people’s drawers. It’s going to reflect human nature.

Sometimes, there will be disasters. You know, the US/Israel used STUXNET to attack not just the Iranian uranium refinement centrifuges, they went after the breeder reactor at Natanz. Some of the computers they screwed up were the ones that operated the cooling system. Natanz is upwind of a city of 400,000 – it could have been 10x worse than Chernobyl – but it wasn’t. So … carry on. If someone did that to us, we’d drop some bombs on someone until someone was satisfied that enough suffering had been spread around. That sounds crazy, but I used that example to illustrate how neatly the shittiness of software dovetails with the shittiness of governments and government policies – we can’t fix either of those so we may as well hunker down and hope it’s not us. Besides, the general shittiness of everything resulted in the chemical leak at Bohpal, and computers weren’t even necessary; it’s not like if we have another 4,000 people die in an industrial accident, whether it was a result of bad software, or a hacker, or just incompetence, we will collectively continue to stumble forward.

Basically, I’m an idealist. I feel like the internet could have been a great thing but now it’s just a conduit for ads and spooks and cops and erection pills. It’s still a great place to learn things, you just have to pay a price in terms of annoyance. I remember how, once, I felt like we were building the new Library of Alexandria but then the 4channers came along and installed a “kid porn” section in the philosophy wing and screamed “free speech!” So much for idealism. Nat was right: it sucks right now and it will never get better. It will only intermittently get worse then pick itself up and stumble along.

It might get massively worse, though. I don’t think it’s likely but I used to worry about a scenario in which the internet balkanizes. That would be an international reaction to the US continuing to treat the internet as though it is a colonial power. What if China just said “fuck it” and made their own China-net? We couldn’t do much about it (although the NSA would probably cripple it, periodically, and the Chinese PLA would reply in kind) it would be an economically incentivized strategy, not a political one: why let Apple and Microsoft have the money when the money can stay here? [This gets me close to the multi-hundred-million-dollar idea I mentioned before] Balkanizing would mean that there is considerable new opportunity for providing local application stacks. Countries would need their own software loadouts that were not under US control, and there is a lot of money to be made there. Imagine if you went to the Chinese government and offered to make a China-only operating system environment that they could own, and Microsoft would no longer be bleeding billions out of their economy. In return, they pay you an F-35-like one-time fee for the codebase, and in order to guarantee that it’s not full of NSA malware, you’d commit to live in a nice house under their observation with the understanding that they could torture you to death if it turned out that you’d betrayed their trust. I could do that; the street food in China is incredible, there’s lots of art, beautiful people, music, whatever. What would Iran pay for an Iran-only operating system and app store stack that ran on an ARM processor or something that they could build simple systems around? Is it verifiable? Sadly, no, it’s not. But you might still be able to get that F-35-like money, before they figured that out.

Another balkanization option, which really bugs me, is it might break along corporate lines instead of national lines. Imagine if Facebook ties up a deal with SpaceX so that SpaceX offers a low price captive network that carries only Facebook’s ecosystem? Amazon is locked out. Apple can only update IoS but everything else is locked down to a captive ecosystem. That would allow the vendor to completely dominate the ad market for that ecosystem, essentially paying itself with its own money, and they could drop the cost of the service way down because – basically, it’s the software equivalent of a “company town.” As long as the captives aren’t too badly abused (let them get email from outside, but strip the Google ads off first) it’d be attractive for the vast majority of people who don’t give a shit about software ecosystems. The US has some feeble anti-trust laws designed to keep this sort of thing from happening, but anyone who looks at amazon.com ought to see the writing on the wall. I don’t think that balkanization like that will happen, overtly, but it’s already happening when you realize that the Trumpian FCC is allowing carriers to modulate the service speed of user traffic based on the destination. If you can say “since we have an ad partnership with amazon, you get full data rate with them!” then it’s not hard to say “we don’t have an ad partnership with Baidu so in fact all traffic with China will be slow and we’ll blame them for it.” How would Bob the Basic User know?

So what does that future hellscape look like? Blade Runner, except not cool. Unless you’re some kind of profoundly disturbed person who thinks that Verizon is cool and all the shit that Disney is going to shove at its customers is “art.” Basically that hellscape looks like picking your service plan for your cable TV and having to live in it for years. Which, in a nutshell, is “nationalism” and, we can see it sucks when you write it out large like that.

John Morales [stderr] writes:

IOW, it is possible to build a better internet, without those flaws.

Being possible, it follows that whoever first builds and employs it should have a significant advantage, no?

It’s possible, but it may not be cost effective. And, worse, your competitors can re-write your cost model using hacking as a way of raising the development cost of doing something good.

The ultimate “never fight a land war in Asia” scenario is “never get involved in a great big software development project where your enemy can affect your schedule and your budget.”

That’s another thing that scares me about all of this. Imagine if SolarWinds had doubled their development budget for years in order to not suck. It wouldn’t have mattered – a sufficiently motivated attacker could still have done them massive economic damage.

That’s another thing that worries me: vendors could start doing “hits” on eachother, to manipulate sales. Or outsiders could do “hits” on vendors to manipulate the stock market. Back to the question of “how bad can it get?” It can easily get that bad, and it won’t affect anyone except the shareholders and customers and a bunch of people who lose some of their retirement money.

Back around 2000 me and some of my friends were doing some thinkin’ and drinkin’ and calculated that you could farm software flaws as a way of manipulating the market and it’d be pretty damn hard for anyone to pin insider trading on you. When you think about this stuff, it’s so easy to make so much money if you’re just a nihilist and greedy: you can use hacking to manipulate markets. $100,000/day – would that be enough for you? The SEC hardly ever investigates insider trading and as long as you’re paying your full tax load, they generally ignore you.

I don’t know if you’ve seen this one, but here’s a great opportunity: [news] A Dutch security ‘researcher’/pen tester/hacker compromised Donald Trump’s Twitter account the other day. Apparently, he thought, “Trump is stupid. What kind of password would a fucking moron use?” And, sure enough, when he tried a couple of moronic passwords, maga2020! worked. Being a good guy, he did not assemble a small group of people to invest in shorting SolarWinds stock (even at its current low price, which would be great because you can short more of it the lower the price goes) and then tweet out “Executive Order to get SolarWinds off all USG networks drops tomorrow. Make American Software great again!”

See what I mean? The internet is already shitty because it’s Dorian Gray’s mirror, and it will simply reflect the worst of humanity’s bad habits.

“They paved paradise and put up a parking lot” – in case you can’t tell, I am really angry about this stuff. I feel like I wasted my life trying to make it less shitty than it is, and completely wasted my time. I could have built bicycles and made some kids happy, or made swords and made some people unhappy, or anything – but I got sucked into trying to fix some of the worst aspects of human nature using software. That doesn’t work. My bad. I mean, it worked out OK for me, personally, and I did manage to help a few people, individually, but – what a fucking mess, and it’s all unnecessary and pointless. Enjoy the banner ads and the scammers crawling all over everything like lice. It’s a whole ecosystem of scammers from the very big to the very little and the only cheerful bit about that is that they feed on eachother, too.

How to have fixed Trump’s password: Someone should have punched in a 16-char random string and then told Trump’s twitter app “remember my password” or installed some kind of password vault. 5 minutes. The hard part would be keeping from breaking the Big Baby’s fingers when he tried to grab the phone while you were halfway through setting it up. Meanwhile, some of us were hypothesizing what a mess the NSA would make trying to manage that exposure – they’d give Trump a multi-level secure workstation with an eyeball reader and dog knows what else, and the idiot would just txt his crap to a gopher who’d tweet it. There are so many ways around security if you’re an idiot, and an energetic and aggressive idiot can come up with surprisingly bad ways around. My friend Dan Geer told a tale once, back in the day, of how they tried to get a particular Wall St company to use SecurID two-factor authentication for the traders’ workstations. Dan said they all expected a lot of complaints, but, surprisingly, there were none. None. So Dan got curious and wandered in a bit early, just in time to see one of the executive assistants walking from workstation to workstation with a stack of SecurIDs in labelled envelopes, logging all of the traders in so they could just walk in and get to work.

The bitter and sarcastic security calendar posters came from a marketing bit I did for SourceFire before they got bought by Cisco. It was a lark; I had the idea and went to a couple vendors and the first 3 blew me off until I talked to Marty at SF and he said, “go for it!” I had a budget of a mere $16,000 to produce 1,000 calendars including photography and costs, etc. The calendar won some stupid marketing award (go figure: Marcus hates marketing and is an award-winning marketing guy) and SF wound up doing calendars for years and they were very popular. I did the copy, and it’s my hands and my butt and my computer, etc., for the objects. You can see the whole set at [ranum.com]