Now that I’ve established a few of the basics of data reliability, and I can see that many of you understand the problem fairly well, I have a couple additional points I’d like to make on a related topic.

• What was the backup process for Hillary Clinton’s server? If it was competently managed, there are two possible scenarios:
  – Option 1: It was not backed up at all, because the administrator(s) asked “Do you want backups of this thing or would you prefer us to run it in a RAID-10 configuration with a thermite grenade wired to the top of the drive stack so you can pull the pin if the FBI shows up?”
  – Option 2: It was backed up, because that’s what competent system administrators do. So when all of the “personal files” regarding Hillary Clinton’s yoga appointments were deleted, they are still sitting on backup out in the cloud or in some hard drives or tapes in a pelican case in a media safe. Unless all of those were accidentally dropped into a black hole.
• If there were tens of thousands of messages on the server, wasn’t it clearly being used for archival purposes? I used to have all of my emails dating back to the early 1980s. Now, I “only” have them dating back to 2002. It’s 20gb worth. I probably get less email than Clinton (actually, that’s not a sure thing; my email address is all over publications in tech journals and websites and I’ve been getting about 3,000-5,000 spams/day since the early 00’s)  If Clinton was keeping the emails as an archive (as I do) so that she can go back and zing some friend, or remember how much she promised she’d do for so-and-so in return for such-and-such, then there’s pretty much a 0.0001% chance that there are no backups of the messages. By the way: 20gb. That fits comfortably on a USB stick around someone’s neck. Clinton must inspire great fear or loyalty among her consigliere, or there are copies of her archive sitting in various sock drawers on the internet.
• What is the oldest email found in the server archive? That tells us a certain amount about how long it had existed for archival purposes. That would be a benchmark of how long Hillary Clinton has managed to remain conveniently ignorant of how email works.
• What about the mail server’s transaction logs? On a UNIX system, mail server transaction logs record the sending/receipt of messages. This is an example off the web of a typical SMTP transaction log:
  

  Jul 15 17:11:21 thor.foo.com sendmail[22398]: e6FFBLP22398: from=<jan(a)foo.com>, size=589, class=0, nrcpts=1, msgid=<200007151510.e6FFAC316448(a)odin.foo.com>, proto=ESMTP, daemon=MTA, relay=jan(a)odin.foo.com [192.168.1.1]
  Jul 15 17:11:21 thor.foo.com sendmail[22400]: e6FFBLP22398: to=<gerrit(a)bar.com>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=30589, relay=frigga.bar.com. [192.168.1.3], dsn=2.0.0, stat=Sent (e6FFAFv24566 Message accepted for delivery
  

  The State Department’s servers would be able to clearly identify how many messages were forwarded to @Clintonemail.com assuming the State Department is not so utterly insane as to delete their SMTP logs. But Hillary’s server was running Windows so it might not have comparable logs. Exchange Server keeps its own logs, of course; they would have to be edited as well, or you’d have an exact count of how many “personal” messages Clinton’s people deleted, simply by subtracting existing message count from total “stat=Received” lines. Then, you match the message-IDs from the “stat=Received” lines – that’s the msgid=<200007151510.e6FFAC316448(a)odin.foo.com> bit above. And then you know who sent the messages that got deleted.

  I do know of system administrators who regularly delete their logs. Those fall into three categories: “incompetent amateurs” “people who are trying to hide stuff” and a very very small percentage of “IT people who understand what they are doing and have an application that they understand to have unimportant logs because the information is not long-term interesting and never will be.”  Email is never, ever in the latter category, unless you’re running a hookup site, an anonymizer site, or a forum like Silk Road out on TOR.

  Come to think of it, it sounds like Clinton was running a sort of political hookup site.

• If you’re backing up systems, you inevitably backup the logs. In my career there have been times that I’ve done analysis on logs that were sometimes a year old. Any system administrator with any experience at all is going to understand that “logs can get subpoena’d” and they retain them, back them up, or destroy them according to that understanding. No system administrator worth their paycheck makes an important policy decision like not retaining logs unless they have been made to understand that they should make that decision and “will someone rid me of these troublesome forensic trails?”
• If you have an exchange server on the internet with no firewall or other security, you’re owned by everyone – not just the Russians. You’re owned by the twelve year-old up the street and her nine year-old brother. If you’re owned by everyone, that shows up in system logs, server logs. A system that incompetently set up does not function for years, it has a life-span on the order of months (at best) and when some random hacker who’s scanning networks discovers they have Hillary Clinton’s email, you hear about it at Black Hat or in the New York Times. So I assume there’s some security and patching going on – competent system administration, in other words – which also means there’s backup and log backup going on because that’s also competent system administration. I assume there’s a firewall of some sort and if it’s competently run there will be firewall logs too.

The FBI and any experienced system administrator already know this stuff. OK, some people at the FBI know this stuff, and any experienced system administrator does.

The great big kabuki dance being danced is that some people are pretending not to know this stuff, other people are lying about this stuff, and still more people are lying about accepting stories that are – on the face of it – laughable.

Here’s another one: if Hillary Clinton’s people deleted (because you know that SHE didn’t sit there for 40 hours deleting emails) “personal” emails then they had to … trawl Clinton’s personal emails. Sure, I understand that you gotta trust people. But if I trust my chief of staff to delete my emails between me and my girlfriend, then I can trust the FBI to go “oh, that looks like email between him and his girlfriend” and not look deeper unless they have reason to believe she’s paying me millions of dollars for access on behalf of lobbyists.

I’ll remind you all that this is an easy problem not to have: you have your “personal email” and your “work email” and you do your “work” in your “work email” and your personal stuff in your “personal email” and you don’t mix the two. That might equate to a great big in-box at State Department that nobody ever read – in which case: perfect – the system logs would show that! Problem solved.

This is a problem Hillary Clinton brought on herself when she started bypassing records-keeping requirements that were put in place for exactly the reason this is now an issue. The record-keeping requirements are an attempt to keep politicians from doing sneaky stuff while they’re on the job, and everyone knows they will and everyone knows they’ll try to hide it, so the kabuki dance must be danced. Hillary Clinton simply didn’t dance the kabuki dance and – now that she’s gotten caught out – she’s claiming that of course she’d been dancing the kabuki dance in her shower, by herself, for years, so it’s OK.

My first article about Hillary Clinton’s Email Server on FtB

I don’t particularly care about this stuff, what annoys me is the way everyone’s lying and a lot of people are playing dumb and acting as if the lies are plausible. They’re not even remotely plausible.

I’ll vote for her, because the alternative is so much worse.

What she should have said – immediately – is “that records retention stuff is for little people, and I am not little people. Fuck off.”  That’s what all this means, she just won’t say it. Given the insane crap Trump is saying, she could have done that and been over and past this by now and none of us would have to hear of that Gowdy guy ever again.

By the way: This is exactly the issue that came up during Iran/Contra, in which the Reaganites deleted a bunch of emails from the White House’s PROFS mail system and didn’t understand that the email system (which was run by professional system administrators) was backed up to tape, regularly. Oops. Now we know exactly which messages to look at: the ones that are on the tape but not in your email archive, bozo.

By the way: During the whole Lewinski thing I was repeatedly /facepalming as everyone pretended that nobody kept the President Of The United States’ appointment book, and nobody could tell how often and when he was seeing Monica. Are you kidding me? A good data analyst could calculate pretty accurately how long it took Clinton to cum based on his appointment schedule, given enough sample data-points.