Police IT staff checked wrong box, deleted 25% of body cam footage
In 2014, Oakland Police Dept. made fateful error, hadn’t set up backups, either.
Maybe they hired their IT staff from Hillary Clinton, from www.RoseMaryWoods.com IT staffing.
One-quarter of all body-worn camera footage from the Oakland, California, police was accidentally deleted in October 2014, according to the head of the relevant unit.
As per the San Francisco Chronicle, Sgt. Dave Burke testified on Tuesday at a murder trial that this was, in fact, a mistake.
This incident marks yet another setback in the efforts to roll out body-worn cameras to police agencies nationwide.
In late August 2016, the Seattle Police Department reported a similar IT glitch involving body camera footage.
“Nothing should have ever been lost from the system,” Burke said in court, later adding, “The settings were set to never delete.”
(Source: Ars Technica)
Back in 2009 I did some consulting for Taser regarding security architecture for data transmission between cop-cameras and wearable point-of-presence (Taser calls it the Axon) and their cloud service. Taser’s concern was that their customers – the police departments – needed to be sure enough that the data in the cloud storage was going to be available, tamper-proof, and backed-up. The premise of the cloud service, of course, was that the cop-cameras’ data didn’t need to be stored locally, which meant no IT footprint at all; everything would be neatly managed from the cloud, and of course! This is my thing! My bread and butter! All accesses were going to be recorded and there would be identification and access control (I&A) and high security at the cloud data center, with backups going to Iron Mountain.
I have discussed this several times in this blog: if you cannot do basic data management, you need to step aside and let Google and Amazon and other grown-ups do it. In formal security-ese that means “availability”, “access control”, and “integrity” – it is utter bullshit that someone forgot to check a box and suddenly a whole lot of data was erased – and, oh, conveniently, there were no backups either. “A backup system had been purchased but not installed” and the erasure glitch was a result of a software upgrade.
Got that?
We are supposed to believe that there exist chucklefucks in the world that are so utterly incompetent that they do a software upgrade on a production system without backups? That strains my credulity. Unless Donald Rumsfeld has become a system administrator, it is simply impossible that anyone would be so incompetent. I know junior systems administrators who’ve just got root on their first Linux machine, that would never, ever, make a mistake like that.
Whole DOZENS of terabytes.
Elsewhere in the article, it scopes the size of the video archive as “dozens of terabytes.” Oh, I am so impressed. I’m a hobbyist who plays with digital media, digital photography, done footage, slow motion, yadda-dadda and my desktop computer is spinning 16tb. In addition to all the other stuff I do, I manage that in triplicate. I even post lengthy blog postings on how. And when I do, some rational person asks “Why not just use Carbonite?” which is a great question! You mean the cloud? Like Taser suggested?
What a great idea!
Technical neeping aside, let’s get to the money shot, shall we?
Though officers activated body-worn cameras when they arrived at Fern Street near Fairfax Avenue, no footage from the July 24, 2013, slaying could be found because of the reported data erasure one year later.
Ah, so, the data was erased a year later and the backup system wasn’t installed a year later. But the data that was deleted was undiscovered. So IT Specialist doing a software install a year later with no backups a year later and nobody did any backups on the production system in the year between the incident and the erasure? So the backup system was in the state of “hasn’t been installed yet” for a year?
Burke was asked to explain the deletions in court because Annie Beles, the attorney for defendant Mario Floyd, has said that footage taken from officers who arrived at the scene of Salamon’s killing would contradict witness statements.
Beles said the footage would likely show that no trash bins were knocked over in the street, which is notable because Ford, the prosecutor, has argued that Floyd threw a trash bin at Salamon as he demanded her phone minutes before co-defendant Stephon Lee allegedly fired three shots, killing her.
Pull the other one. It’s got bells on.
Cops tampering with evidence in a judicial murder case? That’d never happen. Right?
I never got back with the Taser folks because I got pulled off in other directions, but there’s a part in my consultant’s report where I mentioned that there may be “externalities” governing various police forces’ behavior, which might not have anything to do with whether the cloud storage was secure, or not. One of the points I emphasized was that having everything up in the cloud meant that it would dramatically reduce the likelihood that someone might leak video – you know, if Officer Porko tells his buddy “Hey I saw $hot_young_actress puking her guts up and she failed a breathalizer and I could totally see her underpants” there’s no way for them to go get it because the cloud service would log and audit all accesses and a supervisor would have to counter-approve any access once the video had been uploaded to the cloud. I felt that the concerns that cloud administrators might be sitting there watching the videos were not significant because there’d be too much to watch and unless they knew which camera Officer Porko was wearing and that the incident occurred at all, it’d just be a needle in a haystack of data, to them.
The cloud storage option should be mandatory for all cop departments. We can’t trust them. We shouldn’t trust them.
Oh, one last thing: whatever kind of software upgrade gives you the option to delete 25% of your data? I’ve built software upgrade processes, and run/installed countless application upgrades. I’ve never encountered a software upgrade that gives the user the option to delete terabytes of data without clicking “OK” and “YES I AM SURE” a couple times. Software upgrades are one of the trickiest bits of product design out there, because you have to extremely carefully forward-move site-specific settings and application data, specifically to prevent that kind of thing from happening. No vendor builds a system that does that, they’d tell the customer “if you want a fresh install without the data, install it on a new system and swap over to it once it’s up and running.” But, in that case, you’d have those old hard drives in a media safe, somewhere, wouldn’t you?
In 2013 a 2tb hard drive was “the thing” and they cost about $200 apiece. So let’s say 24tb is $2400. If you’re putting that in a rackmount with 24 bays, you’re looking at $3000 for the rack, let’s say $10,000 all told in 2013 pricing. Slap BSD and ZFS on it and let’s bump up the storage space to capacity (48tb) for another $2400 – call it $15,000 for the server and an intermediate level systems admin would have that up and running in a couple days. A novice in a week. An old grumpy badger would have it working by lunchtime. A 48tb rack-mount tape library is about $5,000 in 2013 costs, and can do unattended backups since the sizeof(tape) is equal to or greater than the sizeof(storage). Setting up a tape backup library is maybe a couple days’s work. Actually, the way I roll, I’d forgo the tape backup and have 2 of the $15,000 storage servers in 2 locations, one of them colo’d at a service provider, or I’d let Iron Mountain do it. Problem solved. OH, AND TURN ON SYSTEM LOGGING ON THE FUCKING THING.
PS – I just checked and Iron Mountain’s cloud backup isn’t cheap – about $1k/tb/month. So a competent system administrator and backup-to-disk or filesystem replication between a pair of $15,000 servers and you never lose or get corrupted data.
Of course then you have the problem that, once again, cop-cam shows cop lied and killed someone and tried to cover it up. Easier to blame IT, huh?
I wonder if there is some system administrator in the Oakland PD reading this thinking “I told them that experienced system administrators would see right through that story.” Yeah, pal, they threw you under a bus and it didn’t fool everyone.
Actually, I can totally believe that. Public sector IT admins are the worst.
Of course not. Amateurs care. To screw up that badly takes a professional. Have you never heard the saying “backup is for punters”?
Yeah, that’s the bit I totally don’t buy. Even if the upgrade does offer the option to blow away your data, why only 25%? In my experience, these things tend to be all-or-nothing.
“We are supposed to believe that there exist chucklefucks in the world that are so utterly incompetent that they do a software upgrade on a production system without backups?”
Unfortunately, yes…
One job I once worked at (one of the largest ISPs in my country!) only kept a backup of the previous version of their system. Then they upgraded twice in a row in quick succession. Guess what happened… First upgrade was bugged, second upgrade backed up the bugged version and deleted the only working version.
Dunc@#2:
Have you never heard the saying “backup is for punters”?
I’ve never heard anyone say anything that stupid, unless it was in a D&D game, some 1st level mage “I am attacking the orc clan! Because I am tired of this game.”
OK, it appears that the phrase has only ever been used in satirical articles / poems by Verity Stobb: https://books.google.co.uk/books?id=WMyTANSaI0wC&pg=PA141&lpg=PA141&dq=%22backup+is+for+punters%22&source=bl&ots=p8moweFZkn&sig=bbWcawtIaP1hidd-9fQaaE0UR6c&hl=en&sa=X&ved=0ahUKEwi5vPTny5vPAhWKIMAKHeNJAsMQ6AEIITAA#v=onepage&q=%22backup%20is%20for%20punters%22&f=false
robert79@#3:
One job I once worked at (one of the largest ISPs in my country!) only kept a backup of the previous version of their system. Then they upgraded twice in a row in quick succession. Guess what happened… First upgrade was bugged, second upgrade backed up the bugged version and deleted the only working version.
Competence: If you think hiring experts is expensive, you should try hiring a few ignorant people and see how it works for you.
Dunc@#5:
Well, that is memorable! Very cute.
Are you familiar with “Three dead trolls in a baggie”? They do songs about system administration. Start with this one:
Marcus@#7: I’ve certainly heard of them…
Maybe they installed image recognition software that was tasked to delete all footage showing people getting shot…
Seriously, what’s wrong with cameras that start recording when pulled off the charger, can’t be turned off and automatically download to a secure, backed up, logged vault when plugged back in? Oh, that’s right, the foxes get to buy the hen’s security system…
[pedantry]
Rog: that should be “automatically upload”.
(Data going to a device goes “down”, data going from a device goes “up”)
[Ack! Sorry, Rob.
(Believe me, I am duly embarrassed!)]
Like Rob, I concur that cop cams start recording when pulled off the charger, can’t be turned off and automatically download to a secure, backed up (off site), logged vault when plugged back in. Add the rule that 100% recording apply to police INSIDE the station house as well when meeting the public anywhere. Yes, that includes bathroom breaks.
Some years ago I asked a BART ([San Francisco] Bay Area Rapid Transit) cop what he thought of wearing such a recording device. He said something to the effect of, “Hell Yes. Then people will know what kind of crap we have to put up with.”
Cops need fast READ ONLY access to their recordings to assist in preparing written reports. Subject to a few rules, those in contact with the officer can obtain relevant footage.
One concern: Cop visits
snitchCI. Lawyer for snitched upon gets access to the video. Bad things happen to the CI.Rob@#9:
Seriously, what’s wrong with cameras that start recording when pulled off the charger, can’t be turned off and automatically download to a secure, backed up, logged vault when plugged back in?
You basically described the Taser system. Low rate local recording all the time, plus when the gun is out of the holster it kicks into time-and-GPS coded full rez. The charger/base is the gateway that pushes the encrypted video to the evidence collector. It’s not rocket science.
Cops resist it. Because they want to be criminals and get away with it. There is no other possible reason.
Peter B@#12:
Yes, that includes bathroom breaks
Not to hammer on your point, but that is an objection that is sometimes made. And, as you point out, it’s bullshit. I mean, seriously, who is going to want to go watch Officer Porko’s view of the inside of a urinal?
As society gives these people so much trust that we are willing to allow them to carry killing weapons in our names, they must accept the responsibility that power confers. And part of that responsibility is a loss of their privacy.
That (not to hammer the issue too much) is why it’s so wrong when politicians deliberately skirt records-keeping requirements: there is no right to privacy when you are wielding that much power.
Marcus,
I’d resist it too, but because I would not want to be subjected to the panopticon.
Think about it. Every moment of your job is potentially being monitored. Every forgetful moment, every fumble, every mistake on record. Every interaction with others. All the time.
Now, you might consider that a feeble reason, but reason it is — and nothing to do with criminality.
[Actually, a bit like writing a blog… :) ]
John Morales@#15:
I’d resist it too, but because I would not want to be subjected to the panopticon.
Panopticon is an interesting idea, but it doesn’t really apply. Bentham’s design was for a regime of constant potential surveillance. If we had surveillance footage of every moment of every cop’s working day, it would be implausible to actually review it all, all the time. Thus, it wouldn’t really be a panopticon because the potential for constant surveillance is not there.
There would be a huge potential for retroactive surveillance (more like “audit” than “surveillance”) – you know, if someone got a warrant they could see everything a cop did on a certain day. But without a warrant they’d be fiiiiiine. It’s not like anyone would ever violate a warrant or anything.
A bit like writing a blog. And, since I’m dancing around certain topics in philosophy that are tricky, I am pretty sure that there are members of the commentariat that will pounce on me if I make a sufficiently grave error. So, I must simply not do so.
[meta]
Well, I’ll not be the one to bother you; already done my worst.
In reference to errors:
Word games. Surveillance footage of every moment of every cop’s working day necessitates potential for constant surveillance; this is analytical: were the potentiality not extant, neither could the actuality be.
John Morales@#18:
Surveillance footage of every moment of every cop’s working day necessitates potential for constant surveillance; this is analytical: were the potentiality not extant, neither could the actuality be
Yeah, that’s true. I wasn’t trying to play word-games, though. We’d be able to implement better protections and guarantees than they expect us to live with.
With regard to the question of when one is under surveillance, we can either wallow in linguistic nihilism, or I’d say I was using the word “surveillance” in accordance with the dictionary “close observation” – if all cops were being recorded all the time it would be impractical to have them all under close observation all the time, so I wouldn’t say they were under surveillance.
I’ll note that this is an important issue. The NSA appears to have adopted a radically different definition of “surveillance” than I think most people would accept, one which is more in line with the dictionary’s definition. That has some huge ramifications. I’ve been meaning to do a post on that topic for a long time (since before I started the blog) it’s an interesting topic but one that really unsettles me.
So… a YouTube channel for each cop?
(Not exactly technologically or practically intractable; Bentham might have peed his pants at the prospect)
Final post on this thread: I’d rather politicians than cops, if consequentialism is the rationale.
John Morales@#20:
So… a YouTube channel for each cop?
No, I’d envision as Rob described @#9: the cops are under constant “surveillance” (whatever that is!) with a system that records their actions for later audit under warrant controls. It was you that was equating the mere existence of a recording with ubiquitous surveillance/panopticon. I am not sure whether a recording of everything I do, that nobody ever looks at, is “surveillance” or not.
I’d rather politicians than cops, if consequentialism is the rationale
If consequentialism were the rationale, I’d say that corporate executives and any wealthy individuals trying to translate their wealth into political power (e.g.: Adelson, Koch, Bloomberg, Perot..) I’m comfortable with saying “I don’t like cops” “I don’t trust cops” and “I want to be able to disempower them as much as possible because of that.”
Let’s ask the NSA! (Oh, I just notice that you’ve touched on this…)
I think the argument that they forfeit any right to privacy (on the job) because of the degree of power conferred on them is much stronger than the argument that it’s not really surveillance if nobody actually looks at it.
Dunc@#23:
I think the argument that they forfeit any right to privacy (on the job) because of the degree of power conferred on them is much stronger than the argument that it’s not really surveillance if nobody actually looks at it.
It appears that way to me, as well. That’s part of why I am so disappointed by the NSA’s making the “not surveillance if we don’t look!” angle – but, of course, they are collecting everything from everyone, which means they can’t use the first argument. If NSA was only spying on cops, millionaires, CEOs, investment bankers, etc, then that argument would open up for them.