I am about two-thirds of the way through the fascinating book The Burglary: The Discovery of J. Edgar Hoover’s Secret FBI by Betty Medsger that deals with the 1971 break-in to the FBI offices in Media, PA by a group of eight anti-war and civil rights activists who took away all the files and then revealed all those that showed that the FBI was engaged in all manner of illegal activities, such as spying on and harassing people who were engaged in purely legal actions of dissent and maintaining extensive dossiers on thousands of people. Medsger uses that story as a springboard to also write a comprehensive true history of the FBI as revealed by the documents and subsequent discoveries as opposed to the myths that the agency cultivated. (See here, here, and here for earlier posts on this topic.)

Medsger, then a young reporter for the Washington Post, was one of five people who were sent copies of the files by the burglars but was the only one to publish stories about them, setting in motion an unraveling of the FBI’s reputation, carefully cultivated by then director J. Edgar Hoover, as an upstanding law enforcement agency that was fighting crimes efficiently when in reality it was an incompetent, tightly controlled, secretive, and rogue organization that, rather than focusing on organized crime, seemed to see as the greatest threat people who were in civil rights, anti-war, women’s rights, gay rights, and other movements. But there was such a cult of Hoover that no one in the administration or Congress dared to say anything against him and instead fawned over him and gave him all the money and resources has asked for. The FBI then was like the NSA today.

One of the enduring mysteries was that despite the FBI launching a massive search to identify and capture the burglars, they were never caught, keeping their secret for over four decades, with six of them coming forward and identifying themselves beginning only last year. This is even more incredible when you realize that all the burglars were ordinary people, complete amateurs in crime, never having done anything like this before. One of them learned how to pick locks from a correspondence course and had to make his own tools.

Compare this group with the burglars working for president Nixon who tried to break into the headquarters of the Democratic National Committee at the Watergate building in 1972, actions that eventually led to Nixon’s resignation. These were people who had worked for the FBI and CIA and had at their disposal professional lock breakers and any resources they needed. And yet this group of professionals trying to break into a private office were quickly caught and exposed while a group of rank amateurs breaking into an FBI office got away cleanly.

Reading Medger’s book you begin to understand why the burglars evaded detection. One reason is the meticulous planning that they did. But another is that the FBI was collecting so much information on so many people that they were simply swamped with data. Because they had all the resources and money they needed, they pursued every possible avenue, created dossiers on every possible suspect, and ended up wildly guessing and focusing on the wrong targets. Even though the names of some of the Media burglars turned up in their data sets because they were involved in antiwar and civil rights activities, the FBI investigators simply couldn’t ‘connect the dots’, to use the current cliché, because there were so many damned dots that could not see any pattern.

The fact that again we frequently hear people bemoaning the fact that before 9/11 people failed to ‘connect the dots’ and realize what was going on suggests that the problem has not gone away. It is true that with modern computers one can create algorithms that in principle can quickly search databases for patterns, something that was not available to the FBI in 1971. But the problem is that computers also enable vastly more data to be collected and the ‘collect it all’ mentality that has become the goal of the NSA means that the number of dots that are collected has also increased exponentially.

This is why, as Peter Maass describes, some NSA analysts have warned in internal memos against the ‘collect it all’ mentality.

As members of congress struggle to agree on which surveillance programs to re-authorize before the Patriot Act expires, they might consider the unusual advice of an intelligence analyst at the National Security Agency who warned about the danger of collecting too much data.
…

“We in the agency are at risk of a similar, collective paralysis in the face of a dizzying array of choices every single day,” the analyst wrote in 2011. “’Analysis paralysis’ isn’t only a cute rhyme. It’s the term for what happens when you spend so much time analyzing a situation that you ultimately stymie any outcome …. It’s what happens in SIGINT [signals intelligence] when we have access to endless possibilities, but we struggle to prioritize, narrow, and exploit the best ones.”

The document is one of about a dozen in which NSA intelligence experts express concerns usually heard from the agency’s critics: that the U.S. government’s “collect it all” strategy can undermine the effort to fight terrorism. The documents, provided to The Intercept by NSA whistleblower Edward Snowden, appear to contradict years of statements from senior officials who have claimed that pervasive surveillance of global communications helps the government identify terrorists before they strike or quickly find them after an attack.
…

The documents noted the difficulty of sifting through the ever-growing haystack of data. For instance, a 2011 document titled “ELINT Analysts – Overcome by Overload? Help is Here with IM&S” outlined a half dozen computer tools that “are designed to invert the paradigm where an analyst spends more time searching for data than analyzing it.” Another document, written by an intelligence analyst in 2010, bluntly stated that “we are drowning in information. And yet we know nothing. For sure.”

It is quite extraordinary how similar this situation is to the one that confronted the FBI in 1971. It seems like people never learn. Just like Hoover warned then that any restrictions at all on the FBI’s ability to collect all the information it wanted, irrespective of whether the methods used were illegal or unconstitutional, would lead to terrorists (at that time synonymous with Communists) destroying the nation, we hear those same warnings now about the danger of any restrictions at all on the NSA’s massive data-collection operations.

The Daily Show comments on the strange fact that the NSA does not seem to be able to actually do anything useful with all the data it says it must collect to keep us safe.

(These clips aired on June 1, 2015. To get suggestions on how to view clips of The Daily Show and The Nightly Show outside the US, please see this earlier post. If the videos autoplay, please see here for a diagnosis and possible solutions.)