Trapped in a cunning prison!

Oh, no, what are you doing now?

[Read more…]

You can lie with numbers as effectively as you can with words, so this collection of rules for critically evaluating Big Data claims is timely. I think they missed at least one, though. It was caricatured in a recent xkcd:

xkcd

I’m seeing a lot of these lately. For example, here are the most popular porn searches by state. I’m sorry to say that this is mostly garbage data, useless to everyone.

These data are produced by basically subtracting (or dividing) away the mean and amplifying the differences. I suspect that there is a great sea of banal commonality to porn searches, and they’re more or less the same everywhere…but all the similarities are erased to accentuate slight variations that might be minuscule. If everyone in America were searching for “insect porn”, which would be an interesting and weird piece of data, and one guy in New Hampshire slipped and typed in “incest porn” instead, New Hampshire would be lit up in these maps as the freaky state that wasn’t watching mantis copulation videos.

You might be saying to yourself that this is a trivial example — OK, let’s be cautious in interpreting data techniques that rely on amplifying minor differences, since they can mislead you about the overall state of the system. However, there is a technique that gets published all over the place in the scientific literature that is doing exactly the same thing: it’s called fMRI. This is not to imply that MRI data is bogus, because it’s very good at detecting consistent differences in pattern, but it’s also very good at highlighting chance variation, and it takes a lot of processing to smooth out the roughness in the raw data, and the whole point of the technique is to erase background activity.

I used to do ratiometric imaging, which has similar potential pitfalls. We used a fluorescent dye that would exhibit subtle wavelength shifts in the presence of calcium, so you would visualize activity in the brain by taking a photo at one wavelength, and then a second photo at a slightly different peak wavelength a fraction of a second later, and then taking the ratio of the two. If there was no shift at all, the images would be identical, so every pixel would have a ratio of 1 — which we’d scale to a displayed color of black. If a pixel was fluorescing a little more at the second wavelength, you’d have a ratio slightly greater than 1, and we’d pseudocolor that to something a little brighter.

Again, this is a perfectly legitimate processing technique and the fluctuations we observed were valid and consistent, and you could even calibrate the ratios against known concentrations of calcium and get good estimates of the actual amount of free calcium at each time point in a recording. However, here’s the thing: if you looked at the raw data, or if you looked through the eyepieces at the tissue, you’d see that everything was gently glowing, and that there was actually artifactual fluorescence all over the place, and there was also continuous, low-level calcium flux everywhere, all the time. That information was discarded. Also, when you see the pseudocolored images rendered by these sort of techniques, there’s an awful lot of point variation that is smoothed away, because we tend not to like our pretty pictures spattered with lots of salt-and-pepper noise. We blur it all out.

This is a special problem for methods like MRI, which tend to be at a painfully low resolution (each pixel represents thousands upon thousands of cells), and is also grossly indirect — it’s measuring oxygen and blood flow, not actual electrical activity.

To that list of Big Data cautions, I’d also add that you have to be conscious of what is actually being measured, limitations of the technique, and how you can be misled by assumptions about the resolution. Data can be massaged into all kinds of ridiculous conclusions if you’re not aware of every step of its manipulation.

None of the above: it’s a unique canid, the long-legged South American maned wolf. But it’s not a wolf.

As you might guess, this collection of photos is not safe for work, even though there is nothing particularly prurient about it. One hundred men stood in nearly identical poses, and were then photographed between waist and thighs, and there they are, a hundred weird-looking dinguses in an array.

What’s striking is how much variability there is. It looks to me like evolution has not been paying much attention to this feature: they all work well enough so the differences really don’t matter much. “Normal” is a word that covers a surprisingly wide range here.

This will not end well. Social scientists are happy to see human studies rules relaxed.

If you took Psychology 101 in college, you probably had to enroll in an experiment to fulfill a course requirement or to get extra credit. Students are the usual subjects in social science research — made to play games, fill out questionnaires, look at pictures and otherwise provide data points for their professors’ investigations into human behavior, cognition and perception.

But who gets to decide whether the experimental protocol — what subjects are asked to do and disclose — is appropriate and ethical? That question has been roiling the academic community since the Department of Health and Human Services’s Office for Human Research Protections revised its rules in January.

The revision exempts from oversight studies involving “benign behavioral interventions.” This was welcome news to economists, psychologists and sociologists who have long complained that they need not receive as much scrutiny as, say, a medical researcher.

I would have expected social scientists to be even more acutely aware of the bias of self-interest than us clueless nerds over in the other sciences. If there’s anything we should have learned from the history of scientific experimentation it’s that scientists do not provide good ethical oversight of their own research. Some do seem to know that.

“Researchers tend to underestimate the risk of activities that they are very comfortable with,” particularly when conducting experiments and publishing the results is critical to the advancement of their careers, said Tracy Arwood, assistant vice president for research compliance at Clemson University.

Yes. Onerous and annoying as they are, we have human research review committees to specifically provide input from outside the blinkered perspective of the researcher. That’s necessary. Not everyone sees it that way.

A vocal proponent of diminishing the role of institutional review boards is Richard Nisbett, professor of psychology at the University of Michigan and co-author of the opinion piece in The Chronicle of Higher Education.

Social science researchers are perfectly capable of making their own determinations about the potential harm of their research protocols, he said. A behavioral intervention is benign, he said, if it’s the sort of thing that goes on in everyday life.

“I can ask you how much money you make or about your sex life, and you can tell me or not tell me. So, too, can a sociologist or psychologist ask you those questions,” Dr. Nisbett said.

He’s a psychology professor, and he thinks that in a study in which a professor is asking personal questions of a student, there are no social pressures on the student, and they are completely free to ignore the question? Jesus. I guess I wouldn’t trust any papers published by that guy, then.

If those questions are really benign, then shouldn’t the study proposal fly through the institutional review board process without a hitch?

I mean, I could propose a whole bunch of experiments that involve having students drink lots of vodka before undergoing various cognitive tests, and drinking to excess is the sort of thing students do in everyday life, so it must be benign, and why should I get an IRB to rubber stamp something that the students are doing anyway, am I right?

You don’t want to know what I could argue biology experimenters ought to be able to do without outside assessment because students are already doing it anyway.