Seeing the Ceiling

Say I want to settle the question of who’s more moral, atheists or the religious. I’ve got a lab and a grant and some spare time, (A girl can dream, can’t she?) so I set up the experiment that will solve the question once and for all.

Say I bring a bunch of religious people and a bunch of atheists into my lab. I’ve got two research assistants, one of whom plays as if they’re a participant too. So each time someone, religious or atheist comes into the lab, they sit down next to a stranger (my research confederate), and I call them both into the lab together. They sit down, sign lots and lots of consent forms, and do some silly tasks. None of these tasks matter, they’re just there to distract the participants from realizing what I’m actually paying attention to. At the end of doing all the questionnaires, the confederate stands up, and ‘accidentally’ lets and expensive looking watch fall. Without appearing to notice, they leave the room. I see the watch fall and in a distressed voice, ask the real participant if they don’t mind going to the next room and giving the ‘participant’ his watch back, since I’m so busy entering the data.

Now, the real participant could take the watch, walk into the next room, sprint past the fake participant, and leave with a nice new watch. They are, after all, holding onto a watch, I am looking at my computer and entering data, and the ‘participant’ appeared not to notice they’d dropped a watch.

They could.

But most of them don’t. It doesn’t matter if the participant is religious or an atheist, they tend to pick up the dropped watch, walk into the next room, and give it back to our confederate in this experiment.

Case closed, says I! Religious people and nonreligious people are equally moral! After all, they had equal rates of watch-stealing (that is, none at all).

Not so fast, says you. Practically nobody will steal a watch when you’re just sitting there watching* them! You’re there at your computer, the fake participant is in the next room, and you have their name from participant registration and the consent forms! This is a terrible measure of morality–you have to be fantastically immoral to fail this test! In fact, what you’ve done is determine that nonreligious people and atheists have equally low levels of Horribly Immoral and Brazen Watch Thieves.

In fact, says you (why you’ve come into my laboratory to shout at me, I’m unsure), atheists are more moral! If you made this study more complicated–made it easier to steal the watch without suffering consequences, fewer atheists would steal the watch. You’re wrong, says my religious lab assistant! Fewer religious people would steal the watch!

And all the while, I sit there in puzzlement, because I did this study, right? And I was testing for morality, right? Everybody agrees that stealing a watch is Bad and not stealing a watch is Good.** And my research assistants sit there in outrage, because OBVIOUSLY the [religious/atheists] would be more moral if you made the test harder!

This, dear readers, is the ceiling effect. My bar (or ceiling) for Moral Person is far far too low. Everyone returns the watches, but there’s no way to distinguish between the ones who give the watch back and then glare at puppies on the walk home and the ones who return the watch and wander over to the soup kitchen to volunteer.

Take another, real life example. Jacob says men are better at math than women. Elizabeth says this is clearly false. (Both of them are grievously oversimplifying ‘math’, but we’ll let them get away with it.)

Elizabeth points out that This Math Test (TMT, an official exam given to every high school student in our fictional universe) shows that men and women don’t differ significantly. Therefore, men and women are basically about the same in math ability.

Jacob disagrees. He claims that this test is too easy–that men and women do score the same on the TMT, but that doesn’t mean they have the same ablities–the test is too easy. After all, says he, standardized tests hardly examine the highest possible skill level–they cover basic material. He claims that Elizabeth is just demonstrating the ceiling effect–when you give people a really hard test, men outscore women. Jacob is actually right, but this gap is rapidly shrinking, and men also are overrepresented on the other end–with unusually low math performance.*** (Third section after the abstract, here)

And these ceiling debates play out in a number of parts of psychology research. (And in case you didn’t have enough architecture metaphors in your life, we also have the floor effect.) Here’s a more complicated version of the gender-ceiling issue. You can have sparkling methodology, a huge and representative sample base, but if you’re creating a test with a ceiling problem…you might get entirely unhelpful, or worse, misleading, answers.

This is the best and worst of psychology, for me. That there’s always just a little bit more than the research, always a little bit more to debate and argue and question. Maybe the study is too old, maybe you got a weird subset of the population. Maybe the rats are afraid of the gender you always use for research assistants. Maybe there’s a ceiling. Or a floor.

*sorry, this was unintentional.
…mostly.

**That one girl who stole the watch in order to sell it for medication to save her dying father was dismissed as an outlier. 

***Basically, men have higher variance of performance: they’re some of the best and worst performers. Women have a narrower bell curve of math performance. 

Bayesian Bigots, Bad Charts, Bias: Adventures in Scientific Journals

Bones

“Evidence for JPSP researcher precognition. Although the data for this figure were not actually collected, the evidence it presents clearly demonstrates that such data collection is unnecessary.”

As I finish up my quarter (three papers, one debate, one exam, and a very long packing list left!) I’m flipping through old papers. Here a few of the oddities:

Bayesian Bigot? Statistical Discrimination, Stereotypes, and Employer Decision Making
A serious paper, despite the odd title. In short, curious researchers decide to do a series of in-depth interviews to determine if employers are making rational decisions about who they should hire. In results surprising very few, people discriminate, but don’t do so as the result of updating based on experiences.

Actually, a picture is worth less than 45 words: Narratives produce more false memories than photographs do
Nice figure of speech ya got there. It’d be a shame if we were to…test it.

Memory Distortion in People Reporting Abduction by Aliens
I don’t think anyone is surprised that there’s memory distortion in people who think they’ve been abducted by aliens, but did you know that there’s some overarching characteristics in domains unrelated to green men? Still curious: are there predictive characteristics beforehand?

We Knew the Future All Along : Scientific Hypothesizing is Much More Accurate Than Other Forms of Precognition: A Satire in One Part
Some social psychologist gets annoyed at the publication of Bem’s precognition research, writes snark, gets it published in the same journal that published the ESP research. Also, uses charts like the one found in this post’s heading.

 Sterling (1959) first documented psychologists’ remarkable precognitive capacities. He showed that 97% of articles across a random selection of psychology journals reported positive results. Further, Fanelli (2012) showed that psychologists have kept up this impressive prediction rate through the present. Moreover, his evidence suggests that psychologists have more positive results than virtually every other scientific discipline.

For example, biologists have excellent precognition but still not as good as psychologists (Fanelli, in press), political scientists appear to be guessing randomly (Tetlock, 2005), and economists are wrong about virtually everything (see Economics, all of it).  Psychology is number one!

AND

The evidence for precognition is psychological science itself. Just open a random issue of any psychology journal. In it, you will find dozens of a priori hypotheses anticipating findings that eventually occurred.

 

 

 

 

 

 

I Demand Causation!

alt text from original: Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively while mouthing, “look over there”.

[I do not, in fact, demand causation. At current abilities, this would be wildly unethical to actually get, but the sheer number of things that were listed as correlated and almost definitely, somewhat, probably related was leaving me grumpy. There are these things! They might be related! Brains are incredibly complicated and we’re not really sure how any of the feedback loops and mediating variables could be at play here! But here are correlations! Hopefully they’re the hinty kind!]

Things that I have been informed, via a single paper, are correlated in some degree of statistical significance with being a poor reader at the point of having a developmental dyslexia classification:

-Inability or low ability to ignore background noise when attending to language.

-Dyslexia is highly correlated with having other sensory processing issues. About 50% of people with dyslexia have some other sensory processing impairment.

-Inability to improve sensory processing with repetition. Part of how we pick up language as babies comes from attending to what sounds repeat (This leads us to the theory that language learning in childhood is fundamentally different from in adulthood because young children learn new syllables and syllable combinations, rather than trying to learn at the word-level.) In tasks with lots and lots of trials, good readers improve over time at picking up new syllables but poor readers do not.

–Stop consonants, who are also, delightfully, called plosives, like the sounds in [da], [ga], and [ba] can be tricky to tell apart. High differentiation is correlated with high reading ability in children, poor differentiation between the sounds is correlated with poor reading.

-Language can be divvied up into harmonics, timing, and pitch. Processing of timing and harmonics in the subcortex seem to be correlated with reading capabilities….

-…in fact, in particular with the sound [da. Significant relationship between reading and encoding of timing and harmonics of /da/ (But the pitch-encoding of [da] didn’t seem to matter.)

-Having cortical symmetry in the auditory cortex is (say it with me now) is also correlated with poor reading abilities.

 

Chandrasekaran, B. and Kraus, N. (2012). Biological factors contributing to reading ability: Subcortical auditory function.  In Benasich, A. and Fitch, R. (Eds.), Developmental dyslexia: Early precursers, neurobiological markers and biological substrates.  Baltimore, MD: Brookes. (pp. 83-98).

featured image via xkcd