Biased Tests

[cn: Bayesian math]

Suppose that I create a test to measure suitability for a particular job. I give this test to a bunch of people, and I find that women on average perform more poorly. Does this mean that women are less suitable for the job, or does it mean that my test is biased against women?

Psychologists do this all the time. They create new tests to measure new things, and then they give the tests to a variety of different groups to observe average differences. So they have a standard statistical procedure to assess whether these tests are biased.

But I recently learned that the standard procedure is mathematically flawed. In fact, rather than producing an unbiased test, the standard procedure practically guarantees a biased test. This is an issue that causes much distress among psychometricians such as Roger Millsap.

Following Millsap, I will describe the standard method for assessing test bias, sketch a proof that it must fail, and discuss some of the consequences.

[Read more…]

Paper: Attack of the psychometricians

Suppose that you want to demonstrate that baby boomers are more narcissistic than other generations, or that women are more agreeable and neurotic than men, or that people of different races have different amounts of intelligence. How do psychologists do that? Can they in fact do that?

Typically, the method is to come up with a bunch of questions that superficially appear to measure the intended characteristic. Then the questions are “validated”, for example, by making sure the questions all correlate with one another. Once the questionnaire is declared valid, psychologists can then measure a variety of different groups and make far-reaching claims about how our current political/social situation was caused all along by the thing that they happen to study.

If you find this methodology questionable, but aren’t sure exactly what went wrong, you might be interested in hearing about psychometrics, the field concerned with psychological measurement. According to psychometricians, part of the problem is that psychologists are failing to follow best practices. That is the subject of this paper:

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440.

[Read more…]

Christian Doubt

This is a repost of an article from 2014. Usually I like to repost articles that are related to my recent topics, but this is unrelated and just for fun.

When I grew up in Catholicism, I was never taught to think that doubt was a bad thing.  In fact, doubt was a good thing, ennobling even.  Doubts were something that everyone experiences.  Why then, is it said that Christianity is all about faith, dogma, and purging all doubt?  Where does this image come from?

Let me tell you what happened next.  I started doubting Catholicism.  And even though I was never taught that doubting was bad, I knew that the particular way I was doing it was bad.

What I was doing was reading on some arguments against Catholic beliefs, comparing them to the arguments for it.  I knew that changing my mind on so many things all at once was impossible, so I considered each issue independently, one at a time.  I worried about the consequences of deciding one way or the other, but I tried not to let that affect my judgment.  Finally, I collected my many thoughts and tried to draw some overall conclusions on Catholicism and God.

In my mind, this is more or less the proper way to deal with doubt, so why did I know in my gut I was running afoul of some rule of my religious upbringing?  The truth is that doubt was accepted in the Catholicism I grew up in, but only if the doubt fit into a specific narrative.  Doubt was not an epistemological tool, but a personal struggle to be overcome.  This is a fundamentally negative depiction of doubt.

[Read more…]

Boghossian is no Sokal

Peter Boghossian and James Lindsay (henceforth B&L) have an article titled “The Conceptual Penis as a Social Construct: A Sokal-Style Hoax on Gender Studies” on Skeptic.com.  The article describes a nonsense paper submitted, and accepted, to the journal Cogent Social Sciences.  The authors consider it an indictment of gender studies and pay-to-publish journals.

This being a Sokal-style hoax, it’s worth recapping some of the strengths and weaknesses of the original Sokal hoax.  First the weaknesses:

  • Sokal’s paper was accepted to The Social Text, which is a journal of only mediocre impact.
  • Peer review isn’t intended to weed out bad faith actors, but to enforce some minimum standard.  The real test is later, when the academic community cites (or ignores) the publication.
  • Sokal only had N=1. Distinguishing between good and bad papers is in general a difficult problem, and one expects that in the perfect balance, some good papers would be rejected, and bad papers accepted.

Now the strengths:

  • The Sokal hoax is immediately compelling to general public, even when people don’t look into the details.  There’s value in bringing the issue to popular attention.
  • When I did look into the details last year, I found the paper’s content to be a damning indictment of the entire field:

    It’s not simply that Sokal liberally salts his article with absurdities, it’s that he quotes plenty of postmodern academics doing the same damn thing.

    Even if Sokal’s paper were rejected, one would have to account for all the nonsense already published and respected within the field.

  • There was a clear way that The Social Text could have avoided being hoaxed, if anywhere in the review process they had asked someone in physics, biology, or math to glance at it.

B&L’s and  attempt at a hoax falls short of Sokal, having worse weaknesses, and missing important strengths.

[Read more…]

How to actually avoid generalizations

“Don’t make assumptions.” “Criticize the idea, not the person.” “Avoid generalizations.”

These are a few common rules about polite conversation. But they are broken so systematically that it raises the question of whether the rules are any good. One may vocally oppose generalizations, and in the next breath make a sweeping generalization of their own.

It seems that when someone else makes assumptions or generalizations, we hate it. But when we ourselves have the opportunity, we suddenly remember that assumptions and generalizations have some redeeming value after all. And when we next hear someone else make a generalization, we again forget what that value was.

I assert that the value of a generalization is quite simple. People like to state opinions, they like to hear opinions, and they like to use them to inform behavior. They also like to consider opinions and even disagree with them. And if the opinion is stronger by way of generalization, then all the better.

The question for me is not why we like generalizations, but why some generalizations turn out so wrong. What is the source of our aversion? And how can we avoid the kind of generalizations that produce such negative reactions?
[Read more…]

I used to think Santa was a myth

I know I said I’m on blogging break, but I still want to do my monthly repost thing.  This is a classic I wrote in 2011.

‘Tis the season for anecdotes…

I didn’t ever take Santa very seriously when I was younger. Or at least, not as far as I can recall. And I thought that no one else took Santa seriously either.

I mean, kids believing in Santa, that’s just something that happens in the movies, right? There are countless movies depicting little kids who believe in Santa Claus. They’ll write letters to Santa. They’ll wait excitedly at the stairs for Santa to come, deliver presents, and eat the cookies and milk. Kids believe in all these elaborate legends and rituals, sometimes even in the face of disbelief from their parents or older kids.

Of course, in these movies, Santa also happens to be real. But Santa isn’t real. So why should I think that belief in Santa be real? For me, belief in Santa was all part of the mythos, along with the elves, reindeer, and red suit. [Read more…]

Made in Criticalland

This is a repost of an article I wrote in 2014.  Relevant to my recent review of the Sokal paper.  Note that the blog Scientia Salon is now defunct.

Massimo Pigliucci started a new blog Scientia Salon, which is already bearing fruits.  I enjoyed this essay by Alan Sokal (yes, that Sokal) about academic postmodernists and extreme social constructivists.  In the 80s and 90s there were many such academics claiming that science was entirely based on prejudices.  Interestingly, Sokal claims that they have now backed off from the most extreme views, particularly because they were upset at the way the Bush regime used postmodernism to justify its anti-science policies.

Sokal’s primary citation for this is “Why has critique run out of steam? From matters of fact to matters of concern” by sociologist of science Bruno Latour in 2004.  I thought it was worth a read. [Read more…]