Building a Science Detector

Oh, let us count the ways…

The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on new ideas and approaches for creating (semi)automated capabilities to assign “Confidence Levels” to specific studies, claims, hypotheses, conclusions, models, and/or theories found in social and behavioral science research. These social and behavioral science Confidence Levels should rapidly enable a non-expert to understand and quantify the confidence they can have in a specific research result or claim’s reliability, reproducibility, and robustness.

First off, “confidence levels?” We’ve already got “confidence intervals,” and there’s been a decades-long push to use them in place of hypothesis testing.[1][2] This technique is fully compatible with frequentism (though over there it doesn’t mean what you think it does), and it even predates null-hypothesis significance testing! Alas, scientists find “we calculate a Cohen’s d of 0.3 +- 0.1” less satisfying to type than “we have refuted the null hypothesis.” The former shows a pretty weak effect, while the latter comes across as bold and confident. If those won’t do, what about meta-analyses? [3]

Second, these “confidence levels” would only apply to published research. Most research never gets published, yet those results are vital to understanding how strong any one finding is.[4] We can try to estimate the rate of unpublished works, and indeed over the decades many people have tried, but there is no current consensus on how best to compensate for the problem.[5][6][7]

Thirdly, “social and behavioral science?” The replication crisis extends much farther, into biomedicine, chemistry, and so on. Physics doesn’t get mentioned much, and there’s a reason for that (beyond their love of confidence intervals). Emphasis mine:

Even if you adjust the acceptable P value, a test of statistical significance, from 0.05 to 0.005—the lower it is, the more significant your data—that won’t deal with, let’s say, bias resulting from corporate funding. (Particle physicists demand a P value below 0.0000003! And you gotta get below 0.00000005 for a genome-wide association study.)

Just think on that. “p < 0.0000003″ means “if the null hypothesis is true, we would find a more extreme result in less than 1 in 3,333,333 trials on data like what we have observed.” If you wanted to see one of those exceptions, you’d have to do one experiment a day for 6,326 years just to have a better than 50/50 chance of spotting it. For comparison, the odds of a particular US citizen being struck by lightening over a year are 1 in 700,000; worldwide, the yearly odds of death by snake bite are about 1 in 335,000; and over the lifetime of a US citizen, the odds of them dying by dog attack are 1 in 112,400. p < 0.0000003 is a ridiculously high bar to leap, which means either a) false positives are easy to generate in physics, either via the law of large numbers or shoddy statistical techniques, or b) the field has been bitten so many times by results that can’t be replicated, even when they were real, that they’ve cranked the bar ridiculously high, or c) both.

Fourth, confidence isn’t everything. The Princeton Engineering Anomalies Research lab did studies where people tried to psychically bias random number generators. Over millions of trials, they got extremely significant results… but the odds of success were still around 50.1% vs. the expected 50%. Were they now confident that psychic abilities exist, or merely that luck and reporting bias could introduce a subtle skew into the data? Compacting those complexities into a number or label that a lay-person can understand is extremely difficult, perhaps impossible.

Basically, what DARPA is asking for has been hashed out in the literature for decades, and the best recommendations have been ignored.[8] They may have deep pockets and influence, but what DARPA wants requires a complete overhaul in how science is conducted across the globe, spanning everything from journals to how universities are organized.[9] When even quite minor tweaks to the scientific process are met with stiff opposition, pessimism seems optimistic.

[1] Gardner, Martin J., and Douglas G. Altman. “Confidence intervals rather than P values: estimation rather than hypothesis testing.” Br Med J (Clin Res Ed)292.6522 (1986): 746-750.

[2] Rozeboom, William W. “The fallacy of the null-hypothesis significance test.” Psychological bulletin 57.5 (1960): 416.

[3] Egger, Matthias, et al. “Bias in meta-analysis detected by a simple, graphical test.” Bmj 315.7109 (1997): 629-634.

[4] Rosenthal, Robert. “The file drawer problem and tolerance for null results.” Psychological bulletin 86.3 (1979): 638.

[5] Franco, Annie, Neil Malhotra, and Gabor Simonovits. “Publication bias in the social sciences: Unlocking the file drawer.” Science 345.6203 (2014): 1502-1505.

[6] Rosenberg, Michael S. “The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis.” Evolution 59.2 (2005): 464-468.

[7] Simonsohn, Uri, Leif D. Nelson, and Joseph P. Simmons. “P-curve: a key to the file-drawer.” Journal of Experimental Psychology: General 143.2 (2014): 534.

[8] Sedlmeier, Peter, and Gerd Gigerenzer. “Do studies of statistical power have an effect on the power of studies?.” Psychological bulletin 105.2 (1989): 309.

[9] Rawat, Seema, and Sanjay Meena. “Publish or Perish: Where Are We Heading?” Journal of Research in Medical Sciences : The Official Journal of Isfahan University of Medical Sciences 19.2 (2014): 87–89. Print.

Sex Around the World

Oh, Jerry Coyne. I’m amused with his defense of a sex binary…

In Drosophila and humans, the two species with which I’m most familiar, the behavior, appearance, and primary and secondary sex characteristics are determined almost completely by whether the chromosomal constitution is male (XY) or female (XX).

… since, like most such “scientific” defenses, he immediately turns around and shoots it in the foot.

Yes, there are a few exceptions, like AIS, but the various forms of that syndrome occur between 1 in every 20,000 to 1 in only 130,000 births.  Is that “too many examples” to all0w us to say that biological sex is not connected with chromosomes? If you look at all cases of intersexuality that occur in people with XX or XY chromosomes (we’re not counting XOs or XXYs or other cases of abnormal chromosomal number), the frequency of exceptions is far less than 1%. That means that, in humans as in flies, there is almost a complete correlation between primary/secondary sex characteristics and chromosome constitution.

Ah yes, chromosomes determine human sex except in the 0.05% to 1.7% of cases where they don’t. Brilliant logic, that.

But it’s easy to get trapped by your filter bubble. The internet is a lot bigger than North America, after all, and other places have their own view of sex. Take Sweden, for instance, where it’s  government policy to avoid teaching gender stereotypes. One kindergarten made headlines not too long ago by declaring itself “gender-neutral.” As the founder put it,

00:10:10,909 –> 00:11:03,329
I’m going to show you what we call the “whole life spectra.” We tend to divide this life spectra into two pieces, one for boys and one for girls. More often pink is for girls, and blue is for boys. When we call a boy “cool” and “strong,” and to girls we more often say that they should be “helpful,” “nice,” “cute,” we have different expectations [for how they behave]. We take away this border, and we don’t separate into “boyish” and “girlish,”  we give the whole life spectra to everyone. So we are not limiting, we are just adding. We are not changing the children, we are changing our own thoughts.

That video is worth watching, as it follows around two gender non-conforming kids with an intersex “ma-pa.” The few bigots on screen seem right out of 1984, claiming that expanding or eliminating gender stereotypes somehow constrains kids in some mysterious fashion. Every kid, in contrast, is either at ease with gender role fluidity or made uncomfortable when asked to label their gender.

But even Sweden appears behind the curve when contrasted with the Khawaja Sira of South Asia.

For centuries, South Asia has had its own Khawaja Sira or third gender culture. The community, identifying as neither male nor female, are believed by many to be “God’s chosen people,” with special powers to bless and curse anyone they choose. The acceptance of Khawaja Sira people in Pakistan has been held up internationally as a symbol of tolerance, established long before Europe and America had even the slightest semblance of a transgender rights movement.

But the acceptance of people defining their own gender in Pakistan is much more complicated. The term transgender refers to someone whose gender identify differs from their birth sex. This notion is yet to take root in Pakistan and the transgender rights movement is only beginning to assert itself formally. Now, some third gender people in Pakistan say the modern transgender identity is threatening their ancient third gender culture.

The problem is that the Khawaja Sira are allowed to exist within South Asian culture because they renounce both male and female gender roles, thus don’t challenge either. Trans* people, on the other hand, reject the role assigned to the Khawaja Sira and invoke the male or female one instead. This upsets every gender’s apple cart. It doesn’t help either that the Khawaja Sira in Pakistan have recently fallen onto hard times, facing increasing bigotry and hate; the increasing number of trans* people feels like an invasion of “Western” ideals, at a time when their community is ill-equipped to cope.

But do you remember hearing about Oyasiqur Rhaman, the atheist blogger murdered in Bangladesh? His murderers were outed by a courageous “hijra,” which is similar in meaning to “Khawaja Sira” but not quite the same.

Transgender people occupy an unusual social stratum in South Asia, where conservative societies still consider same-sex intercourse to be a crime but also allow the existence of a third gender — a well-established category that dates back to the age of the “Kama Sutra.” Nepal, Pakistan, Bangladesh and India have all legally recognized the existence of a third gender, including on passports and other official documents.

In India, in fact, “kinnar” freely mixes gender identity with non-binary sex. Compare and contrast this with Mexico’s “muxes,” who are called a third gender but in practice act more like trans* women, and Balkan sworn virgins who are more like trans* men. There’s no intersex component to the latter two, so lumping everybody under the banner of “third gender” or “transgender” is quite misleading.

Our binary view of sex and gender seem terribly archaic (which is ironic, as it may be a recent invention). It should not be controversial in North America to have a non-conforming parent or be raised in a genderless environment, yet it is. We could learn a thing or two from the rest of the world, especially when it comes to sex.

“Science Is Endangered by Statistical Misunderstanding”

He’s baaaaaack. I’ve linked to David Colquhoun’s previous paper on p-values,[1] but I’ve since learned he’s about to publish a sequel.

Despite decades of warnings, many areas of science still insist on labelling a result of P < 0.05 as “significant”.   This practice must account for a substantial part of the lack of reproducibility in some areas of science. And this is before you get to the many other well-known problems, like multiple comparisons, lack of randomisation and P-hacking. Science is endangered by statistical misunderstanding, and by university presidents and research funders who impose perverse incentives on scientists. [2]

[Read more…]

P-values are Bullshit, 1942 edition

I keep an eye out for old criticisms of null hypothesis significance testing. There’s just something fascinating about reading this…

In this paper, I wish to examine a dogma of inferential procedure which, for psychologists at least, has attained the status of a religious conviction. The dogma to be scrutinized is the “null-hypothesis significance test” orthodoxy that passing statistical judgment on a scientific hypothesis by means of experimental observation is a decision procedure wherein one rejects or accepts a null hypothesis according to whether or not the value of a sample statistic yielded by an experiment falls within a certain predetermined “rejection region” of its possible values. The thesis to be advanced is that despite the awesome pre-eminence this method has attained in our experimental journals and textbooks of applied statistics, it is based upon a fundamental misunderstanding of the nature of rational inference, and is seldom if ever appropriate to the aims of scientific research. This is not a particularly original view—traditional null-hypothesis procedure has already been superceded in modern statistical theory by a variety of more satisfactory inferential techniques. But the perceptual defenses of psychologists are particularly efficient when dealing with matters of methodology, and so the statistical folkways of a more primitive past continue to dominate the local scene.[1]

… then realising it dates from 1960. So far I’ve spotted five waves of criticism: Jerzy Neyman and Egon Peterson head the first, dating from roughly 1928 to 1945; a number of authors such as the above-quoted Rozeboom formed a second wave between roughly 1960 and 1970; Jacob Cohen kicked off a third wave around 1990, which maybe lasted until his death in 1998; John Ioannidis spearheaded another wave in 2005, though this died out even quicker; and finally the “replication crisis” that kicked off in 2011 and is still ongoing as I type this.

I do like to search for papers outside of those waves, however, just to verify the partition. This one doesn’t qualify, but it’s pretty cool nonetheless.

Berkson, Joseph. “Tests of Significance Considered as Evidence.” Journal of the American Statistical Association 1942;37:325-35. International Journal of Epidemiology, vol. 32, no. 5, 2003, pp. 687.

For instance, they point to a specific example drawn from Ronald Fisher himself. The latter delves into a chart of eye facet frequency in Drosophila melanogaster, at various temperatures, and extracts some means. Conducting an ANOVA test, Fisher states “deviations from linear regression are evidently larger than would be expected, if the regression were really linear, from the variations within the arrays,” then concludes “There can therefore be no question of the statistical significance of the deviations from the straight line.”

Berkson’s response is to graph the dataset.eye facets vs. temperature, Drosophila Melangaster, graphed and fit to a line. From Fisher (1938).

The middle points look like outliers, but it’s pretty obvious we’re dealing with a linear relationship. That Fisher’s tests reject linearity is a blow against using them.

Jacob Cohen made a very strong argument against Fisherian frequentism in 1994, the “permanent illusion,” which he attributes to a paper by Gerd Gigerenzer in 1993.[3][4] I can’t find any evidence Gigerenzer actually named it that, but it doesn’t matter; Berkson scoops both of them by a whopping 51 years, then extends the argument.

Suppose I said, “Albinos are very rare in human populations, only one in fifty thousand. Therefore, if you have taken a random sample of 100 from a population and found in it an albino, the population is not human.” This is a similar argument but if it were given, I believe the rational retort would be, “If the population is not human, what is it?” A question would be asked that demands an affirmative answer. In the hull hypothesis schema we are trying only to nullify something: “The null hypothesis is never proved or established but is possibly disproved in the course of experimentation.” But ordinarily evidence does not take this form. With the corpus delicti in front of you, you do not say, “Here is evidence against the hypothesis that no one is dead.” You say, “Evidently someone has been murdered.”[5]

This hints at Berkson’s way out of the p-value mess: ditch falsification and allow evidence in favour of hypotheses. They point to another example or two to shore up their case, but can’t extend this intuition to a mathematical description of how this would work with p-values. A pity, but it was for the best.

[1] Rozeboom, William W. “The fallacy of the null-hypothesis significance test.” Psychological bulletin 57.5 (1960): 416.

[2] Berkson, Joseph. “Tests of Significance Considered as Evidence.” Journal of the American Statistical Association 1942;37:325-35. International Journal of Epidemiology, vol. 32, no. 5, 2003, pp. 687.

[3] Cohen, Jacob. “The Earth is Round (p < .05).” American Psychologist, vol. 49, no. 12, 1994, pp. 997-1003.

[4] Gigerenzer, Gerd. “The superego, the ego, and the id in statistical reasoning.” A handbook for data analysis in the behavioral sciences: Methodological issues (1993): 311-339.

[5] Berkson (1942), pg. 326.

Change Of Plans

I’ve had a draft cooking for a while over Laci Green’s view of trans* people. I don’t claim to know why she’s hanging out with MRAs or treating TERFs as if they were feminists, but if she’s going to sit down and attempt to make logical arguments the least I could do is return the favor.

But then this happened. [Read more…]

A Quick Note on So-Called “Bathroom Bills”

[CONTENT WARNING: TERFs]

Gendered restrooms are a relatively recent phenomenon. Before then restrooms were unisex, but not in the way you’re thinking.

… public facilities in Western nations were male-only until the Victorian era, which meant women had to improvise. If they had to be out and about longer than they could hold their bladders, women in the Victorian era would urinate over a gutter (long Victorian skirts allowed for some privacy). Some would even carry a small personal device called a urinette that they could use discretely under their skirts and then pour out, [Sheila] Cavanagh said. […]

This lack of female facilities reflected a notable attitude about women: that they should stay home. This “urinary leash” remains a problem in some developing nations, said Harvey Molotch, a sociologist at New York University and co-editor of “Toilet: The Public Restroom and the Politics of Sharing” (New York University Press, 2010). Women in India today, for example, often have to avoid eating or drinking too much if they have to be out in public, because there is no place for them to go, Molotch told Live Science.

But with the rise of the Industrial Revolution and changing attitudes towards gender, forcing women back into the home wasn’t tenable. Instead, during the last quarter of the 19th century a new philosophy became dominant.

Scientific discoveries at the time showed that working women were “unable to [physically] withstand strains, fatigues, and privations as well as [men],” so sex-separated restrooms provided “a protective haven . . . where a woman could seek comfort and rest when her weak body gave out on the job.” Maintaining separate facilities that were “properly screened” also provided more privacy to both men and women with regard to their bodies and bodily functions, an obsession derived from Victorian society. By providing a separate space for the special needs of women and protecting the privacy of all workers, sex-separated bathrooms upheld “[l]ate nineteenth century concerns about germs and sanitation . . . [and] early nineteenth century ideological concerns of pure womanhood.”

Governments began mandating sex-segregated washrooms in the workplace, starting with Massachusetts in 1887. As attitudes towards women changed, however, the reasons for segregation shifted.

Though modern thinking has certainly progressed and women are not treated as inherently inferior as they once were, the current argument that sex-separated restrooms provide greater safety for women harkens back to the nineteenth century justifications for separate restroom facilities. For example, literature opposing transgender bathroom access focuses heavily on protecting the safety, privacy, and dignity of women and girls, yet rarely mentions any issues men might have with sharing a restroom with a female-to-male transsexual. Even some transsexual women wish to maintain the “safe haven in a male dominated world” of a women’s restroom “where women can have their own space without needing to worry what a man might do (in front of them, to them, or to their daughters and young sons.)” At the very least, these opinions expose an underlying belief that women and girls are more fragile than men, have a deeper need of privacy than men, and are more likely than men to be afraid or offended by the notion of sharing a restroom with a male-born transgender woman.

Faced with this information, you’d think a feminist would tread very carefully. Yes, there’s a gender imbalance in who commits sexual assault, but the historic use of washrooms to control women should give pause about banning someone else from using them.

TERFs don’t pause, they’re fully in favour of “bathroom bills.” Even when a butch lesbian gives a convincing plea against this legislation, they still find a way to justify support.

Those of us who believe that men belong in the men’s washroom come in two major types—conservatives and feminists—but this author doesn’t distinguish between the two groups. Conservatives understand that certain men will use any excuse to prey on women and children and they want to protect them. They are also homophobic and do not accept ordinary lesbians and gays, and they promote traditional gender roles and marriage. Feminists know that men with sexual fetishes like to declare that they have a gender identity and therefore have a right to expose themselves in women’s locker rooms. We differ completely from conservatives because we are against gender roles and sex stereotypes. We want the entire range of women in all our diversity to feel comfortable in women’s spaces, which will be accomplished by eliminating sexism and homophobia. […]

She’s implying here that the reason for sex-segregated facilities is the misguided notion that women need protection from men, and that people only believe that women need protection because of gender roles/stereotypes about women. But in the real world, women do need protection from men, because men abuse women on a regular basis through assault, rape, harassment, stalking, flashing, taking photos without consent, and the list goes on. Unfortunately this writer didn’t check the stats on violence against women before writing her article.

This is evidence that TERFs are not truly feminists: they advocate for the elimination of sex stereotypes, yet push a stereotyped view of sex. They are ignorant of feminist history, and advocate for sexist policies that date back to the Victorian era. The aforementioned division between “conservatives” and “feminists” is rich, especially since the two love to team up to oppose the rights of trans* people.

The real issue behind “bathroom bills” is control over who gets to enjoy the public sphere, security is secondary at best.

Mystery Solved

I’m surprised I don’t read Wonkette more often.

Rachel Maddow did a BIG SCOOP on Thursday night, and we think it’s a pretty big fuckin’ deal. To cut to the chase, somebody (she doesn’t know who YET) used her “Send It To Rachel” tool to send her something that looks like a highly classified document about collusion between Donald Trump and Russia, but is actually a FORGERY. WHOA IF TRUE, right?

It is pretty “whoa,” in fact I was about to sit down and type something up on it until I saw Wonkette scooped me.

What’s fascinating about this weird forgery is that it appears to have been copied off the highly classified document NSA contractor Reality Winner sent to Glenn Greenwald’s The Intercept. Remember how The Intercept published a bombshell on Monday, June 5, that Russians had specifically targeted voting machine manufacturers and election officials during their 2016 cyberwar against American democracy, and that they got further than anybody ever knew? […]

Maddow found the EXACT SAME MARKINGS and the EXACT SAME CREASE on the document she got. Forgery detected! (Later in the segment she explained that there were several other screwy things about the document, including that it actually named a high-up American citizen/Trump campaign person. According to the intelligence experts Maddow consulted, this type of document, if real, wouldn’t name an American all willy-nilly like that.)

There was one intriguing mystery left: the file received by Maddow was created on June 5th, 2017, at 12:17:15, yet the Intercept’s article went online at 13:44 15:44. How could the person who sent the document get access to it before the article was published? I was about to sit down and type about that instead, but…

That’s because time stamps on the documents published by The Intercept designate the creation date included in the PDF we publish on DocumentCloud: In this case, that occurred just over three hours prior to publication of our article. Both versions — the one we published and the one Maddow received — reflect the same time to the second: literally the exact moment when we created and uploaded the document.

In other words, anyone who took the document directly from The Intercept’s site would have a document with exactly the same time stamp as the one Maddow showed. Thus, rather than proving that this document was created before The Intercept’s publication, the time stamp featured by Maddow strongly suggests exactly the opposite: that it was taken from The Intercept’s site.

Ah, thank you Glenn Greenwald. It looks like the Intercept has an automated system to process their documents. Downloading the original for myself, I can tell they use an old-ish copy of ImageMagick to do the grunt work. This probably helps them redact information; the boxes they use to cover information look digitally made, yet are burnt into the source images that make up the PDF. This could have the pleasant side-effect of wiping away the original document’s metadata, if it was digital. On the other hand, I also see the original title was “GRU-final,” which probably didn’t come from the Intercept.

I get something slightly different from Greenwald when I dump the document’s info, though.

File Modification Date/Time : 2017:06:05 13:43:03-06:00
PDF Version : 1.4
Linearized : No
Create Date : 2017:06:05 12:17:15
Modify Date : 2017:06:05 12:17:15
Page Count : 5

In his case, the bolded bit reads “2017:07:06 21:33:15-04:00,” the exact time he downloaded his copy. My tool is slightly newer than his, however, which could easily explain the discrepancy.

So, that’s one mystery solved: the person or people who sent the document to Maddow used the Intercept’s document as a base. That still leaves who sent it, though. Was it the Kremlin,  someone associated with Trump, or somebody else? That one is in the hands of Maddow’s team.

(A hat tip to Lynna, OM in PZ’s Political Madness thread, for the Wonkette article.)

[HJH 2017-07-08: Damn time zones. And I was even going to mention them in my original post…]

The Ouroboros of Hate

I’ll confess I’ve said that if bigots were smart, they wouldn’t be bigots. Reality is a bit more complicated than that, but there is a way to rescue the sentiment.

  1. Opponents of Social Justice movements generally have a poor grasp of social justice concepts.
  2. As a consequence, some of them think these concepts lack any firm meaning. They instead act either as in-group/out-group signifiers, or as synonyms for “I don’t like you.”
  3. As a consequence, some of them have difficulty telling if these concepts are used in their proper manner.
  4. A few opponents of social justice, motivated either by a desire to show #2 to be true or simply to grief, will stage faux social justice campaigns.
  5. As a consequence, the subset mentioned in #3 will think the opponents from #4 are sincere, and given enough exposure may start thinking social justice concepts lack meaning.

I’ve seen this in action; while one group of bigots were trolling me, I saw another group think the trolling was sincere. Just recently, I spotted another example.

Older members of the crowd carried Confederate flags, while the younger, internet-driven masses wore patches with 4chan’s Kekistan banner. Rally-goers in homemade armor and semi-automatic rifles paced Houston’s Hermann Park, waiting for an enemy to appear.

The crowd, several hundred strong, gathered in the park on Saturday to defend a statue of Sam Houston, a slaveholder. They had gathered in response to reports that leftist protesters had planned a rally to remove the statue, despite Houston Mayor Sylvester Turner publicly stating that removing the statue wasn’t “even on my agenda.” But as sniper rifles and Infowars-branded jackets crowded the park, it became evident that the left protesters were not coming. They had never planned to come. The rumors of an antifa protest were actually a hoax, orchestrated by an anti-left group defending Confederate monuments.

I suspect these scenarios are more common than we realize, if only because the same thing happened again a month later.

A “patriot” who brought a revolver to Gettysburg National Military Park Saturday amid rumors of desecration of memorials accidentally shot himself in the leg Saturday. […]

Dozens of self-described Patriots came to the park about noon Saturday after hearing rumors that Antifa protesters might crash the park’s events and try to desecrate memorials. Members of Antifa caused a ruckus in Harrisburg recently at an Anti-Sharia rally and one member was arrested for swinging a wooden pole with a nail attached at a police horse.

The rumors on Saturday appeared to be just that: rumors, as no Antifa members were seen at Gettysburg park Saturday.

The result of all this is a self-supporting feedback loop, where people opposed to social justice keep getting fooled by false flags into thinking social justice is as loopy as they’ve been told, and some of them graduate to generate those false flag campaigns.