To a first approximation, science is about differences between groups: if death is a major motivator for human behavior, then we should expect people who have been reminded of their mortality to act differently than people who were not. Sometimes, a group is only conceptual: if the Higgs boson exists, then when slamming two protons together there should be more photons observed in the aftermath, compared to the amount we predicted. Sometimes, we generate groups after the fact: if we plot star colour and luminosity on a Hertzsprung-Russell diagram, stars naturally settle into four major groups. Sometimes, the “difference” we care about is that there is no difference at all: if a cosmetic is safe to use, then if we compare a group of people who use it to people who don’t, we should observe no difference in health. These divisions are so common that we often neglect to clearly delineate our groups: “does a daily dose of Aspirin prevent strokes?” implies that people who take Aspirin are less likely to get a stroke than people who do not.
At some point these groups must be clearly delineated, however; when they are not, a common problem in epidemiology, we lose our ability to find differences between them. Worse, fuzzy groups allow us to manufacture differences that don’t exist, say by classifying legitimate data as illegitimate “outliers” to get the results we want. This “differences between groups” metaphor is surprisingly powerful, to the point that’s a good solution to the demarcation problem. A core claim of astrology is that people differ based on the day they were born; if we divide people into those groups, yet fail to find differences, then astrology cannot be true. The Myers-Briggs personality test claims that we can divide people into specific groups, yet studies that use difference to reconstruct groups have failed to see those groups materialize.
By convention, if we are testing whether or not some change leads to a difference, we call the group we don’t change the “control” group. This group is often conceptual, thanks to frequentist statistical techniques, but that only works if the tools we use to find difference are perfectly calibrated; if they are not, the data might be biased and you’d never know. As a result, lacking a control group is considered a reason to suspect your results.
I apologize if all that was painfully obvious; I grasped these concepts way back in Junior High, well before I was legally allowed to drive. Still, I needed to type it out to convey the pain of what comes next.
Other online commenters said the hoax papers lack a control group of papers for comparative purposes.
Pluckrose, Lindsay, and Boghossian, reached by phone in Portland, said the papers that were rejected serve as a control of sorts.
If you are proposing that it is easy to hoax “grievance studies” journals, then you are dividing all scientific journals into two groups, and proposing a difference between those two groups. This trio has only one group, “grievance studies” journals, so they lack any ability to observe a difference between groups.
Remember their methodology? If a paper was rejected, it was re-written until it was accepted. There is no boundary between “accepted” and “rejected” papers, so they cannot be used to form a control and test group. For that matter, there’s no commonality between “accepted” paper that demarks them as a group. Suppose one published paper was in a journal that had lax quality standards; another was in one that was rigorous, but it got through because the authors just happened to write a good “grievance studies” paper without realizing it. We can’t tell if a paper would be accepted or rejected by looking at its quality, hence we have no ability to group them. As I pointed out last time, they also lack any way to differentiate “grievance studies” journals from non-“grievance studies” ones.
This is basic shit, painfully obvious to anyone who’s taken a few science courses.
Better yet, they said, consider this meta-control thought experiment: Look at your journals and the articles they published, and see if you can distinguish them from the hoax articles. If the answer is often no, then there is your control.
Quick, is this a hoax article?
It has been notoriously difficult to construct a meta-stable de Sitter (dS) vacuum in string theory in a controlled approximation. This suggests the possibility that meta-stable dS belongs to the swampland. In this paper, we propose a swampland criterion in the form of |∇V| ≥ c·V for a scalar potential V of any consistent theory of quantum gravity, with a positive constant c. In particular, this bound forbids dS vacua. The existence of this bound is motivated by the abundance of string theory constructions and no-go theorems which exhibit this behavior. We also extend some of the well-known no-go theorems for the existence of dS vacua in string theory to more general accelerating universes and reinterpret the results in terms of restrictions on allowed scalar potentials.[1]
I hope you’ve realized by now the world is complex. If you want to exhaustively study parts of it, you’re naturally going to develop specialized language and implicit assumptions. These act as shortcuts, so every time you publish you’re not forced to explain what “de Sitter space” or “string theory” is to people who already know what they are. Unfortunately, people who don’t know are stuck without an understanding of the common language and assumptions of your field. These people lack the ability to tell if someone is getting the basics right, and therefore lack the ability to tell hoaxes from legitimate work.
Again, this is basic shit, familiar to anyone with a passing knowledge of science, and these jokers don’t understand it!
[Yascha] Mounk, by phone, also said the control-group criticism is misguided. He called it a “confused attempt to import statistics into a question where it doesn’t apply.” If the authors were claiming that their work proves that some publications are, say, 50 percent more susceptible to hoaxes than the average, or that 100 percent of articles published are nonsense because these seven articles were accepted, then you would obviously need controls. But the authors “do nothing of the sort. They demonstrate that it’s possible, with relatively little effort, to get bullshit published.”
But we also know bullshit gets published, with relatively little effort, in non-“grievance studies” journals. SCIgen is a program that automatically generates Computer Science papers. The authors were able to get one such paper accepted to a scientific conference in 2005. Eight years later, some researchers discovered that a whopping 120 SCIgen papers had been published across thirty conferences. That same year, John Bohannon submitted a paper on “medicinal lichen” that was machine translated from English to French and back again to 300 open-access journals; it was accepted in 157, and of the 36 submissions where peer reviewers caught the hoax a whopping 16 journals published the paper anyway. Not recent enough for you? While Boghossian and his two friends were toiling away on twenty papers, one person submitted an obvious hoax paper to fourteen biology journals; it got published in three journals and accepted in a further five.
Again, their core claim is that “grievance studies” journals are more prone to hoaxes than non-“grievance studies” journals. If we use my examples as controls, their hypothesis lacks sufficient evidence to be considered true; if we do not, then they don’t have control group and their hypothesis lacks sufficient evidence to be considered true.
Boghossian, Pluckrose, Lindsay, and Mounk do not have the science knowledge of an alert High School student. They should be deeply ashamed and laughed out of academia, instead of rewarded with wingnut welfare.
[1] Obied, Georges, et al. “De Sitter space and the swampland.” arXiv preprint arXiv:1806.08362 (2018).