Science magazine has published the formal criticisms of the claim to have found extremophiles that substituted arsenic for phosphorus in their chemistry. It’s a thorough drubbing, and the most disappointing part is that Wolfe-Simon’s rebuttal simply insists that they were right, and doesn’t even acknowledge that many valid criticisms of the study were made. That’s not how you do it. Instead, she should answer the complaints by saying that they will do the experiments in a way that specifically addresses the perceived shortcomings of the study; she and her lab have their credibility invested in this research now, and the answer to the barrage is not to batten down the hatches and do nothing, but to do more experiments to show that the critics are wrong.

Nature also has a discussion of the issues, and this article bothered me in other ways. It rationalizes not doing anything to replicate or refute the work.

However, most labs are too busy with their own work to spend time replicating work that they feel is fundamentally flawed, and it’s not likely to be published in high-impact journals. So principal investigators are reluctant to spend their resources, and their students’ time, replicating the work.

“If you extended the results to show there is no detectable arsenic, where could you publish that?” said Simon Silver of the University of Illinois at Chicago, who critiqued the work in FEMS Microbiology Letters in January and on 24 May at the annual meeting of the American Society for Microbiology in New Orleans. “How could the young person who was asked to do that work ever get a job?” Silver said.

It’s true that this ought to be a relatively low priority for labs that are busy with good research, but it’s depressing to see that 1) whether something is publishable in high impact journals is such an important criterion for what we do, 2) skeptical science that replicates and refutes is considered a waste of effort, and 3) students are discouraged from carrying out such work, because there is some strange bias that will hurt their chances of employment.

I’m not disagreeing with those arguments, but I’m suggesting that they are symptoms of something rotten in the world of science. Testing claims ought to be what we do. If the journals are going to fill up with positive claims thanks to the file-drawer effect, and if nobody ever wants to evaluate those claims, and if negative results are unpublishable, the literature is going to decline in utility for lack of rigor and evaluation.

Of course, there is one group that has real incentive to get in there and get their hands dirty refining the results: the Wolfe-Simon lab. But her response implies she’s not going to make the effort (although I hope she really is doing something). And this attitude above suggests that, while the positive claim received a lot of media hoopla, any discovery of alternative explanations is going to get ignored. Methinks I see a ratchet at work.

I also notice that Rosie Redfield, brilliant genius that she is, has a relatively simple test of the claims. It’s not my field and I’m not equipped to do any of it, but I don’t see why anyone would find it a waste of effort to assign that project to a first-year grad student, as an exercise in techniques and skill, and as a way to get a quick (I know, low impact!) publication.

And I’m still bewildered that the scientific community would consider tests of a hypothesis a poor investment of its resources. This isn’t like creationism; Wolfe-Simon has a very specific claim that can be evaluated with standard laboratory techniques.