One of the tenets of science is that the results be reproducible. One consequence of this maxim is that any paper that is published should have sufficient information that would enable anyone who wishes to do so to replicate the results. But there is no real incentive for people to try and replicate the work of others. It takes a lot of time and effort and one cannot publish a confirmation of someone else’s result unless the original result was so revolutionary that supportive evidence is called for. The cold fusion and the faster-than-light neutrino stories were examples of such high-profile cases.
As a result the only scientists who bother to replicate the work of others are those who either disagree with the result and think that something went awry or seek to build on that work and thus want to make sure they have a solid foundation to work on. But it turns out that even if you get results that contradict previously published results, it is not easy to get them into print because the journals that accepted the original results are often reluctant to publish refutations.
This article describes the frustrations of a researcher who got results that contradicted the existence of precognition, even though the original claim to have found evidence of precognition was pretty sensational and got a lot of publicity in the science media. The journal that published the original paper refused to publish the refutation, as did many other leading journals. As the author says:
Although we are always being told that “replication is the cornerstone of science”, the truth is that the “top” journals are simply not interested in straight replications – especially failed replications. They only want to report findings that are new and positive.
Most scientists are aware of this bias and will rarely bother with straight replications. But straight replication attempts are often exactly what is required, especially when dealing with controversial claims.
This is a serious problem since claims that are not refuted carry the presumption of truth. I am not sure what can be done about this except perhaps the creation of an online journal devoted exclusively to publications that refute earlier work or, in the case of controversial claims that merit them, confirmations.
I understand the problem of bias towards positive results (my PI once recounted his mentor jokingly suggesting he could always get his results published in the “Journal of Negative Results”, but it turns out there really are such journals.) but then, isn’t the problem also one of the peer-review process? Seems like insufficient replication should be a big red alert for anyone wishing to publish in a respected journal, but do we think some editors are so desirous of intriguing results that they will intentionally overlook such clear problems? Or is this more a simple matter of cognitive bias not seeing the trees through the forest?
I’ve also been highly critical when articles are published that use incorrect statistical procedures to analyze data, but I think in many cases, that’s the result of the authors and/or reviewers not having sufficiently strong statistical backgrounds.
I totally agree with you, and as a scientist, this is what irks me about the American scientific process more than anything else.
We are actively dissuaded from working on the same problems as others. In fact, you CAN’T get funding by doing the same experiments--you have to always demonstrate that what you are doing is *novel*.
The result is that we get a ton of once-performed studies with a frazzling of results, most of which are probably false. It makes scientific consensus much harder to reach.
Thanks for telling me about these journals! I was not aware of them. For once one of my ideas for improvement turned out to be not crazy.
This is because science, in this case, and the public at large, in general, have allowed publishers and other middlemen to become gatekeepers rather than enablers. And at a time when we really don’t need such enablers, and they really do little enabling anyway, they seek to constantly reinforce their gatekeeper position, frequently via bad legislation.
Hey Mano, it seems like you have yet again opened up an interesting topic. What I am completely surprised about however, is that no one has jumped in to fray to bring up the topic of Global Warming or Climate Change (or whatever they are calling it this week) I still believe in the old fashioned, (out dated?) concept that it is a foundational tenet of science. That is to be able to replicate findings before everyone decides that the first study is the absolute. I thought we all covered that in the fifth grade! However, as is the case of global warming, we are told straight up that the issue is settled, while ignoring all the conflicting data already published. Really? How many times has “settled science” been turned on it’s head? The truly troubling aspect of modern science is discovering that many of today’s scientists seem to be looking out for their paychecks first and in satisfying the wishes (wink, wink) of those who are funding their departments. Can you imagine the same standard being applied to other fields of study such as quantum physics or oncology research?
While your analysis is completely correct this actually is not:
“One of the tenets of science is that the results be reproducible.”
No single historical branch of science provides reproducible data. That’s one reason it’s so hard to understand what the Big Bang was about and why detailed reports have to be written about fossils and archeological findings. I would have thought that Jack Bosch had learned this in the 6th grade.
” the journals that accepted the original results are often reluctant to publish refutations.”
This is not just a problem, this is potentially disastrous. So I was glad to read Zugswang’s reaction.
While the historical events themselves are not reproducible, the research results that shed light on them should be. For example, when people publish results on the ages of fossils, others researchers, working with those same or similar fossils, should be able to reproduce the same ages. With global warming different research teams have analyzed various aspects of the data such as average global temperatures over time and it is that convergence in their findings that lends support to the idea that the Earth is warming.
Yeah, I read about that particular case a few days ago. I find that somewhat shocking. I totally understand that researchers are not exactly eager to publish their non-replications, and understand that journals are not exactly eager to accept such boring results, and this has been and will remain a difficult problem to overcome. But it seems to me that a journal has an ethical imperative to publish at least the first one or two non-replications they receive of a result that they themselves published, as long as there are no serious methodological problems. To do otherwise seems deeply irresponsible.
It’s one thing if journal X refuses to publish a non-replication of a result that appears in journal Y. But if the result appeared in your journal, it seems to me you can’t refuse a non-replication of that result just because non-replications are boring…!!!
Given that something near 100% of people would like global warming to go away, one would suspect that such a pressure would be for the opposing effect and not the way you would have it. Since understanding climate is so important, climatologists would still have jobs even if this was the case.