I just read this fascinating story on RealClimate.org about some authors who decided to publish a paper about reproducing a group of contrarian papers that is routinely cited by people and organizations actively trying to discredit climate science, such as The Heartland Institute. The story of the series of rejections by various journals can be found here. Whatever stance you have about climate change, I hope we can all agree that replicating previous studies to learn from mistakes is a valuable exercise.
They begin with their central motivating question:
“Among papers stating a position on anthropogenic global warming (AGW), 97 % endorse AGW. What is happening with the 2% of papers that reject AGW?”
Some argue that the 97% consensus is exaggerated, and I wouldn’t be surprised if it was. But I have no doubt that the majority of actual climate scientists agree on the basic conclusion about the effects of increased CO2.
“The chance of finding errors among the outliers is higher than from more mainstream papers.”
They got a lot of flack about how they chose the contrarian papers. They were very transparent about cherry picking the papers. You could focus on how they didn’t do a random sample of the scientific literature, but I really don’t see how it affects the validity of their results. This seems to be more of a political concern that the reviewers and editors were twisting into a methodological one.
“Our hypothesis was that the chosen contrarian paper was valid, and our approach was to try to falsify this hypothesis by repeating the work with a critical eye.”
The craziest part of this story is when they found out that one of the reviewers that caused them to be rejected early on, was one of the authors of the contrarian papers. This guy didn’t see the need to acknowledge a conflict of interest, which I actually don’t see as an issue because scientists are often asked to review papers that are based on their previous work. To me it is reasonable to have those previous authors review the article since they are obviously experts on the subject material. However, in this case the reviewers did not reject the paper on scientific grounds. Here’s how Rasmus Benestad explains this in his blog post:
“…I learned that the critical reviewer swaying the decision at ‘Climate Research’ or ‘Climatic Change’ had been Ross McKitrick (link). He was an author of one of the papers we included in our selection of contrarian papers that we had replicated.
McKitrick had apparently not declared any conflict of interest, but had taken on the role as a gatekeeper, after having accused climate scientists to do so himself…”
I haven’t really dug into the results yet, but I was really moved by the sentiment of the paper, as well as the idea of actively learning from mistakes in such a public way. I think contrarians would have to at least agree with the authors’ attitude about the role of a scientist.
“A scientists job is to be to-the-point and clear as possible; not cosy up to colleagues. So, is is really “inflammatory” to point out the weakness in other analyses?”
I also like that this paper is filling a hole for non-climate scientist, and giving them a better leg to stand on in the climate debate. I hope there are more papers like this coming in the future.
“The motivation for the original plea for a formal rebuttal paper was that educators should be able to point to the peer-reviewed literature ‘to fight skepticism about the fundamentals on climate change’.”