All "97% Consensus" Studies Refuted by Peer-Review

After showing how 97 articles thoroughly refuted the most prominent "consensus" study, Cook et al. (2013), consensus proponents inevitably moved the goal posts and fell back on other "97% consensus" studies: Doran & Zimmerman (2009), Anderegg et al. (2010) and Oreskes (2004) (which is really a 100% consensus study). However, these have all been thoroughly refuted in the scholarly literature and the following are the refutations of them.

Update: Since this article was first published additional studies have been used to perpetuate the long-debunked "97% consensus" talking point, such as Verheggen et al. (2014). While others like Stenhouse et al. (2014) actually demonstrated a marginal 52% consensus.

---

[1] Oreskes (2004) - "The Scientific Consensus on Climate Change"

The Letter Science Magazine Rejected (PDF) [Archive]
(Energy & Environment, Volume 16, Numbers 3-4, pp. 685-688, July 2005)
- Benny Peiser

Abstract: On 3 December 2004, Science published an article entitled "The scientific consensus on climate change" by Naomi Oreskes (Vol 306, Issue 5702, 1686). Oreskes claims to have analysed 928 abstracts she found listed on the ISI database using the keywords "global climate change". The article suggested that for the first time, empirical evidence was presented that appeared to show a unanimous, scientific consensus on the anthropogenic causes of recent global warming. Between 3 December 2004 and 4 January 2005 I conducted a similar analysis. The results of my findings contradicted Oreskes and essentially falsified her study. On 4 January 2005, I submitted these results in a letter to Science. On 18 February, editors from Science contacted me to suggest that they would consider publishing a shorter version of the letter. This shorter version was submitted on 23 February. On 13 April, Science responded, saying "After realizing that the basic points of your letter have already been widely dispersed over the internet, we have reluctantly decided that we cannot publish your letter." No evidence was provided for this technically contrived excuse. As far as I am aware, neither the details nor the results of my analysis were cited anywhere. Journals such as Science have an obligation to correct errors, especially as activists, journalists and science organisations have endlessly repeated claims made in Oreskes (2004). The sad reality is that by refusing to publish corrections to a fatally flawed paper, they undermine their own credibility, that of their contributors, and the integrity of science.


Scientific Consensus on Climate Change? (PDF) [Archive]
(Energy & Environment, Volume 19, Number 2, pp. 281-286, March 2008)
- Klaus-Martin Schulte

Abstract: Fear of anthropogenic "global warming" can adversely affect patients' well-being. Accordingly, the state of the scientific consensus about climate change was studied by a review of the 539 papers on "global climate change" found on the Web of Science database from January 2004 to mid-February 2007, updating research by Oreskes, who had reported that between 1993 and 2003 none of 928 scientific papers on "global climate change" had rejected the consensus that more than half of the warming of the past 50 years was likely to have been anthropogenic. In the present review, 31 papers (6% of the sample) explicitly or implicitly reject the consensus. Though Oreskes said that 75% of the papers in her former sample endorsed the consensus, fewer than half now endorse it. Only 7% do so explicitly. Only one paper refers to "catastrophic" climate change, but without offering evidence. There appears to be little evidence in the learned journals to justify the climate-change alarm that now harms patients.

---

[2] Doran & Zimmerman (2009) - "Examining the Scientific Consensus on Climate Change"

Comment on "Examining the Scientific Consensus on Climate Change" (PDF)
(Eos, Transactions American Geophysical Union, Volume 90, Number 27, July 2009)
- Roland Granqvist

Abstract: In a summary of their survey on the opinion about global warming among Earth scientists (see Eos, 90(3), 20 January 2009), Peter Doran and Maggie Kendall Zimmerman conclude that the debate on the role of human activity is largely nonexistent, and that the challenge is "how to effectively communicate this fact to policy makers" and to the public. However, I argue that neither of these conclusions can be drawn from the survey. For example, one issue that is much discussed in the public debate is the role of greenhouse gas emissions in global warming. Perhaps there is not much debate about this issue among scientists, but this cannot be concluded from the survey, in which nothing is said about such emissions. In the second question of their survey, Doran and Kendall Zimmerman refer only to "human activity."


Further Comment on "Examining the Scientific Consensus on Climate Change" (PDF)
(Eos, Transactions American Geophysical Union, Volume 90, Number 27, July 2009)
- John Helsdon

Abstract: The feature article "Examining the scientific consensus on climate change," by Peter Doran and Maggie Kendall Zimmerman (see Eos, 90(3), 20 January 2009), while interesting, has a primary flaw that calls their interpretation into question. In their opening sentence, the authors state that on the basis of polling data, "47% [of Americans] think climate scientists agree… that human activities are a major cause of that [global] warming…." They then described the two-question survey they had posed to a large group of Earth scientists and scientifically literate (I presume) people in related fields. While the polled group is important, in any poll the questions are critical. My point revolves around their question 2, to wit, "Do you think human activity is a significant contributing factor in changing mean global temperatures?" Note that the opening sentence of their article uses the phrase "major cause" in reporting the results of the polling, while the poll itself used the phrase "significant contributing factor." There is a large difference between these two phrases.

---

[3] Anderegg et al. (2010) - "Expert credibility in climate change"

Climate denier, skeptic, or contrarian? (PDF) [Archive]
(Proceedings of the National Academy of Sciences, Volume 107, Number 39, September 2010)
- Saffron J. O'Neilla, Max Boykoff

Abstract: Assigning credibility or expertise is a fraught issue, particularly in a wicked phenomenon like climate change—as Anderegg et al. (1) discussed in a recent issue of PNAS. However, their analysis of expert credibility into two distinct "convinced" and "unconvinced" camps and the lack of nuance in defining the terms "climate deniers," "skeptics," and "contrarians" both oversimplify and increase polarization within the climate debate.


Expert credibility and truth (PDF) [Archive]
(Proceedings of the National Academy of Sciences, Volume 107, Number 47, November 2010)
- Jarle Aarstad

Abstract: Anderegg et al. (1) state that 97–98% of climate researchers most actively publishing in the field "support the tenets of [anthropogenic climate change] ACC … the relative climate expertise and scientific prominence of the researchers unconvinced of ACC are substantially below that of convinced researchers" (1). The contribution illustrates the predominating paradigm in climate research today. However, whereas expert credibility and prominence may dominate the opinion of what is true, it can never alter truth itself.


Regarding Anderegg et al. and climate change credibility (PDF) [Archive]
(Proceedings of the National Academy of Sciences, Volume 107, Number 52, December 2010)
- Lawrence Bodenstein

Abstract: The study by Anderegg et al. (1) employed suspect methodology that treated publication metrics as a surrogate for expertise. Credentialed scientists, having devoted much of their careers to a certain area, with multiple relevant peer-reviewed publications, should be deemed core experts, notwithstanding that others are more or less prolific in print or that their views stand in the minority. In the climate change (CC) controversy, a priori, one expects that the much larger and more "politically correct" side would excel in certain publication metrics. They continue to cite each other's work in an upward spiral of self-affirmation. The authors' treatment of these deficiencies in Materials and Methods was unconvincing in the skewed and politically charged environment of the CC hubbub and where one group is in the vast majority (1). The data hoarding and publication blockade imbroglio was not addressed at all. The authors' framing of expertise was especially problematic. In a casting pregnant with self-fulfillment, the authors defined number of publications as expertise (italics). The italics were then dropped. Morphing the data of metrics into the conclusion of expertise (not italicized) was best supported by explicit argument in the Discussion section rather than by subtle wordplay. The same applied to prominence, although here the authors’ construct was more aligned with common usage, and of course, prominence does not connote knowledge and correctness in the same way as expertise.

---

[4] Cook et al. (2013) - "Quantifying the consensus on anthropogenic global warming"

Climate Consensus and 'Misinformation': A Rejoinder to Agnotology, Scientific Consensus, and the Teaching and Learning of Climate Change (PDF) [Archive]
(Science & Education, pp. 1-20, August 2013)
- David R. Legates, Willie Soon, William M. Briggs, Christopher Monckton of Brenchley

Abstract: Agnotology is the study of how ignorance arises via circulation of misinformation calculated to mislead. Legates et al. (Sci Educ 22:2007–2017, 2013) had questioned the applicability of agnotology to politically-charged debates. In their reply, Bedford and Cook (Sci Educ 22:2019–2030, 2013), seeking to apply agnotology to climate science, asserted that fossil-fuel interests had promoted doubt about a climate consensus. Their definition of climate 'misinformation' was contingent upon the post-modernist assumptions that scientific truth is discernible by measuring a consensus among experts, and that a near unanimous consensus exists. However, inspection of a claim by Cook et al. (Environ Res Lett 8:024024, 2013) of 97.1 % consensus, heavily relied upon by Bedford and Cook, shows just 0.3 % endorsement of the standard definition of consensus: that most warming since 1950 is anthropogenic. Agnotology, then, is a two-edged sword since either side in a debate may claim that general ignorance arises from misinformation allegedly circulated by the other. Significant questions about anthropogenic influences on climate remain. Therefore, Legates et al. appropriately asserted that partisan presentations of controversies stifle debate and have no place in education.


Quantifying the consensus on anthropogenic global warming in the literature: A re-analysis (PDF) [Archive]
(Energy Policy, Volume 73, pp. 701-705, October 2014)
- Richard S. J. Tol

Abstract: A claim has been that 97% of the scientific literature endorses anthropogenic climate change (Cook et al., 2013. Environ. Res. Lett. 8, 024024). This claim, frequently repeated in debates about climate policy, does not stand. A trend in composition is mistaken for a trend in endorsement. Reported results are inconsistent and biased. The sample is not representative and contains many irrelevant papers. Overall, data quality is low. Cook's validation test shows that the data are invalid. Data disclosure is incomplete so that key results cannot be reproduced or tested.


Quantifying the consensus on anthropogenic global warming in the literature: Rejoinder (PDF) [Archive]
(Energy Policy, Volume 73, pp. 709, October 2014)
- Richard S. J. Tol

Abstract: In my critique of Cook et al. (2013), I raised a number of issues (Tol, 2014). Cook et al. (2014) respond to a few only. They do not dispute

(1) that their sample is not representative,
(2) that data quality is low,
(3) that their validation test is not passed,
(4) that they mistake a trend in composition for a trend in endorsement,
(5) that the majority of the investigated papers that take a position on (anthropogenic) climate change in fact do not examine any evidence, and
(6) that there are inexplicable patterns in the data.


Comment on 'Quantifying the consensus on anthropogenic global warming in the scientific literature' (PDF) [Archive]
(Environmental Research Letters, Volume 10, Number 3, March 2015)
- Benjamin John Floyd Dean

Abstract: I read the study by Cook et al with great interest [1]. The study used levels of endorsement of global warming as outlined in their table 2; however, I could see no mention as to how these levels were created and how reliable they were in terms of both inter-rater and intra-rater reliability (Cohen's kappa). Best practice on rater reliability indicates that both inter-rater and intra-rater should have been measured and documented in a study such as Dr Cook's [2] and I am surprised that this fact appears to have been neglected. It would be of considerable benefit to readers for some robust rate reliability metrics to be included, if at all possible.


Comment on 'Quantifying the consensus on anthropogenic global warming in the scientific literature' (PDF) [Archive]
(Environmental Research Letters, Volume 11, Number 4, April 2016)
- Richard S. J. Tol

Abstract: Cook et al's highly influential consensus study (2013 Environ. Res. Lett. 8 024024) finds different results than previous studies in the consensus literature. It omits tests for systematic differences between raters. Many abstracts are unaccounted for. The paper does not discuss the procedures used to ensure independence between the raters, to ensure that raters did not use additional information, and to ensure that later ratings were not influenced by earlier results. Clarifying these issues would further strengthen the paper, and establish it as our best estimate of the consensus.

---

[5] Verheggen et al. (2014) - "Scientists' Views about Attribution of Global Warming"

Comment on "Scientists' Views about Attribution of Global Warming" (PDF) [Archive]
(Environmental Science & Technology, Volume 48, Issue 16, pp. 8963–8971, November 2014)
- José L. Duarte

Abstract: Verheggen et al. report a survey of scientists' views on climate change. However, they surveyed a large number of psychologists, pollsters, philosophers, etc. The number of nonclimate scientists who responded is undisclosed, and is likely unknowable given the design. Thus, the valid results of the study are unknown, and it should be withdrawn. Moreover, the core method of including mitigation and impacts researchers creates a structural inflationary bias, ultimately conflating career choice with consensus. Finally, an estimate of the consensus is unlikely to be reliable without accounting for the extraordinary personal cost of dissent, especially when an issue is moralized.

---

Rebuttals to Criticisms:

Criticism: "Some papers on the list are hidden behind a paywall."

Rebuttal: Whether a full copy of a paper is made freely available is at the discretion of the journal's publisher. Any similar list would have the same limitations since archiving a paper without a publisher's permission would violate copyright law. Where a full copy of a paper was found online, a (PDF) link was added after a paper's name.

---

Criticism: "Some papers on the list are not scholarly peer-reviewed (refereed)."

Rebuttal: This is a strawman argument, as it is not claimed that these papers were scholarly peer-reviewed (refereed) prior to publication but rather that they are a form of "peer-review" that has been published in a scholarly journal. Peer-review can be defined as an "evaluation of scientific, academic, or professional work by others working in the same field". In a scholarly journal like the PNAS, the document format used to "constructively address a difference of opinion with authors of a recent PNAS article. ...or point out potential flaws in studies published in the journal" are called "Letters".

---

Criticism: "Tol (2014) was rejected by other journals for being flawed."

Rebuttal: Dr. Tol's 2014 paper was actually censored by Environmental Research Letters (ERL) due to having multiple outspoken alarmist scientists on its editorial board (e.g. Peter Gleick and Stefan Rahmstorf) and politely declined by two other journals for being "out of scope" (off topic) not flawed.

---

Criticism: "Tol (2014) has 24 errors."

Rebuttal: Dr. Tol refuted all of these claims in a post online and in his published rejoinder.

---

Criticism: "Dr. Tol confirmed the 97% consensus."

Rebuttal: This misleading claim originated with one of the co-authors of Cook et al. (2013) - Dana Nuccitelli. Dr. Tol refuted this claim in a scathing editorial he wrote for The Guardian:

"Dana Nuccitelli writes that I “accidentally confirm the results of last year’s 97% global warming consensus study”. Nothing could be further from the truth. I show that the 97% consensus claim does not stand up. [...]

In his defence, Nuccitelli argues that I do not dispute their main result. Nuccitelli fundamentally misunderstands research. Science is not a set of results. Science is a method. If the method is wrong, the results are worthless." - Dr. Richard Tol

---

Criticism: "Dr. Tol claims the consensus is actually 91%."

Rebuttal: This misleading claim originated with one of the co-authors of Cook et al. (2013) - Dana Nuccitelli and falsely spread by websites like Politifact. Dr. Tol refuted this claim in an email to Politifact.

"I never claimed that the consensus rate is 91%. [...]

Do check the grammar: "would […] in that case" does in no way indicate my agreement with the number. In fact, I make it very clear that any number based on Cook’s data is unreliable." - Dr. Richard Tol

---