Google Scholar illiteracy in the PNAS

A recent paper published in the PNAS, "Expert credibility in climate change" is being used as propaganda to claim that 97% of all climate scientists agree with the IPCC and the need for government action on climate change. An analysis of this paper does not support these conclusions.

Note: All Google Scholar numerical result totals will change over time and sometimes every time you search.

PNAS reviewers and author's William R. L. Anderegg, James W. Prall, Jacob Harold and Stephen H. Schneider are apparently Google Scholar illiterate since searching for just the word "climate" with an author's name will bring results from non-peer-reviewed sources such as books, magazines, newspapers, patents, citations, duplicate listings and all sorts of other erroneous results. Such as 16,000 from the Guardian, 52,000 from Newsweek and 115,000 from the New York Times. There is no "peer-reviewed journal only" search option in Google Scholar.


Cherry Picking:

It is clear the authors cherry picked away skeptics using subjective criteria,

"we imposed a 20 climate-publications minimum to be considered a climate researcher."

So if a scientist published only 19 or less papers on the climate he is not considered an "expert". They did this intentionally as they noted,

"researchers with fewer than 20 climate publications comprise ≈80% the UE group."

Volume of publications does not indicate scientific truth nor does it denote expertise. It cannot be ignored that skeptics extensively publish peer-reviewed papers so they have to use this propaganda to subjectively define "experts". An objective criteria for determining if an author has done climate research would be if an author has or has not published a paper on the climate. Expertise is simply an opinion and who is considered an expert will change based on who you talk to.


Climate Patents:

By default Google Scholar is set to search both "articles and patents" yet no mention of searching only for articles is in the paper.



So why were they searching for patents and how is a patent that contains the search word "climate" a relevant "climate publication"?


Verification:

An attempt to reproduce the results using their methods was unsuccessful,

"we collected the number of climate-relevant publications for all 1,372 researchers from Google Scholar (search terms: "author:fi-lastname climate")"

Using their search terms: "author:fi-lastname climate" I searched Google Scholar for the infamous CRU director Phil Jones,

author:P-Jones climate

Results: 6,580

The first result listed is "Very high resolution interpolated climate surfaces for global land areas" by author Peter G. Jones of the International Center for Tropical Agriculture, Cali, Colombia.

The third result listed is "Organizational climate: A review of theory and research" in Psychological Bulletin by author Allan P. Jones of the Department of Psychology, University of Houston.

The seventh result listed is "Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions" in Organizational Behavior and Human Performance by author Allan P. Jones of the Department of Psychology, University of Houston.

Clearly these were not papers by Phil Jones of the CRU on climate change.

Looking on Prall's list of IPCC AR4 Working Group 1 Authors referenced from their Supplemental Information you see Phil Jones listed with 724 climate publications not the 6,580 that I found using their search method. A link is provided under "GS queries" for Phil Jones labeled "CLIM", clicking on this link brings a surprising revelation, the search term is changed to "author:PD-Jones climate". When their paper explicitly said "author:fi-lastname climate" and no mention is made of including the middle initial. It appears Prall added the middle initial arbitrarily to the authors on the list further undermining the consistency of their results. Using this search term I again searched Google Scholar,

author:PD-Jones climate

Results: 5,370

The sixth result down is "Climate since A. D. 1500", a 1992 book by Phil Jones not a peer-reviewed paper.

Chapter 13 from the same book is found later in the same results as a separate listing, "13 Climatic variations in the longest instrumental records", thus counting the same book twice.

The book's introduction is also found later in the same results "Climate since AD 1500: Introduction", now counting the same book three times but it gets worse,

Citations for this book are counted over 20 times in Google Scholar, further inflating the erroneous results. No mention of turning off citations is in their paper as this feature is on by default in Google Scholar and in the "CLIM" link from Prall's page.

The climate total number of 724 for Phil Jones on Prall's list is unverifiable using the methods outlined in their paper and appears to be made up.

It is clear they used the total number of climate publications because this is explicitly stated in their paper,

"We ranked researchers based on the total number of climate publications authored."

However no verification of these results was done by the authors as they only mention,

"We verified, however, author identity for the four top-cited papers by each author."

It appears they only verified the top four results for their "citation analysis" not for the total amount of results using the search word "climate" for each of the 1372 authors. As demonstrated here, without complete verification it is not possible to draw any meaningful conclusions that include erroneous results.

Conclusion: the study is worthless due to Google Scholar illiteracy and Cherry Picking.


References:
Google Scholar at the Academy (National Post, Canada)