Google Scholar Illiteracy at Skeptical Science

Rebuttal to "Meet the Denominator"


* Google Scholar's interface has changed since 2011

In a desperate attempt to diminish the value of the list of peer-reviewed papers supporting skeptic's arguments, former bike messenger and man-purse maker Rob "Scumbags" Honeycutt from Skeptical Science not only lies but puts on a surprising display of his Google Scholar Illiteracy. He fails to use quotes when searching for phrases, is unable to count past 1000 and fails to remove erroneous results such as, the script for "Batman Returns" - believing it to be a peer-reviewed paper about global warming. It is clear that not only does he not understand how to properly use Google Scholar, he has no idea of the relevance of any of the results he gets.

Update: Rob was forced to concede I was correct (though never owns up to blatantly lying) and has desperately made a flawed updated "analysis". His original inaccurate number of 954,000 results went down to 189,553 results (which he fails to mention in his update) of which 160,130 (84%) CANNOT BE VERIFIED due to the 1000 result limit imposed by Google Scholar. The remaining results are irrefutably filled with erroneous nonsense such as, "Santa Claus, Last of the Wild Men: The Origins and Evolution of Saint Nicholas" that has to be individually removed before any sort of accurate count can be taken (see the updates for more information). None of which was done leaving his post to be worthless and those who cite it computer illiterate.

Update 2: Skeptical Science has censored (deleted) ALL of my hundreds of comments at their site.

1. Rob begins with a strawman argument that the list is only presenting one side of the argument. This has never been denied as one of the reasons the list was created was to show the skeptical side of the argument supported by the peer-reviewed literature. This purpose is explicitly stated on the list,

Purpose: To provide a resource for peer-reviewed papers that support skepticism of AGW or the negative environmental or socio-economic effects of AGW and to prove that these papers exist contrary to widely held beliefs.


2. Rob mentions previous criticisms of the list that have been completely refuted,

Rebuttal to "450 more lies from the climate change Deniers"


3. Rob produces erroneous results using a search query without quotes,

He does an advanced Google Scholar search query for the search words,

climate change

In his blog post he incorrectly lists them as a search phrase using quotes, "climate change". There is no mention that no quotes were used. As you will see this dramatically affects the results.

He then adds two advanced filters,

* Biology, Life Sciences, and Environmental Science
* Physics, Astronomy, and Planetary Science

The results,

climate change (no quotes + filters) - 956,000

When quotes are used you get a dramatically smaller result,

"climate change" (filters) - 635,000

It is clear Rob is only interested in results for the search phrase "climate change" yet by not using quotes he included erroneous results that simply included both words in any context, including having nothing to do with "climate change". Such as,

Managing the Process of Engineering Change Orders: The Case of the Climate Control System in Automobile Development
(Journal of Product Innovation Management, Volume 16, Issue 2, pp. 160–172, March 1999)
- Christian Terwiesch, Christoph H. Loch

Why is Rob counting results about climate control systems in automobiles?


4. Rob arbitrarily pulls out 10% of the results to remove citations failing to understand that these can be automatically filtered using Google Scholar,

What are the results marked [citation] and why can't I click on them? (Google Scholar Help)

"To exclude them [citations] from your search results, select at least summaries from the dropdown menu labeled include citations."

The results without citations,

climate change (no quotes + no citations + filters) - 901,000

"climate change" (no citations + filters) - 600,000

This just further demonstrates his inability to properly use Google Scholar.


5. Rob fails to understand that Google Scholar does not have a peer-reviewed only filter and thus he is counting erroneous results,

What do you include in Google Scholar? (Google Scholar Help)

"Google Scholar includes journal and conference papers, theses and dissertations, academic books, pre-prints, abstracts, technical reports and other scholarly literature from all broad areas of research. ...Shorter articles, such as book reviews, news sections, editorials, announcements and letters, may ...be included."

These non-peer-reviewed sources are easily proven to exist in Google Scholar search results; such as 21,000 from the Guardian, 86,000 from Newsweek and 144,000 from the New York Times.

Thus the only way for Rob to verify his numerical result total from his Google Scholar search is by checking that every single search result is a peer-reviewed paper from a peer-reviewed journal. He clearly did not do this.

Note: It is important to remember that every single result needs to be checked that it is a peer-reviewed paper and not simply in a peer-reviewed journal. As Google Scholar makes no distinction between non-peer-reviewed content that appears in peer-reviewed journals from peer-reviewed content.


6. It is impossible for Google Scholar to be used to verify more than 1000 results for any search query because it is hard limited to 1000 verifiable results,

Can I see more than 1,000 search results? (Google Scholar Help)

"Sorry, we can only show up to 1,000 results for any particular search query. Try a different query to get more results."

Thus it is impossible to verify Rob's claims as searched, making his conclusions meaningless.


7. Rob gets caught lying about how many papers he "perused",

"I did a pretty thorough perusal of the 200 pages of articles and it looks like they are all actual papers and not just references to any blogs or websites." - Rob Honeycutt, Skeptical Science

This is a lie as it is impossible to go past page 100 (1000 results) for any search query using Google Scholar.


8. A Google Scholar search result that includes the search phrase "climate change" does not mean it explicitly endorses "Anthropogenic Global Warming" theory. Explicit endorsement would require the inclusion of that phrase. How many results Google Scholar shows using this search phrase can easily be determined,

"anthropogenic global warming" (no citations + filters) = 662

However inclusion of this phrase can also mean criticism of the theory. The context of the phrase can only be determined by reading each and every result. Implicit endorsement would require reading each and every result for alternate search queries.


Conclusion:

No meaningful conclusion can be drawn regarding the number of peer-reviewed papers supporting AGW theory using numerical result totals from Google Scholar searches due to the inclusion of erroneous results. Thus no meaningful comparison of these results can be made to the list of Peer-Reviewed Papers.


Note: All numerical result totals from Google Scholar searches can change at anytime.


Update: After repeatedly pressed on how he "perused" 200 pages Rob eventually decided to change it to say, "I did a pretty thorough perusal of 200 articles of the 100 pages of results and it looks like they are all actual papers and not just references to any blogs or websites.". Unfortunately I do not believe this was an honest mistake as he did not immediately admit to the error instead posting six other comments avoiding it. This is not something anyone else would do if they made a legitimate typo. I believe he originally just made up the number so it appears he made a token effort at validation. The word "pages" and "papers" is not easily confused, let alone "pages" and "articles".

Update 2: Comments on the image (I can't make this stuff up),

"I would even go so far as to say by using that image in the context of Rob's name suggests it was a threat of violence by PT against Rob." - Albatross, Skeptical Science

"PT do you realize that if you leave that picture posted in it's current context that it constitutes an act of terrorism." - Ron Crouch, Skeptical Science

Update 3: The moderators at Skeptical Science have conveniently been deleting my comments wholesale instead of editing out whatever part they "claim" violates their policy. This hypocritical policy allows adherents to the site to make personally attacks, state false allegations and make other libelous claims against those they disagree with. Any attempt by the recipient of these attacks to defend themselves is usually deleted. This has created many false criticisms that it appears I did not address.

Update 4: Rob did an updated analysis, searching between 1971-2011 using the search phrase "climate change" and separately "global warming" per individual year, excluded citations and as he claims "checked for various other erroneous results".

Rob has admitted, "Poptech has brought up several valid points" and his new results as he calculated have went down from 954,000 to 189,553. A reduction of 764,447 results.

Unfortunately it is not possible to validate all of his results for the search phrase "climate change" between 1990-2011 and the search phrase "global warming" between 2002-2010 due to the 1000 search result limit imposed by Google Scholar. That makes it impossible to validate 160,130 (84%) of his results to accurately remove erroneous ones. Without accurately removing erroneous results no meaningful conclusions can be based on these numbers.

Rob's new analysis suffers contamination by very erroneous results such as,

Adam, Eve, and the genome: the Human Genome Project and theology [Book] (S.B. Thistlethwaite, 2003)

"This important book combines a basic primer on genetic research with ethical reflection by an interdisciplinary team on key questions and a deeper look, in light of such research, at what it means to be human."

Tom of Twofold Bay [Book] (V. Kattooparambil, 2007)

"This is the story of Tom, Killer whale of Twofold Bay, Eden, friend to George Davidson (bay whaler) and others like him: it is based upon a true story that will live in the memory of the citizens of Eden for eternity, a museum being built in Tom’s honour upon his death."

Around the States in 90 Days [Book] (A. Moseley, 2009)

"Determined to escape a 9 to 5 life but equally determined not to do anything socially worthwhile, Andy Moseley took the only option available to him and packed his bags and left for America. His plan was to see as much of the country as he could in 90 days. Starting in Washington and ending in San Francisco, he passed through Canada and along Route 66 taking in several places not on any logical route across the country, and eventually covering half of the States of America, and a few bits of Canada too."

Disintegration: The Splintering of Black America [Book] (E. Robinson, 2010)

"A Pulitzer Prize-winning columnist and commentator explains how years of desegregation and affirmative action have led to the revelation of four distinct African American groups who reflect unique political views and circumstances, in a report that also illuminates crucial modern debates on race and class."

Landfall along the Chesapeake: in the wake of Captain John Smith [Book] (S. Schmidt, 2006)

"In 2002, Susan Schmidt retraced John Smith's 1608 voyage on the Chesapeake Bay. In Landfall along the Chesapeake, a cruising guide for Chesapeake boaters and a field log for naturalists, Schmidt compares the beauty of ancestral legacy and childhood memory to her observations on a 100-day voyage in a 22-foot boat."

Extreme Cuisine: Exotic Tastes from Around the World [Book] (Lonely Planet, 2009)

"Imagine tucking into grasshoppers as you wander the Mercado Benito market in Oaxaca, Mexico, or chowing down on juicy witchetty grubs on your travels through Central Australia - such meals can be the perfect entree to a culture. In this book you'll find over 50 delicacies that creep, crawl, sizzle and spit, where they originated from and where you can experience them. You may not salivate over blood, scorpions, chicken's knees or partially digested coffee beans, but travel long enough and you're bound to meet someone who does. Extreme Cuisine is sure to challenge your idea of what makes good eating."

The Last Stand of Chuck Norris: 400 All New Facts About the Most Terrifying Man in the Universe [Book] (Ian Spector, 2011)

"Just when you thought it was safe to read, Chuck Norris is back with another roundhouse kick to the face. Fans of this bestselling series will rejoice at this newest addition to the hilarious anthology. The Last Stand of Chuck Norris contains 400 all-new, kick-ass facts about the book's indomitable namesake, including:

- Chuck Norris sleeps with a pillow under his gun.
- When Chuck Norris crosses the street, the cars have to look both ways.
- Chuck Norris doesn't do push-ups. He pushes the world down.
- Chuck Norris doesn't need a Twitter account. He is already following you.
- Chuck Norris likes his ice like he likes his skulls: crushed."

Planet Mutonia and the Young Pop Star Wannabes [Book] (T. Kallison, 2010)

"Can you imagine a world without music? That's exactly what happened to young Melody Bell and her two friends, Harmon E. Cord and Justin Tyme. These three youngsters share their common aspirations to one day become celebrated pop stars!"

Are these the high impact journals skeptics cannot get published in?

It would be quite interesting to have Rob explain how these fit into climate science research but I do not find these types of erroneous results surprising as I am well aware of the limitations of Google Scholar for the type of analysis Rob is attempting.

Rob claims to have done a statistical sampling of 200 papers and found 6% were erroneous (his sampling methods are not disclosed) but due to the ranking of Google Scholar results a true random sampling would not be possible with results over 1000. This is because Google Scholar uses a ranking system that heavily weighs citations (among other factors) that would place actual peer-reviewed content towards the beginning of the results and erroneous content towards the end.

How are documents ranked? (About Google Scholar)

"Google Scholar aims to rank documents the way researchers do, weighing the full text of each document, where it was published, who it was written by, as well as how often and how recently it has been cited in other scholarly literature."

Sampling fails here with results over 1000 since more erroneous results would likely fall outside of the sample range, producing misleading statistics.

I want to reiterate that peer-reviewed papers that include the search phrase "climate change" does not mean explicit endorsement of "anthropogenic global warming".

It is thus not possible to validate Rob's new number of 189,553 and no meaningful conclusions can be drawn from his analysis, especially in relation to support for "anthropogenic global warming".

Note: To give a further example of why numerical result totals for Google Scholar search queries are unreliable; I repeated Rob's search for the phrase "Global Warming" (filters + no citations) for the 2010 year. Rob got 25,600 results, while I found only 22,900. This is a difference of 2,700 results by simply searching on different days.


Update 5: The Skeptical Science moderator "muoncounter" demonstrated his Google Scholar illiteracy by censoring my replies for what he falsely claimed, "You got what you searched for; the title of the book is in your search field" in regards to my demonstration of how Rob's new analysis suffers contamination by very erroneous results,

Disintegration: The Splintering of Black America [Book] (E. Robinson, 2010)

It is very clear "muoncounter" doesn't understand how Google Scholar works any better than Rob. Google Scholar search queries are cumulative, which means search results must include all search words and phrases in a search query. This is very easy to demonstrate and irrefutable. When I changed the phrase "climate change" to "global warming" the result disappears. If they were not cumulative then the result would still show up.

Note: I tried multiple times to post this to their site (each was deleted). Thus they left me no choice but to further embarrass them.