Which web search engines have their own index? Evidence from website visits

by Stephen Hewitt | Published 27 May 2022 | Last updated 26 September 2022

This article presents statistics for web page requests for cambridgeclarion.org where the User Agent string purported to be that of a search engine spider (also called a crawler or bot). Such statistics provide some evidence of which search engines are maintaining an independent index using their own crawler and the level of activity of that crawler.

These statistics are for a two month period ending on 9 September 2021 and do not include requests for image files.

IP addresses were not logged or checked in this exercise, so there is no proof that the declared User Agent string is genuine. However, the absence of a request with an official User Agent string can demonstrate a negative, namely that a crawler from a particular search engine did not make a page request. For example, no User Agent string that could be associated with Brave was seen, and neither was one for Gigablast, during the entire period.

In general these collected statistics put a reliable upper bound on the number of page requests from the official crawler of each search engine, if not a lower one.

Table 1 shows the number of page requests associated with each search engine. A page is defined here to mean an HTML page, a PDF file or a plain text file, so requests for images and anything else embedded in a page were not counted. It also excludes requests for favicon.ico, robots.txt and the non-existent ads.txt.

Table 2 shows the observed User Agent strings (truncated to a maximum of 189 characters) that contributed to the counts in Table 1.

Some User Agents strings seen in requests do not appear in Table 2 because they did not contribute to the counts. This was the case, for example, when only requests for image files and robots.txt were received with a particular User Agent string. One string has been omitted on the grounds that it was likely fake. It does not match any of the Googlebot strings as presented at https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers in May 2022. None of those included “iPhone”.

This string, used in a total of 18 requests, is as follows:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bo

Appendix

Table 3 shows the public web search page for some of these search engines.