Daily snapshot 2026-10-24

2026-10-24 UTC

The numbers come from one website. They describe what these crawlers did here, not across the web.

Period: 2026-10-24T00:00:00Z to 2026-10-25T00:00:00Z (00:00Z to 24:00Z). Generated 2026-10-25T03:18:30.602Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method

Bars show requests on 2026-10-24. The black tick on each row marks that crawler's 28-day median.

Facts

F-261024-01 Bingbot first fetched the 2026-10-23 snapshot 3 h 54 min after it was published. Source
F-261024-02 Googlebot first fetched the 2026-10-23 snapshot 7 h 3 min after it was published. Source
F-261024-03 GoogleOther first fetched the 2026-10-23 snapshot 9 h 44 min after it was published. Source
F-261024-04 An MCP client named mcp-observatory first called list_changes on 2026-10-24. Source
F-261024-05 Every request on 2026-10-24 from ClaudeBot, Bingbot, Googlebot, OAI-SearchBot and GPTBot was verified against the operator's published method. Source

This selection has expired; it is listed in snapshot.json.

Crawlers

Crawler	Requests	Verified share	Distinct paths	28-day median	Sweep day	First path	robots.txt	sitemap
ClaudeBot	55	100%	5	48	no	/robots.txt	12	42
Bingbot	16	100%	15	9	no	/compare/gptbot-vs-perplexitybot	1	1
Googlebot	16	100%	10	12	no	/robots.txt	7	0
OAI-SearchBot	5	100%	3	5	no	/robots.txt	3	0
GPTBot	3	100%	3	3	no	/sitemap-index.xml	0	3

MCP calls

Client	Tool	Calls
mcp-observatory	list_changes	2
BrickBlueBot	identify_user_agent	1
BrickBlueBot	list_changes	1
BrickBlueBot	lookup_crawler	1
rokmcp-probe	list_changes	1
verifymcp-probe	__verifymcp_auth_probe_44998c83398c3dd1__	1

Snapshot pickup

For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.

Snapshot	Crawler	Minutes after publication	Tag
2026-10-23	Bingbot	234	none
2026-10-23	Googlebot	423	none
2026-10-23	GoogleOther	584	none
2026-10-22	ClaudeBot	214	none
2026-10-22	Googlebot	225	none
2026-10-22	GoogleOther	225	h
2026-10-22	Bingbot	235	none

Open questions

Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)

If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.

Ask for something this snapshot lacks

What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261024.

To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261024 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261024. The optional author_claim field says who is writing: a person, a model or an agent.

Links

Next snapshot: 2026-10-25