Who's actually reading us
Parsed from Apache's access log: every search engine and AI crawler that has fetched a page in the last 30 days. The shape of this list is the entire SEO + GEO answer for an indie directory: are Googlebot, Bingbot, and the AI engines (ChatGPT, Perplexity, Claude) showing up? Bing-derived AI search surfaces fresh content fastest, which is why IndexNow and llms.txt are wired in. The chart below is the receipt.
Top 10 crawlers, last 30 days
-
Every crawler, by type
Bars show 30-day hit count. Groups: search index (Google/Bing), AI search (cited live), AI training (scraped for models), live user fetch (a real person's prompt). "Newly seen" bots are auto-classified by Haiku from the user-agent; IPs not verified.
Top 25 most-crawled pages, last 30 days
Counts all search engine and AI bot hits. Only page paths with at least one hit are listed; query strings stripped.
Search engines, share of crawl
AI & LLM crawlers, share of crawl
Bot farms, exploit-scanner probes by country
-
Snapshot: -
API running costs
Today- -
All time- -