Where the data comes from

NowServingTO pulls from a handful of paid + free APIs. This page tracks our own usage of each - what we've called and what it's cost - so you can see exactly where the budget goes.

By provider

Who's crawling

Search engines and AI assistants that have fetched pages in the last 30 days, parsed straight from the Apache access log. Useful sanity check that the sitemap, IndexNow ping, and llms.txt are pulling the right bots in.

Per-bot row shows hits in the last 7 days · 30 days · last-seen. "Hits" is HTTP requests, not unique URLs — a single bot fetching the sitemap + 50 listings counts as 51 hits. Bots with zero hits are omitted. The "Search engines" section only counts HTML-page crawlers; image / video / news variants (which feed separate indices) are excluded entirely. Requests to common exploit-scanner paths (/.env, /wp-admin/, etc.) are bucketed under "Bot farms" so they don't inflate the legitimate counts. Generated daily from /var/log/apache2/nowservingto-access.log.

These are our internal estimates from a per-call ledger (tools/cache/usage_log.jsonl). Authoritative billing lives in each provider's dashboard: Google Cloud Billing, Anthropic Console, Jina dashboard, X developer portal.

Snapshot: -