Free tool, runs in your browser
Drop in a server access log. See which AI crawlers visit, which pages they read, and which ones your server is turning away. The file never leaves your browser.
22 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta and others, split into training, search and assistant.
The pages each bot asks for most, so you can see whether they reach your money pages or get stuck on filters and feeds.
Blocked responses, 404s and redirects per bot. A crawler that mostly gets 403 will stop coming back.
The free audit reads your site the way a crawler does and scores it out of 100.
No. The file is read and counted inside your browser. Nothing is sent to our server, and closing the tab removes it.
Apache and Nginx combined logs, JSON lines such as Cloudflare Logpush, and CSV or TSV exports that have a user agent column. Up to 500,000 lines are read. For a bigger file, cut it down to the last week first.
On your own server it is usually /var/log/nginx/access.log or /var/log/apache2/access.log. Most hosts offer a raw access log download in the control panel. On Cloudflare, use Logpush or export from Log Explorer. Shopify does not give store owners raw logs.
Training crawlers such as GPTBot and ClaudeBot collect pages for building models. Search bots such as OAI-SearchBot and PerplexityBot build the index that answers cite. Assistant fetchers such as ChatGPT-User load a page at the moment a person asks about it, so each of those hits is a real question.
Yes. The user agent is just text. OpenAI, Perplexity, Google and Bing publish the IP ranges their bots use. If a result matters, check the IPs in your log against those lists.
Either they are blocked before they reach the server (a firewall or bot protection rule), your robots.txt tells them to stay away, or the log you loaded is from a CDN edge that filters them. The free audit on this site checks the first two.
.svg){kind=small}