# AI Crawler Log > See which AI crawlers hit your site from nginx, Apache, Caddy or Cloudflare logs: GPTBot, ClaudeBot, PerplexityBot, top pages, and pages never fetched. - URL: https://bytescope.dev/en/ai-crawler-log/ - Locale: en · Also: [ja](https://bytescope.dev/ja/ai-crawler-log.md) · [zh-Hant](https://bytescope.dev/zh-Hant/ai-crawler-log.md) - Runs: Your logs are parsed in this browser and never uploaded. Only a sitemap URL, if you paste one, is fetched through our Worker; nothing is stored beyond an anonymous success or failure count. - Requires: Any modern browser (Chrome, Edge, Firefox, Safari), desktop and mobile. - Updated: 2026-09-08 ## What it does Drop your access logs, or a whole folder of them, and this tool tells you which AI crawlers came, how often, and what they fetched. Every bot gets its request count with a per-day trend, its top pages, its status codes, its average response time and the median gap between visits. Add your sitemap and you get the most useful table of all: the pages you submitted that no AI crawler ever requested. ### Formats it reads Cloudflare Logpush `http_requests` JSON lines, the nginx and Apache `combined` format (the two are byte-identical, so they share one parser; an optional trailing `$request_time` or `%D` gives the response time), Caddy's JSON access log, and the Vercel and Netlify log drains. The format is picked per file from its first 20 non-empty lines, so one folder can mix them. `.gz` files, rotated ones like `access.log.2.gz` included, are unpacked in the browser. Up to 2,000 files per run. ### How crawlers are recognised By user-agent, in a fixed order: `OAI-SearchBot`, `ChatGPT-User` and `GPTBot` for OpenAI, `Claude-SearchBot`, `Claude-User` and `ClaudeBot` for Anthropic, then Perplexity, DuckDuckGo, Mistral, Meta, Amazon, ByteDance, Common Crawl, Cohere, Diffbot and You.com, plus Googlebot, Bingbot, Applebot, Yandex, Baidu and Petal as search engines. AI crawlers are split into training, search and user-triggered, because blocking one is not the same decision as blocking another. Anything that looks like a bot but matches none of them is listed under other bots with its raw user-agent, so you can see what else is out there. ### What it cannot tell you A user-agent is a claim, not proof: anyone can send `GPTBot` in a header. This is traffic observation, not identity verification; checking that a crawler is genuine takes a reverse DNS lookup of its IP, which this tool does not do. It also does not stream live logs, which would need a server; export a file and drop it in. Log files never leave your browser. The parser reads only time, method, path, host, status, user-agent and response time, never IP addresses, referers or cookies. Only a sitemap URL, if you paste one, goes to our Worker to be fetched. ## Limits Does not verify that a user-agent is genuine (no reverse DNS), does not stream live logs, and does not upload the log. Only a sitemap URL, when you give one, is fetched through our Worker. ## FAQ ### How do I find GPTBot in my nginx access log? Drop the log file here. Each line is parsed in your browser and every request whose user-agent contains `GPTBot` is counted under OpenAI's training crawler, with its pages, status codes and visit times. `OAI-SearchBot` (ChatGPT search) and `ChatGPT-User` (fetches a user asked for) are counted separately. ### Are my log files uploaded anywhere? No. Parsing runs in a Web Worker in this tab, and only the aggregated counts reach the page. IP addresses are not even read. The one thing sent to our server is a sitemap URL, if you choose to paste one, so the Worker can fetch it past the browser's cross-origin rules; an uploaded sitemap file stays local. ### Can a bot fake its user-agent? Yes, and some scrapers do. The counts here are what the requests claimed to be. To confirm a crawler is genuine you have to reverse-look-up its IP and check the vendor's published ranges; this tool does not do that, so treat a surprising spike as a lead to check, not a fact. ### Which sitemap pages has no AI crawler visited? Paste the sitemap URL or upload the XML after analysing your logs. Each sitemap URL is reduced to its path and compared with every path an AI crawler requested; the ones never requested are listed and can be downloaded as a text file. A sitemap index lets you load up to 20 child sitemaps, one at a time. ### Why are some lines counted as bad or skipped? A bad line is one that does not parse in the format detected for its file, such as a truncated JSON line or a line in another format. It costs only that line. Skipped lines are in the right format but are not requests, like Caddy's non-access logs or Vercel build messages. Both counts are shown so nothing disappears silently. ## Related tools - [AI Search Check](https://bytescope.dev/en/ai-search-check.md) — Check one URL for AI search: robots.txt per crawler, llms.txt, JSON-LD. - [llms.txt Generator](https://bytescope.dev/en/llms-txt-generator.md) — Build an llms.txt from your sitemap, with titles and descriptions per page.