AI Crawler Log Analyzer — Find GPTBot in Logs

1. Your access logs

Drop log files or a whole folder. Read: Cloudflare Logpush, nginx / Apache combined, Caddy JSON, Vercel and Netlify log drains. .gz files are unpacked here, in a background worker.

or drop files or a folder here

2. Compare with your sitemap

Paste a sitemap URL or upload the XML to list the pages no AI crawler requested in these logs. Only the URL goes to our Worker; an uploaded file stays here.

What this can and cannot tell you

A user-agent can be faked. This is traffic observation, not identity verification: confirming a crawler is genuine needs a reverse DNS lookup of its IP, which this tool does not do.
No live log streaming: that would need a server. Export a log file and drop it here.
Your logs are parsed in this browser and never uploaded. Only a sitemap URL, if you paste one, is fetched through our Worker; nothing is stored beyond an anonymous success or failure count.

About this tool

Drop your access logs, or a whole folder of them, and this tool tells you which AI crawlers came, how often, and what they fetched. Every bot gets its request count with a per-day trend, its top pages, its status codes, its average response time and the median gap between visits. Add your sitemap and you get the most useful table of all: the pages you submitted that no AI crawler ever requested.

Formats it reads

Cloudflare Logpush http_requests JSON lines, the nginx and Apache combined format (the two are byte-identical, so they share one parser; an optional trailing $request_time or %D gives the response time), Caddy's JSON access log, and the Vercel and Netlify log drains. The format is picked per file from its first 20 non-empty lines, so one folder can mix them. .gz files, rotated ones like access.log.2.gz included, are unpacked in the browser. Up to 2,000 files per run.

How crawlers are recognised

By user-agent, in a fixed order: OAI-SearchBot, ChatGPT-User and GPTBot for OpenAI, Claude-SearchBot, Claude-User and ClaudeBot for Anthropic, then Perplexity, DuckDuckGo, Mistral, Meta, Amazon, ByteDance, Common Crawl, Cohere, Diffbot and You.com, plus Googlebot, Bingbot, Applebot, Yandex, Baidu and Petal as search engines. AI crawlers are split into training, search and user-triggered, because blocking one is not the same decision as blocking another. Anything that looks like a bot but matches none of them is listed under other bots with its raw user-agent, so you can see what else is out there.

What it cannot tell you

A user-agent is a claim, not proof: anyone can send GPTBot in a header. This is traffic observation, not identity verification; checking that a crawler is genuine takes a reverse DNS lookup of its IP, which this tool does not do. It also does not stream live logs, which would need a server; export a file and drop it in. Log files never leave your browser. The parser reads only time, method, path, host, status, user-agent and response time, never IP addresses, referers or cookies. Only a sitemap URL, if you paste one, goes to our Worker to be fetched.

Frequently asked questions

How do I find GPTBot in my nginx access log?

Drop the log file here. Each line is parsed in your browser and every request whose user-agent contains GPTBot is counted under OpenAI's training crawler, with its pages, status codes and visit times. OAI-SearchBot (ChatGPT search) and ChatGPT-User (fetches a user asked for) are counted separately.

Are my log files uploaded anywhere?

No. Parsing runs in a Web Worker in this tab, and only the aggregated counts reach the page. IP addresses are not even read. The one thing sent to our server is a sitemap URL, if you choose to paste one, so the Worker can fetch it past the browser's cross-origin rules; an uploaded sitemap file stays local.

Can a bot fake its user-agent?

Yes, and some scrapers do. The counts here are what the requests claimed to be. To confirm a crawler is genuine you have to reverse-look-up its IP and check the vendor's published ranges; this tool does not do that, so treat a surprising spike as a lead to check, not a fact.

Which sitemap pages has no AI crawler visited?

Paste the sitemap URL or upload the XML after analysing your logs. Each sitemap URL is reduced to its path and compared with every path an AI crawler requested; the ones never requested are listed and can be downloaded as a text file. A sitemap index lets you load up to 20 child sitemaps, one at a time.

Why are some lines counted as bad or skipped?

A bad line is one that does not parse in the format detected for its file, such as a truncated JSON line or a line in another format. It costs only that line. Skipped lines are in the right format but are not requests, like Caddy's non-access logs or Vercel build messages. Both counts are shown so nothing disappears silently.