# AI Search Check > Check one URL for AI search: robots.txt per AI crawler, llms.txt, a Markdown version, sitemap, JSON-LD, headings, citability and JS-only content. - URL: https://bytescope.dev/en/ai-search-check/ - Locale: en · Also: [ja](https://bytescope.dev/ja/ai-search-check.md) · [zh-Hant](https://bytescope.dev/zh-Hant/ai-search-check.md) - Runs: The URL is fetched by our Worker, then scored in your browser. Nothing is stored except an anonymous success or failure count. - Requires: Any modern browser (Chrome, Edge, Firefox, Safari), desktop and mobile. - Updated: 2026-09-08 ## What it does Paste a URL and this tool answers the three questions an AI answer engine asks before it cites a page: can its crawler fetch it, can it read the content without running JavaScript, and is there anything concrete worth quoting. You get a score out of 100 and ten checks. Each one says what it found, why it matters and how to fix it, with snippets you can paste straight into `robots.txt`, `llms.txt` or your page head. ### What it checks `robots.txt` is evaluated crawler by crawler for `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `PerplexityBot`, `Google-Extended`, `Applebot-Extended`, `CCBot` and `Bingbot`, using RFC 9309 matching. Blocking a training crawler is treated as a legitimate choice; blocking a search or user-triggered crawler fails the check, because that is what takes you out of answers. It also reads `/llms.txt` and probes its first ten links, looks for a Markdown version of the page (`.md`, `index.md`, or a `rel="alternate" type="text/markdown"` link), samples five sitemap URLs, lists JSON-LD types with missing required fields, and checks the heading outline, status code, redirects, `noindex` and canonical. Below the ten checks is an optional Query relevance section that does not count toward the score. Type up to five questions a reader might ask: the page is split into passages by heading, an open-source embedding model (EmbeddingGemma 2) turns each passage and question into a vector, and the three closest passages per question are listed with their scores. When even the best passage scores less than 0.05 above the page median, the question is flagged: no passage stands out as closer to it than the others. This simulates retrieval with an open-source model; it is not the model Google, ChatGPT or Perplexity actually use, so a score only means something next to other passages on the same page. Ranking follows closeness of meaning, not understanding, and very abstract wording works poorly. The first use downloads a model of about 234 MB, which then stays in your browser. It needs WebGPU, which means desktop Chrome or Edge; Safari, Firefox and mobile browsers are untested. The page is fetched through our Worker just like the checks above, while the vectors are computed in your browser. Only this one page is analyzed, not the whole site. ### What it does not do It checks this one page only and never crawls the site. The page's JavaScript is not run, just as most AI crawlers do not run it, so the JavaScript check is an estimate from the raw HTML: visible text, script weight, and empty mount nodes such as `#root` or `#__next`. hreflang is checked on this page only (a self-reference, a match with ``, valid codes); confirming that the other language versions point back would mean fetching every one of them. Citability is a counting heuristic, numbers, dates, tables and definition sentences against marketing adjectives, not a judgement of how good the writing is. ### How the fetch works A browser cannot read another site's pages directly, so the URL is sent to our Cloudflare Worker. It fetches the page, `robots.txt`, `llms.txt`, the sitemap and the Markdown candidates with one 8-second deadline, at most 3 redirects, a 2 MB cap per response and no cookies, and refuses private or local addresses, including hostnames that resolve to one. The Worker hands the raw responses back and the scoring runs in your browser. Nothing about the page is stored; the only record is an anonymous count of checks that succeeded or failed. ## Limits Does not crawl a whole site, does not run the page's JavaScript, and does not verify hreflang pages point back. It checks one URL, fetched through our Worker; nothing is stored beyond an anonymous success or failure count. ## FAQ ### Is my site blocking GPTBot? Run the check and open the robots.txt card: it lists every AI crawler as allowed or blocked, with the rule that decided it. `GPTBot` is OpenAI's training crawler. ChatGPT search uses `OAI-SearchBot` and fetches a user asks for go out as `ChatGPT-User`, so blocking `GPTBot` alone does not remove you from ChatGPT answers. ### Does ChatGPT index my site? Nobody outside OpenAI can look inside its index, and this tool does not pretend to. It checks what decides whether indexing can happen: `OAI-SearchBot` and `ChatGPT-User` are allowed, the page answers 200 without `noindex`, and the content is in the HTML rather than painted in by JavaScript. ### How is the score calculated? Each check has a weight: robots.txt, citability and JavaScript dependence 15 each; llms.txt, sitemap, structured data and headings 10 each; Markdown version, response and language 5 each. A pass counts in full, needs work counts half, a fail counts zero, and a check that could not run is left out of the total. ### Do I need an llms.txt file? No search engine requires one. It is a proposed convention (llmstxt.org) for a short Markdown index of your key pages, and some AI tools read it. A missing file costs 10 of the 100 points here. If you want one, the llms.txt generator builds it from your sitemap. ### Why does my JavaScript app score so low? Most AI crawlers do not run JavaScript. If the server sends an empty `#root` plus a large bundle, the crawler sees almost no text. Server-side rendering or a static export puts the content into the HTML. This check reads only the raw HTML, so it shows what such a crawler gets. ### Is the URL I check stored anywhere? No. The Worker fetches it, returns the responses to your browser and keeps nothing; the analytics event only says whether a check succeeded. The Worker is rate-limited per IP address, and it refuses private addresses and hostnames that resolve to them. ### How do I know whether my page answers a reader's question? Use the Query relevance section under the checks. Type the question and it lists the three passages on this page whose meaning is closest, each with a score, and says so when no passage stands out from the rest of the page. That shows which passage a retrieval system would most likely pick, not whether an AI will cite you: the scores come from an open-source model rather than from Google, ChatGPT or Perplexity, and they only compare passages within this page. It needs WebGPU (desktop Chrome or Edge) and a one-time download of about 234 MB. ## Related tools - [llms.txt Generator](https://bytescope.dev/en/llms-txt-generator.md) — Build an llms.txt from your sitemap, with titles and descriptions per page. - [AI Crawler Log](https://bytescope.dev/en/ai-crawler-log.md) — See which AI crawlers actually visit, from your server logs.