One in three sites accidentally blocks the crawlers that feed ChatGPT and Perplexity. This AI crawl checker reads your robots.txt, meta robots and headers, then shows exactly which of 15 AI bots can see your pages.
Homepage or a deep page — the AI crawl checker evaluates the exact path you give it, because robots rules differ per path.
robots.txt rules for 15 AI crawlers, meta robots tags, X-Robots-Tag headers, llms.txt presence and sitemap declarations — the full crawlability test in one pass.
A per-bot allow/block table with the exact robots.txt rule responsible, so you can check website crawlability and repair it in minutes.
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot and more — every bot that matters for AI visibility.
CDN presets and old robots.txt templates silently block AI. If you have ever asked “is my site crawlable by ChatGPT?”, this answers it.
robots.txt is only layer one. We also test website crawlability at the page level: meta robots and X-Robots-Tag noindex directives.
Every verdict cites the exact Allow/Disallow line that caused it — no guessing which of your 40 robots rules fired.
The same pass confirms your llms.txt resolves and your sitemap is declared — the two files AI uses to navigate you.
Being readable is step one; being recommended is the goal. Pair this AI crawlability audit with a free Kairosy brand scan.
What is an AI crawler? It is the bot an AI company sends to read the public web: OpenAI’s GPTBot gathers training data, OAI-SearchBot powers ChatGPT search, ClaudeBot feeds Anthropic’s models, and PerplexityBot builds Perplexity’s index. When buyers ask those assistants for recommendations, the assistants can only talk about sites their crawlers could read. Blocking them is invisibility by configuration — which is why you should check if page is crawlable before worrying about content.
To check website crawlability yourself, read robots.txt for each bot’s User-agent group, then check the page for meta robots and X-Robots-Tag headers — or run this crawlability test and get all of it in five seconds. Asking “is my site crawlable?” once is not enough, either: themes, plugins and CDN rules change robots files silently, so a monthly AI crawlability audit (and a quick re-test after every migration) is the habit that keeps you visible.
“Our WAF template had been blocking GPTBot for a year. Found it in the first run, fixed it the same day.”
“I test website crawlability for every new client with this AI crawl checker. Half of them are blocking at least one AI bot without knowing.”
“The rule-level explanations are gold — it told me exactly which Disallow line hit Bytespider vs everything else.”
“Simple, honest and fast. The three-layer check (robots, meta, headers) caught a noindex header our CDN was injecting.”
A bot that AI companies use to read public webpages so their assistants can learn about — and recommend — what is on them.
Usually yes: no robots.txt means no restrictions, so crawlers assume everything public is fair game. You can still be blocked by meta robots tags, X-Robots-Tag headers or your firewall — which is exactly what this AI crawl checker inspects.
Run the tool on that page URL and find the bot in the results table. The verdict cites the robots.txt rule that applied, so you can check if page is crawlable for GPTBot while, say, Bytespider stays blocked.
Some publishers block trainers (GPTBot, CCBot) but allow search bots (OAI-SearchBot, PerplexityBot) to stay citable. That is a valid strategy — the point is choosing it deliberately, not inheriting it from a template.
Crawlable means readable, not recommended. Run a free Kairosy scan to see what ChatGPT, Gemini, Claude and Perplexity actually tell buyers about you, and what to fix if the answer stings.
Kairosy asks ChatGPT, Gemini, Claude and Perplexity the questions your buyers ask — and shows whether they recommend you, ignore you, or send buyers to a rival.
Run a free AI brand scan