If an AI assistant cannot read your website, it cannot quote it, and it will name someone else. This checker reads your robots.txt the way the crawlers do and tells you in a few seconds which ones you let in.
SHORT ANSWER
Type your website address. The checker fetches your robots.txt, your llms.txt and your homepage, then reports for 12 crawlers (including OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot) whether your rules allow them. It also flags a noindex tag, missing structured data, and a firewall that blocks AI bots. Free, no signup, nothing stored.
Run the check
What does the checker test?
- robots.txt rules for 12 crawlers: OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, ClaudeBot, Googlebot, Google-Extended, Bingbot and Applebot.
- A firewall test. We request your homepage once as a normal browser and once with the OAI-SearchBot user agent. If the second gets blocked, your host or CDN is turning AI crawlers away even when robots.txt allows them.
- noindex. A noindex tag keeps the page out of search results, and out of AI search answers built on them.
- Structured data. Whether your homepage has JSON-LD schema, and which types.
- llms.txt. Whether you publish the small file that points AI tools to your key pages.
- Readable text. How many words are in the raw HTML. A near-empty page often means the text only appears after JavaScript runs, which some AI crawlers do not run.
What do the results mean?
| Crawler | Who runs it | What blocking it costs you |
|---|---|---|
| OAI-SearchBot | OpenAI | Your pages can be left out of ChatGPT search answers. |
| PerplexityBot | Perplexity | Perplexity cannot surface and link your site. |
| Claude-SearchBot | Anthropic | Claude's search cannot index your pages. |
| Googlebot | Google Search, AI Overviews and AI Mode. | |
| Bingbot | Microsoft | Bing results and the search behind Copilot. |
| GPTBot, ClaudeBot, Google-Extended | OpenAI, Anthropic, Google | Used for model training. Blocking them is a choice, and OpenAI states its search crawler is controlled separately. |
The first five are the ones that decide whether you can be quoted. Training crawlers are a business choice, not an error.
How do I fix a blocked crawler?
- Open
yourdomain.com/robots.txt. Look for aUser-agentline naming the blocked crawler, or aUser-agent: *group withDisallow: /. - Add an explicit group that allows the crawler, for example
User-agent: OAI-SearchBotthenAllow: /. - If robots.txt is fine but the firewall test fails, check your host or CDN bot settings (Cloudflare and some hosts block AI crawlers by default).
- Run the check again.
Letting crawlers in is step one. Getting named is the rest: pages, reviews and listings. See AI search visibility for HVAC.
What this checker does not test
- Whether any AI assistant actually names you. That needs real questions, asked many times. The free 7-gap report does that.
- Pages other than your homepage, or path-level rules for deep pages.
- Crawl rate limits, or IP-based blocks that only trigger at volume.
Questions owners ask
How do I know if ChatGPT can read my website?
Check that your robots.txt allows OAI-SearchBot, the crawler OpenAI uses for ChatGPT search, and that your host or firewall does not block it. This checker tests both in a few seconds.
Should I block GPTBot?
That is a choice about model training, not search. OpenAI says OAI-SearchBot and GPTBot are controlled separately, so you can allow search and still block training.
Does an llms.txt file help me rank?
Not on its own. It is a small file that points AI tools to your key pages. It is cheap to add, but pages, reviews and listings do the real work.
Is this checker free?
Yes. No signup and no email. We do not store the address you check.
SOURCES CITED (5)
Every price and figure on this page was checked against these on .
- OpenAI, Overview of OpenAI crawlers, OAI-SearchBot and GPTBot are set independently
- Perplexity, Perplexity crawlers, PerplexityBot
- Anthropic, Does Anthropic crawl data from the web, Claude-SearchBot, ClaudeBot, Claude-User
- Google Search Central, AI features and your website, AI Overviews and AI Mode use normal Search eligibility
- llmstxt.org, The /llms.txt file, the llms.txt proposal