AI search tools like ChatGPT, Perplexity, Gemini and Claude use their own crawlers to read web pages. If your site blocks those crawlers, your pages can't be read, and they're less likely to show up in AI answers. You could even be blocking them without knowing it, through a robots.txt rule, a security plugin, or a CDN setting.
Site Audit checks this for you on every crawl. This article shows where to find the results and what to do if a crawler is blocked.
The short answer
Open your Site Audit project, go to Issues, and filter by AI Search. If any AI crawler is blocked, you'll see an issue here.
To unblock a crawler, update your robots.txt file (or your CDN or security settings), then run a new crawl to confirm.
It lists which crawlers are blocked and on which pages. If that issue isn't there, the AI crawlers that Site Audit checks can reach the pages you crawled.
How to check AI crawler access in Site Audit
In SpyFu, open Site Audit and select your project. If you haven't run a crawl yet, start one. It takes about five minutes for the default 100 pages.
Go to Issues and filter by the AI Search tag.
Look for an issue about AI search bots being blocked. (It only appears when at least one AI crawler is blocked.) The AI Search tag can also include a few related content checks, so you may see other issues here too.
Note any blocked crawlers and the pages they can't reach. A crawler blocked on every page usually points to a site-wide rule. A crawler blocked on only some pages usually points to a rule for one folder or section.
If there's no blocked AI search bots issue, you're in good shape. Most AI crawlers can reach your pages.
Tip: The check covers the pages in your crawl. If your crawl is set to 100 pages and your site is larger, raise the page limit in Crawl Settings to check more of it.
If AI crawlers are blocked on your homepage or across most of your site, Site Audit also flags it as a site-level alert at the top of your Overview, so you'll see it as soon as you open the project.
The AI crawlers that SpyFu's Site Audit checks
Crawler | Platform | What it does |
OAI-SearchBot | ChatGPT (OpenAI) | Finds and indexes pages for ChatGPT search results |
ChatGPT-User | ChatGPT (OpenAI) | Visits a page when a ChatGPT user asks about it or shares a link |
PerplexityBot | Perplexity | Finds and indexes pages for Perplexity answers |
Perplexity-User | Perplexity | Visits a page when a Perplexity user's question calls for it |
Claude-SearchBot | Claude (Anthropic) | Finds and indexes pages for Claude's search results |
Claude-User | Claude (Anthropic) | Visits a page when a Claude user asks about it |
Google-Extended | Gemini (Google) | A robots.txt setting, not a separate crawler. It controls whether Google can use your content for Gemini. It doesn't affect your Google Search rankings. |
What about GPTBot and ClaudeBot?
Those crawlers collect content for training AI models, rather than for search answers. Whether to allow them is a separate decision, and blocking them doesn't stop your pages from appearing in AI search.
Why some crawlers get through and others don't
There are two separate ways a site can keep a crawler out, and they work very differently.
Robots.txt is a request. It's a public list of which crawlers may visit which pages. Well-behaved crawlers read it and follow it, but nothing forces them to. Site Audit can read these rules and show you who they allow or block.
Bot protection is enforced. CDNs, firewalls and bot-management tools sit in front of your site and decide on every single visit whether to let it in. A blocked visitor gets a challenge or block page instead of your content, no matter what robots.txt says.
Bot protection doesn't just check a name. For each visit, it typically weighs:
Who the visitor says it is. Every crawler sends a user agent, a name like OAI-SearchBot.
Whether it can prove it. Major crawlers publish the IP addresses they crawl from, so a tool can confirm a visit really came from them. A name without a matching address looks like an imposter.
How it behaves. A crawler that reads pages at a steady, polite pace looks different from one that loads pages like a browser, runs scripts, or arrives in bursts.
That's why results can differ from one crawler to the next, even on the same site:
A crawler with a known name and published IP addresses may get through, while another visitor that's harder to identify gets a challenge page.
Two crawlers from the same company can get different results if they identify themselves or behave differently.
A crawler can be allowed in robots.txt and still be blocked by bot protection.
One visit can get through while a later one is challenged, because many tools re-score each visit.
This applies to every tool that visits your site, not just AI crawlers. SEO tools, including SpyFu's Site Audit, can reach your pages on one visit and be stopped on another for the same reasons.
The practical takeaway: if robots.txt allows a crawler but you suspect it's still being blocked, the answer is usually in your bot-protection settings. Most of these tools let you allow specific crawlers by name, by verified IP address, or both.
How to unblock an AI crawler
Most blocks come from one of three places. Check them in this order.
1. Your robots.txt file
Open yoursite.com/robots.txt in a browser. Look for a User-agent line naming the blocked crawler, followed by Disallow: /. For example:
User-agent: OAI-SearchBot Disallow: /
To allow the crawler, change Disallow: / to Allow: /, or remove those two lines. Also check for a catch-all rule (User-agent: * with Disallow: /), which blocks every crawler that doesn't have its own rule.
On WordPress, robots.txt is often managed by an SEO plugin such as Yoast or Rank Math. Edit it there rather than on the server.
2. Your CDN or security settings
Some CDNs and firewalls, including Cloudflare, have settings that block AI crawlers. These blocks happen before a crawler ever reads robots.txt. If your robots.txt allows a crawler but you still think it's being blocked, ask whoever manages your CDN or security settings to check the bot-management rules.
3. A security or caching plugin
Some WordPress security plugins block unfamiliar bots by default. Check the plugin's bot or firewall settings for AI crawler names.
4. Run a new crawl
After you make the change, click Re-Crawl in Site Audit. When the crawl finishes, the crawler should show Allowed.
Not sure whether to allow a crawler? If you want your pages to show up in AI search answers, allow the search crawlers in the table above. Blocking them is a business choice some sites make on purpose, and that's fine as long as it's a decision rather than an accident.
Common questions
Will allowing AI crawlers bring me more traffic?
It makes your pages eligible to appear in AI answers. Today, clicks from AI search are a small share of most sites' traffic. The bigger benefit is being visible when people ask AI tools about your topic.
Does blocking AI crawlers hurt my Google rankings?
No. Googlebot, which crawls for Google Search, is separate from these crawlers. Blocking Google-Extended doesn't affect how your pages rank in Google Search.
I didn't block anything. Why is a crawler blocked?
Common causes include a CDN or firewall setting that blocks AI bots by default, a security plugin, or a robots.txt rule added by a developer or plugin. Check the three places listed above.
Do I need to check this more than once?
Yes. Plugin updates, CDN setting changes and site migrations can change who's blocked. Site Audit checks crawler access on every crawl, so scheduling regular crawls in Crawl Settings keeps an eye on it for you.
Is the Site Audit crawler one of these bots?
No. Site Audit crawls as RSiteAuditor. If your security settings block it, your audit may show pages as blocked or empty. See Turning On JavaScript Rendering for Site Audit for how to allow it.