Look at your server logs and you will likely find visitors with names like GPTBot, ClaudeBot and PerplexityBot. These are AI crawlers, and they are now a regular part of web traffic.
Some site owners block them all to protect their content. Others allow everything in the hope of appearing in AI answers. Both approaches can cost you something.
The key is that AI crawlers are not all the same. Some collect content for model training. Others power AI search results. Others only visit a page when a user asks a question.
Blocking the wrong one can remove your site from AI search answers without protecting much at all. Getting it right is a practical part of how AI search decides which sites get cited.
In this guide, you’ll learn what each major AI crawler does, how robots.txt controls them, and how to decide which bots to allow or block based on your goals.
What Are AI Crawlers?
AI crawlers are automated bots operated by AI companies that visit websites to collect content. Depending on the bot, that content may be used to train AI models, to build an index for AI search answers, or to fetch a specific page when a user asks an AI assistant a question.
They identify themselves with a user agent name, such as “GPTBot” or “ClaudeBot”. That name is what you target in robots.txt.
The Three Types of AI Crawlers
- Training crawlers: collect content that may be used to train future AI models
- Search crawlers: index content so it can be shown and linked in AI search answers
- User-triggered fetchers: visit a page because a user asked the assistant about it or asked a question that requires it
This distinction matters more than anything else in this guide. Blocking a training crawler and blocking a search crawler have very different effects.
Why AI Crawlers Matter for GEO
AI search tools can only cite pages they can access. If an AI search crawler is blocked, your content is far less likely to appear as a source in that platform’s answers.
At the same time, many publishers have reasonable concerns about their content being used to train models without compensation.
- Allowing search crawlers supports visibility in AI answers
- Blocking training crawlers signals that you do not want your content used for training
- Most major AI companies now let you make these choices separately
That means you usually do not have to choose between “all in” and “all out”.
Imagine a hypothetical recipe blogger who blocked every AI bot in 2024 to protect their recipes. A year later, they notice competitors being cited in ChatGPT and Perplexity answers for recipe questions while their own site never appears. Their blanket block removed them from AI search, not just from training.
How Does robots.txt Control AI Crawlers?
robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may access. Its rules are defined in the Robots Exclusion Protocol standard (RFC 9309).
How a Rule Works
Each group of rules starts with a User-agent line naming the bot, followed by Disallow or Allow lines for paths.
User-agent: GPTBot
Disallow: /
This example asks GPTBot not to crawl any page on the site.
What robots.txt Cannot Do
- It is a request, not a lock: well-behaved bots follow it, but it cannot technically stop a crawler
- It does not remove content that has already been collected
- Some user-triggered fetchers may not follow it, because a person requested the page
- Each subdomain needs its own robots.txt file
If you need to enforce a block, server or CDN-level bot controls can do that. For most sites, robots.txt is the right starting point.
The Major AI Crawlers and What They Do
The table below summarizes the main AI user agents based on each company’s own documentation. Names and behavior can change, so check the official pages before making major decisions.
| User Agent | Company | Purpose | Follows robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search results | Yes |
| ChatGPT-User | OpenAI | User-triggered page visits | May not apply |
| ClaudeBot | Anthropic | Model training | Yes |
| Claude-SearchBot | Anthropic | Search result quality | Yes |
| Claude-User | Anthropic | User-triggered page visits | Yes |
| PerplexityBot | Perplexity | Perplexity search results | Yes |
| Perplexity-User | Perplexity | User-triggered page visits | Generally no |
| Google-Extended | Gemini training and grounding | Yes (control token) |
OpenAI Crawlers: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI documents its crawlers on its bots overview page. Each one is controlled separately.
GPTBot
GPTBot crawls content that may be used to train OpenAI’s foundation models. According to OpenAI, blocking GPTBot does not affect whether your site appears in ChatGPT search.
OAI-SearchBot
OAI-SearchBot is used to surface websites in ChatGPT’s search features. If you block it, OpenAI says your site will not be shown in ChatGPT search answers, apart from possible navigational links.
ChatGPT-User
ChatGPT-User visits pages when a user asks ChatGPT a question or uses a GPT action. Because these visits are user-initiated, OpenAI notes that robots.txt rules may not apply.
The practical result: you can block GPTBot to opt out of training while allowing OAI-SearchBot to stay visible in ChatGPT search. Tactics for earning those citations are covered in the guide on how to get cited by ChatGPT.
Anthropic Crawlers: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic describes its bots in its help article on how site owners can manage its crawlers.
ClaudeBot
ClaudeBot collects public web content that may contribute to model training. Blocking it signals that your future content should be excluded from training datasets.
Claude-SearchBot
Claude-SearchBot crawls the web to improve the quality and relevance of search results for Claude users.
Claude-User
Claude-User fetches pages when people ask Claude questions. Anthropic notes that blocking it may reduce your site’s visibility in user-directed web search.
Anthropic also supports the non-standard Crawl-delay directive if you want to slow its crawlers down rather than block them.
Perplexity Crawlers: PerplexityBot and Perplexity-User
Perplexity documents two user agents on its crawlers page.
- PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
- Perplexity-User visits pages when users ask questions. Perplexity states that it generally ignores robots.txt because the request comes from a user.
Because PerplexityBot is a search crawler, blocking it mainly reduces your chances of appearing in Perplexity answers. Platform-specific tips are covered in the guide to getting cited by Gemini and Perplexity.
Google-Extended, Googlebot and AI Overviews
Google works differently from the AI-only companies, because its AI features are built into Search.
Google-Extended
Google-Extended is not a separate crawler but a control token. According to Google’s crawler documentation, it manages whether content Google crawls may be used to train future Gemini models and for grounding in Gemini apps and the Vertex AI API.
Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal.
Googlebot and AI Overviews
AI Overviews and AI Mode are part of Google Search, so they rely on Googlebot. There is no separate robots.txt token to remove your site from AI Overviews while staying in regular search. Blocking Googlebot would remove you from Search entirely.
If you want to limit how much of a page Google shows, snippet controls such as nosnippet and max-snippet are the tools to use.
Other AI Crawlers Worth Knowing
- Applebot-Extended: a control token that lets publishers opt out of their content being used to train Apple’s AI models, separate from the regular Applebot used for features such as Siri and Spotlight
- CCBot: the Common Crawl bot, which builds an open web archive that is widely used as a source of AI training data
- Bingbot: Microsoft’s search crawler; Copilot draws on Bing’s index, so blocking Bingbot can affect both Bing and Copilot visibility
New AI user agents appear regularly. Reviewing your server logs every few months helps you spot bots you have not made a decision about yet.
Should You Block AI Crawlers? A Decision Guide
There is no single right answer. The best setup depends on what you want from AI platforms.
| Your Goal | Block | Allow |
|---|---|---|
| Maximum AI search visibility | Nothing | All AI crawlers |
| Visibility without training use | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot |
| Opt out of AI as much as possible | All AI crawlers and training tokens | Regular search crawlers only |
| Protect specific sections only | AI crawlers on chosen paths | AI crawlers everywhere else |
When Allowing Makes Sense
- You rely on visibility and brand awareness to win customers
- Your content is marketing or educational, not the product itself
- You want to be cited as a source in AI answers
When Blocking Makes Sense
- Your content is your product, such as paid research or premium articles
- You have licensing agreements or legal reasons to restrict use
- Bot traffic is causing real server load problems
For most businesses using content to attract customers, the “visibility without training use” or “allow everything” setups are the most common choices.
Example robots.txt Setups
These examples show the pattern. Adjust them to your goals and check the user agent names against each company’s current documentation.
Allow AI Search, Block Training
User-agent: GPTBot
Disallow: /User-agent: ClaudeBot
Disallow: /User-agent: Google-Extended
Disallow: /User-agent: OAI-SearchBot
Allow: /User-agent: PerplexityBot
Allow: /
Block AI Crawlers From One Section
User-agent: GPTBot
Disallow: /members/User-agent: ClaudeBot
Disallow: /members/
If you use Rank Math, you can edit robots.txt under General Settings → Edit robots.txt, as long as no physical robots.txt file exists on the server.
How to Check Your AI Crawler Settings
- Open yoursite.com/robots.txt and read every User-agent group
- Check for broad rules such as “User-agent: *” with “Disallow: /” that block everything
- Confirm each AI user agent name is spelled exactly as documented
- Check your CDN or security plugin for separate AI bot blocking settings
- Review server logs to see which AI bots visit and how often
- Re-check after theme, plugin or hosting changes
Some CDNs and security tools block AI bots by default. Your robots.txt may allow a crawler while your firewall quietly blocks it.
Common AI Crawler Mistakes to Avoid
- Blocking all AI bots without realizing search crawlers are included
- Confusing robots.txt with llms.txt: only robots.txt controls access, as explained in the guide to llms.txt
- Misspelling user agent names, so rules never match
- Forgetting subdomains, which need their own robots.txt
- Relying on IP blocking, which can stop bots from reading your robots.txt preferences
- Blocking Googlebot to avoid AI Overviews, which removes you from Search entirely
- Setting rules once and never reviewing them as new bots appear
AI Crawler Checklist
- Decision made: visibility, training opt-out, or full opt-out
- Training crawlers handled (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot)
- Search crawlers handled (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
- User agent names checked against official documentation
- No accidental site-wide blocks for all bots
- CDN and security plugin settings match your robots.txt
- Server logs reviewed for new AI user agents every quarter
Final Takeaway
AI crawlers are not one group. Training crawlers, search crawlers and user-triggered fetchers do different jobs, and the major AI companies now let you control them separately.
For most businesses, the smartest approach is deliberate rather than blanket: decide whether you want to appear in AI search, decide whether you accept training use, and set rules for each bot accordingly.
Then check that your CDN and security tools agree with your robots.txt, and review your settings regularly as new crawlers appear.
If you’d like help reviewing your crawler settings as part of a wider AI visibility review, our team can help with a GEO audit.
Frequently Asked Questions
What are AI crawlers?
AI crawlers are bots operated by AI companies that visit websites to collect content. Depending on the bot, the content may be used for model training, for AI search answers, or to answer a specific user’s question.
Should I block GPTBot?
It depends on whether you want your content used for training OpenAI’s models. According to OpenAI, blocking GPTBot does not affect whether your site appears in ChatGPT search, which is controlled by OAI-SearchBot.
Will blocking AI crawlers hurt my Google rankings?
Blocking AI-specific crawlers such as GPTBot or ClaudeBot does not affect Google Search. Google also states that blocking Google-Extended does not affect inclusion or ranking in Google Search.
Can I block AI Overviews with robots.txt?
Not separately. AI Overviews are part of Google Search and rely on Googlebot. You can use snippet controls such as nosnippet or max-snippet to limit what is shown, but blocking Googlebot would remove your pages from Search.
What is the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects content that may be used to train Anthropic’s models. Claude-SearchBot crawls the web to improve search results for Claude users. They can be allowed or blocked separately.
Does PerplexityBot use my content for training?
Perplexity states that PerplexityBot is used to surface and link websites in its search results, not to crawl content for AI foundation models. Blocking it mainly reduces your visibility in Perplexity answers.
Do all AI crawlers follow robots.txt?
Not always. Major training and search crawlers say they follow robots.txt, but some user-triggered fetchers may not, because a person requested the page. robots.txt is a request rather than a technical block.
How do I see which AI crawlers visit my site?
Check your server access logs or your hosting and CDN analytics for user agent names such as GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot. Many security plugins and CDNs also report bot traffic.