GuidesAI search
Allowing or blocking AI bots in robots.txt
Two families of bots
Training
Search
The most common robots.txt
# Training: closed
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Search: open
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Three strategies
| Strategy | For whom |
|---|---|
| Open everything | Maximum visibility, content meant to be reused |
| Close training, open search | The most common compromise |
| Close everything | Paid or sensitive content |
The limits
- Visits triggered by a user (ChatGPT-User, Claude-User) follow rules specific to each vendor: read their documentation.
- Blocking doesn’t remove what has already been collected.
- Check the effect in your logs: spotting AI bots in your logs.
The basic syntax: writing a robots.txt.
Frequently asked questions
- Does blocking GPTBot remove my site from ChatGPT?
- No. GPTBot is for training; ChatGPT search goes through OAI-SearchBot. Blocking the first and allowing the second is a common choice.
- Does Google-Extended block Google's AI answers?
- No. Google-Extended covers the use of pages by Gemini and its models; ranking and AI Overviews in Google Search depend on Googlebot.
- Is robots.txt respected?
- The major vendors say they respect it. It is only an instruction: a malicious bot can ignore it, and then you block at the server or CDN level.