GuidesAI search

Allowing or blocking AI bots in robots.txt

By the Banzai Labs teamUpdated on 2 min read

robots.txtUser-agent: GPTBotDisallow: /User-agent: OAI-SearchBotAllow: /TrainingTraining dataSearchCitations and linksOne choice per bot, based on what it does with your pages
In robots.txt, you can close training and keep search open.

Two families of bots

Train, or cite
Training
  • GPTBot, ClaudeBot, CCBot
  • Google-Extended, Applebot-Extended
  • Your texts feed the models, with no link back
Search
  • OAI-SearchBot, Claude-SearchBot, PerplexityBot
  • Your pages can be cited with a link
  • Blocking them means vanishing from those answers

The most common robots.txt

# Training: closed
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search: open
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Three strategies

Strategy For whom
Open everything Maximum visibility, content meant to be reused
Close training, open search The most common compromise
Close everything Paid or sensitive content

The limits

  • Visits triggered by a user (ChatGPT-User, Claude-User) follow rules specific to each vendor: read their documentation.
  • Blocking doesn’t remove what has already been collected.
  • Check the effect in your logs: spotting AI bots in your logs.

The basic syntax: writing a robots.txt.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is for training; ChatGPT search goes through OAI-SearchBot. Blocking the first and allowing the second is a common choice.
Does Google-Extended block Google's AI answers?
No. Google-Extended covers the use of pages by Gemini and its models; ranking and AI Overviews in Google Search depend on Googlebot.
Is robots.txt respected?
The major vendors say they respect it. It is only an instruction: a malicious bot can ignore it, and then you block at the server or CDN level.

Read next

All guides