GuidesAI search

Tutorial: spotting GPTBot, ClaudeBot and other AI bots in your logs

By the Banzai Labs teamUpdated on 2 min read

GET /guides/… GPTBotGET /tarifs MozillaGET /blog/… ClaudeBotGET / MozillaGET /faq PerplexityBotGET /guides/… OAI-SearchBotGPTBotClaudeBotOAI-SearchBotPerplexityBotWho visits, which pages, and how often
In the server logs, AI bot visits stand out by their user agent.

1. The names to look for

User agent Vendor Role
GPTBot OpenAI Training
OAI-SearchBot OpenAI ChatGPT search
ChatGPT-User OpenAI Page opened at a user’s request
ClaudeBot Anthropic Training
Claude-SearchBot Anthropic Search
Claude-User Anthropic Page opened at a user’s request
PerplexityBot Perplexity Search
CCBot Common Crawl Open corpus, reused for training

2. Count the visits

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot" access.log \
  | sort | uniq -c | sort -rn

3. The pages a bot reads

For a log in “combined” format (Nginx, Apache):

grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Caddy writes its logs in JSON: jq pulls out the path and the user agent.

4. Reading the result

  • Search botsTheir favourite pages are your candidates for citations.
  • Pages never seenInternal links, sitemap, speed: worth a review.
  • Training botsAllow them or not, your choice.

Next: allowing or blocking AI bots. The context: how ChatGPT picks its sources.

Frequently asked questions

Does GA4 see these bots?
No: bots generally don't run the analytics JavaScript. Only server or CDN logs show them.
Can a user agent lie?
Yes, anyone can write “GPTBot”. Some vendors, such as OpenAI and Perplexity, publish their IP ranges so you can check.
What do I do with these numbers?
Spot the pages read by AI search bots, the ones never read, and decide on access in robots.txt.

Read next

All guides