GuidesAI search
Tutorial: spotting GPTBot, ClaudeBot and other AI bots in your logs
1. The names to look for
| User agent | Vendor | Role |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ChatGPT-User | OpenAI | Page opened at a user’s request |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Search |
| Claude-User | Anthropic | Page opened at a user’s request |
| PerplexityBot | Perplexity | Search |
| CCBot | Common Crawl | Open corpus, reused for training |
2. Count the visits
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot" access.log \
| sort | uniq -c | sort -rn
3. The pages a bot reads
For a log in “combined” format (Nginx, Apache):
grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
Caddy writes its logs in JSON: jq pulls out the path and the user agent.
4. Reading the result
- Search botsTheir favourite pages are your candidates for citations.
- Pages never seenInternal links, sitemap, speed: worth a review.
- Training botsAllow them or not, your choice.
Next: allowing or blocking AI bots. The context: how ChatGPT picks its sources.
Frequently asked questions
- Does GA4 see these bots?
- No: bots generally don't run the analytics JavaScript. Only server or CDN logs show them.
- Can a user agent lie?
- Yes, anyone can write “GPTBot”. Some vendors, such as OpenAI and Perplexity, publish their IP ranges so you can check.
- What do I do with these numbers?
- Spot the pages read by AI search bots, the ones never read, and decide on access in robots.txt.