GuidesAI search
How ChatGPT and Perplexity pick their sources
The path of an answer
- 1RephraseThe question becomes one or more search queries.
- 2SearchIn a web index: the assistant’s own or a partner engine’s.
- 3ReadOnly a few pages, sometimes a few passages.
- 4CiteThe passages that answer cleanly, with a link.
What gets a page picked up
- FindableIndexed, fast, AI crawlers allowed in robots.txt.
- ReadableText in the HTML, headings phrased as questions, tables.
- PreciseA one-sentence answer at the top of each section, dated facts, a source.
Crawlers to know
| Crawler | For |
|---|---|
| OAI-SearchBot | ChatGPT search |
| GPTBot | Training OpenAI models |
| PerplexityBot | Perplexity search |
| Googlebot | Google, including its AI answers |
Blocking a training crawler does not stop a search crawler from citing you: they are two separate settings.
The full method: Generative engine optimization (GEO). The file meant for models: llms.txt.
Frequently asked questions
- Do you need to rank well on Google to be cited?
- It helps: assistants rely on search indexes, their own or a partner’s. A page that search engines cannot find is often invisible to them too.
- Do AI crawlers read JavaScript?
- Not always. Content present in the HTML sent by the server is the safest. See SSR, SSG, SPA.
- Can you pay to be cited?
- Organic answers are not for sale. What works: a page that answers a precise question better than the others.