GuidesAI search

How ChatGPT and Perplexity pick their sources

By the Banzai Labs teamUpdated on 3 min read

12Pages readCited sourcesSearch
The assistant searches, reads a few pages and cites only some of them in its answer.

The path of an answer

  1. 1RephraseThe question becomes one or more search queries.
  2. 2SearchIn a web index: the assistant’s own or a partner engine’s.
  3. 3ReadOnly a few pages, sometimes a few passages.
  4. 4CiteThe passages that answer cleanly, with a link.

What gets a page picked up

  • FindableIndexed, fast, AI crawlers allowed in robots.txt.
  • ReadableText in the HTML, headings phrased as questions, tables.
  • PreciseA one-sentence answer at the top of each section, dated facts, a source.

Crawlers to know

Crawler For
OAI-SearchBot ChatGPT search
GPTBot Training OpenAI models
PerplexityBot Perplexity search
Googlebot Google, including its AI answers

Blocking a training crawler does not stop a search crawler from citing you: they are two separate settings.

The full method: Generative engine optimization (GEO). The file meant for models: llms.txt.

Frequently asked questions

Do you need to rank well on Google to be cited?
It helps: assistants rely on search indexes, their own or a partner’s. A page that search engines cannot find is often invisible to them too.
Do AI crawlers read JavaScript?
Not always. Content present in the HTML sent by the server is the safest. See SSR, SSG, SPA.
Can you pay to be cited?
Organic answers are not for sale. What works: a page that answers a precise question better than the others.

Read next

All guides