How AI assistants choose the sources they cite
Ask ChatGPT or Perplexity a buying question — "best contract-review software for a small legal team," say — and the answer arrives with footnotes. A handful of pages, out of everything on the web, got picked as evidence. Whoever owns those pages is now part of the sales conversation. Everyone else isn't.
It is worth being precise about how those few pages get chosen, because the mechanism is documented, and it is not what most people assume.
Two systems, not one
An assistant answering with citations is running two separate systems. The language model supplies fluency and synthesis. The citations come from a retrieval layer: the assistant turns your question into searches, pulls candidate pages from a web index, and grounds its answer in what it retrieved. The model writes; the retrieval layer decides which sources exist.
The index behind that retrieval is built by crawlers, same as classic search. OpenAI documents a dedicated crawler, OAI-SearchBot, for surfacing sites in ChatGPT's search features — explicitly separate from GPTBot, its training crawler. Perplexity draws the same line: PerplexityBot builds its search index and, per Perplexity's own documentation, "is not used to crawl content for AI foundation models."
So the first gate is old-fashioned: can the crawler reach your pages, and are they in the index. A site that renders only in JavaScript, blocks the wrong user agents, or hides its substance behind interaction has left the race before any AI-specific factor applies.
What survives selection
Retrieval produces candidates. Selection narrows them to the three or four pages actually cited, and everything observable about that step says it favors pages that behave like good answers: a claim stated plainly near the top, structure a machine can parse, a visible date, an identifiable author or organization. Pages that answer the question a reader asked, rather than pages that gesture at a topic.
None of that is exotic. It is the same set of properties search engines have rewarded for years — which is why sites with strong technical and content fundamentals tend to show up disproportionately in assistant citations without having done anything "for AI."
Where intuition fails
The common mental model — the AI knows about us from training, so it will mention us — breaks in both directions.
A model can know a brand thoroughly from training data and still cite a competitor, because citations follow retrieval, and retrieval surfaced the competitor's page. Training knowledge does not put your URL in the footnotes. Only the index does.
The reverse mistake is quieter. Some teams blocked GPTBot in 2023 or 2024 to stay out of training data, assumed that settled the matter, and never revisited. Blocking GPTBot does nothing to ChatGPT search visibility — OpenAI's crawler documentation is explicit that GPTBot is not used for search. Blocking OAI-SearchBot, or letting a CDN block it for you, is what removes you from the system that produces citations. Two robots.txt lines, opposite commercial consequences, routinely confused for each other.
There is also a third category of agent most robots policies never anticipated: user-triggered fetchers like ChatGPT-User and Perplexity-User, which visit a page live because a human asked about it. Both companies note these may not honor robots.txt, on the argument that the request is user-initiated. Control over this layer is thinner than most access policies assume.
Citations are not a reward for being known. They are a reward for being retrievable.
The practical question
When a brand is absent from assistant answers, the cause sits in one of three layers: access (crawlers cannot reach you), indexing (you are not in the retrieval corpus), or selection (you are retrievable but never the best answer). Each layer leaves different evidence and calls for a different fix. Diagnosing which one is failing — before optimizing anything — is most of the work.