The crawler decisions that quietly control your AI visibility

For twenty years robots.txt was a formality — three lines nobody revisited after launch. It is now the file where a company's AI visibility gets decided. Sometimes by someone who left the company years ago; sometimes by a vendor default nobody read.

The agents requesting your pages today fall into three categories with three different commercial meanings, and they are routinely confused for one another.

Three kinds of agent

• Training crawlers — GPTBot (OpenAI), Google-Extended, CCBot (Common Crawl). They collect content for model training. Blocking them is an intellectual-property position; by each vendor's own documentation, it does not remove you from search results or citations.
• Search crawlers — OAI-SearchBot (ChatGPT search), PerplexityBot, and Googlebot itself. These build the indexes assistants cite from. OpenAI is blunt about the stakes: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."
• User-triggered fetchers — ChatGPT-User, Perplexity-User. They fetch a page live because a person asked about it. Both vendors document that these may not honor robots.txt, since the visit is user-initiated. Policy at this layer is close to unenforceable.

The framework follows from the categories. Blocking training bots costs approximately nothing in visibility today and is a defensible rights position. Blocking search bots is a visibility decision with revenue attached. Writing rules for fetchers is mostly theater.

The block you didn't choose

The sharper problem: many sites block search crawlers without ever having decided to. On July 1, 2025, Cloudflare switched its default for newly onboarded domains to block AI crawlers, framing the change as a stand on content compensation. A reasonable position for publishers — and a silent visibility cliff for everyone who put a site behind Cloudflare afterward and never opened the bot-management settings.

The failure mode is invisible from inside. Content is good. robots.txt looks permissive. And the CDN returns 403s to the exact agents that produce citations. The content team ships, sees nothing appear in assistants, and concludes AI search does not matter for their category. Managed WAFs and bot-protection products from other vendors create the same effect with different defaults.

Your effective robots policy is whatever your infrastructure actually serves — not what the file says.

The ten-minute audit

• Request your key pages with each agent's actual user-agent string, from outside your own network, and record the status codes.
• Read the AI and bot sections of your CDN and WAF configuration — not just robots.txt.
• Decide each category deliberately: training, search, fetchers. Write the decision down, with a date.
• Re-run the check after any CDN migration or security-product change. Defaults move; yours may have moved with them.

We ran that first check on this site while preparing this note — every agent above came back with a 200. It took about four minutes, which is roughly the ratio of effort to consequence this entire layer has.