A clearer view of AI.
THE INDEPENDENT AI.

AI SEARCH TIP · 21 OCTOBER 2026

Check that AI search engines are actually allowed to read your site

1 source · Part of the 21 October 2026 edition

Before working on content for AI answers, check that the engines can read your site at all. OpenAI uses separate crawlers: OAI-SearchBot decides whether your pages can appear in ChatGPT search answers, while GPTBot is for training, and you can allow the first while blocking the second. If OAI-SearchBot is blocked in your robots.txt, or by your host's or CDN's bot or firewall settings, your pages won't appear in ChatGPT's search answers, apart from possible plain navigational links. So open yoursite.com/robots.txt, look for 'Disallow' rules under OAI-SearchBot, GPTBot or 'User-agent: *', and if you use a security or CDN service, check its AI-bot settings too. Changes take about 24 hours to register with OpenAI. Treg's AEO workflow above runs this same 'are AI bots allowed in?' check as part of its audit.

That's it for today.

Sources

How we knowCompany figures are labelled as their own claims; anything we couldn't verify is labelled as a report or a claim.

MORE FROM 21 OCTOBER 2026

  1. TOP STORIESMistral's Voxtral 3: voice agents that cope with interruptions and switch languages
  2. TOP STORIESClaude ran most of a protein-design project by itself
  3. TOP STORIESAnthropic's IPO filing, as reported: revenue up fast, costs up faster
  4. BIG LAB UPDATESChatGPT team tasks: recurring work your whole team can check
  5. BIG LAB UPDATESOpenAI's API can now keep your instructions cached for 12 hours
  6. BIG LAB UPDATESOpenAI buys Orchestra Bio, whose agents run drug-development operations
The whole day ↗