A website owner can decide whether to allow these bots access. For visibility in AI answers it is usually worth allowing them.
What it means in practice
AI crawlers are automated programs from companies building AI models and search engines that visit websites, e.g. GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended and PerplexityBot. Some collect data to train models, others fetch content for live answers.
Why it matters for your business
If you block these crawlers in robots.txt, your website may not appear in AI answers. For local businesses that want to be recommended by ChatGPT or Perplexity it usually pays to allow them.
Examples and good practice
- Check whether your robots.txt blocks AI crawlers.
- Distinguish training and search crawlers – you can decide separately.
- Keep your website fast so crawlers can fetch it easily.
- Publish content you want AI to quote.
How Novi handles it
On Novi websites AI search crawlers have access to the content, so your business can appear in answers from ChatGPT, Perplexity and Google AI.
