The AI Crawler Guide: Configuring robots.txt for GPTBot, ClaudeBot & More
Which AI bots visit your site, what each one does, and how to allow AI search visibility while keeping control over model training.
Read articleThe most common technical and content gaps that stop AI assistants from recommending otherwise great websites, and a practical checklist to close them.

You rank on the first page of Google, your site loads fast and your content is solid. Yet when a prospect asks ChatGPT or Perplexity for a recommendation in your category, your brand never comes up. This is one of the most common problems we see, and it usually has a handful of fixable causes.
When an AI assistant answers a question with live web results, it runs a search, fetches a small set of pages and summarises what it finds. To be cited, your page has to clear three hurdles: the crawler must be allowed to fetch it, the model must be able to extract a clear answer from it, and the source must look trustworthy enough to recommend.
Many sites block AI crawlers without realising it: a blanket rule in robots.txt, a security plugin, or a CDN bot-protection setting. If OAI-SearchBot, PerplexityBot or Claude-SearchBot can’t reach your pages, those assistants simply can’t show them.
Language models look for passages that answer a question on their own. A 2,000-word page that only gets to the point in paragraph nine is hard to quote. Pages that lead with a clear, self-contained answer of roughly 40–60 words, then expand, are much easier to cite.
Structured data (JSON-LD) tells machines exactly who you are, what you sell and which questions you answer. Without it, a model has to infer all of this from layout and prose, and it may get it wrong or skip you entirely.
AI models build their picture of a brand from many sources: your site, review platforms, directories, news coverage and community discussions. If your name, services or locations differ from one source to the next, the model has less confidence recommending you.
robots.txt and your CDN settings.llms.txt file that points to your most important content.None of these steps require rebuilding your website. In most audits we find that a few targeted technical fixes and some content restructuring make the biggest difference.

Which AI bots visit your site, what each one does, and how to allow AI search visibility while keeping control over model training.
Read article
Where classic search optimization still matters, where answer engines need a different approach, and how to run both without doubling your workload.
Read article
A practical walkthrough of the emerging llms.txt convention, with a template you can adapt for your own site in under an hour.
Read articleGet your free AI readiness audit, or talk to our team about a full GEO & AEO strategy for your brand.