Crawler access and measurement
How To Measure AI Visibility Without Pretending It Is A Ranking
A clear split between what a website scan can measure directly and what needs prompt tracking, analytics, and off-site research.
Updated 2026-04-25 · 6 min read
Website scans measure inputs, not guaranteed recommendations
A scan can confirm whether crawlers can access the page, whether the content is parseable, whether schema is valid, and whether the site exposes enough proof. It cannot guarantee that an AI system will cite or recommend the business.
- Treat scores as readiness indicators.
- Separate direct findings from rollout recommendations.
- Use rescans to confirm that website inputs improved.
Robots, sitemap, and AI crawler rules are direct checks
Robots.txt and sitemap.xml are foundational. AI crawler access is worth checking because some answer engines use their own crawlers. llms.txt can be useful guidance, but it is optional and not required for Google AI features.
- Allow public pages that should be discoverable.
- Reference the canonical sitemap.
- Review AI crawler blocks intentionally instead of inheriting accidental deny rules.
Citation measurement is a separate workstream
To measure actual answer-engine visibility, track a set of real prompts over time, record whether the brand appears, which sources are cited, and whether those visits convert.
- Use long-tail prompts from sales, support, reviews, and customer language.
- Track share of voice across multiple runs, not one answer.
- Pair prompt tracking with analytics for AI referral traffic and conversion quality.
Measure your site
See which signals are missing on your site.
Run the free Visibility scan to check crawl access, answer structure, schema, entity clarity, proof, and competitor gaps.
Run Free Scan