Skip to main content

The Nine Signals

Crawler access

Whether the crawlers and fetchers behind AI answers can reach and read key business facts.

SignalCrawler access
What it checksRobots rules, WAF behavior, renderability, sitemaps, canonicals, and bot logs
MovesMentioned, Cited
Fix cardPages are blocked or hard to render
VerifySynthetic fetch and re-run check
At risk
Needs work
Strong
04070100

What it measures​

Crawler access asks whether relevant crawlers can reach the page, render the important business facts, discover the page consistently, and keep it fresh. It covers search crawlers such as Googlebot and Bingbot, plus assistant-specific crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Applebot where relevant.

This signal is foundational, not sufficient. A crawlable page can still lose on authority or answerability, but a blocked page cannot reliably be cited.

Why it moves your EAC​

If an assistant cannot fetch or render the page, it cannot use the page as evidence. Bot challenges, accidental disallow rules, blocked render assets, client-side-only content, login walls, and stale sitemaps all reduce citation opportunities.

Crawler access also affects freshness. A corrected page does not help if the relevant systems cannot discover the change.

What the agent sees​

Weak read

The page returns a bot challenge, raw HTML lacks the business facts, and the booking content appears only after JavaScript interaction.

Strong read

The page returns a clean 200 response, key facts appear in initial HTML, assets render, sitemaps list canonical URLs, and bot logs confirm successful crawler visits.

How to improve​

  1. Audit robots.txt and meta robots. Allow public location, service, and profile pages to be crawled and indexed.
  2. Allow relevant search crawlers. Do not block Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, or Applebot unless you intentionally want to reduce visibility.
  3. Publish facts HTML-first. Name, address, phone, hours, services, and action links should be present in initial HTML.
  4. Keep sitemaps and canonicals clean. Submit canonical pages and use meaningful lastmod only for real updates.
  5. Check WAF and CDN rules. Verified crawlers should not be forced through CAPTCHA or 403 challenges.

How to verify​

Can crawlers read the fixed page?

Run synthetic fetches, inspect rendered HTML, check bot logs, then re-scan the assistant answer.

Re-run check

Research basis: docs/research/deep-research-report(11).md