Skip to main content

Fix Playbook

Open up crawler access

Ensure search and assistant crawlers can reach, render, and refresh your public business facts.

Use this playbook when the report flags blocked crawlers, thin initial HTML, JavaScript-only facts, missing sitemaps, or bot-challenge problems.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /admin/
Allow: /assets/
Allow: /wp-content/uploads/
Sitemap: https://example.com/sitemap.xml

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

If you want to block training while preserving search visibility, use training-specific controls rather than blocking the search crawler.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Crawlability checklist​

  • Public location, service, and contact pages return 200 responses.
  • Key facts appear in initial HTML, not only after user interaction.
  • CSS and JS needed for rendering are not blocked.
  • Canonical URLs are stable and self-consistent.
  • XML sitemaps include the important canonical URLs.
  • WAF rules do not challenge verified crawlers.
  • Logs show successful visits from the relevant bots.

JavaScript risk test​

Fetch the raw HTML and answer these questions:

  1. Does the HTML include the business name?
  2. Does it include address, phone, hours, and service text?
  3. Does it include the booking or quote link?
  4. Does the page still make sense with JavaScript disabled?

If the answer is no, publish the critical business facts with SSR, SSG, or another HTML-first path.

How to verify​

Can bots read the same facts users see?

Run synthetic fetches, inspect bot logs, submit or refresh sitemaps, then re-scan the assistant answer.

Re-run check

Related: Crawler access, Freshness