# Crawler and access reference > How SuperSchema reads your pages: the exact bot user agent, what the crawler does and does not do, and how to let it through robots.txt, Cloudflare, and other firewalls so scans, verification, and monitoring all work. *Category: Reference · Last updated: 2026-07-07* --- Scanning, verification, and monitoring all work by reading your public pages the way a search engine or AI bot would. If your site blocks automated readers, SuperSchema cannot see your content and those features will fail. This page tells you exactly how the crawler identifies itself and how to let it through cleanly. ## The SuperSchema crawler user agent SuperSchema identifies itself as SuperSchemaBot. Its full user agent string is: ```text Mozilla/5.0 (compatible; SuperSchemaBot/1.0; +https://superschema.ai/bot) ``` > **Note:** The token to look for in your logs or firewall rules is SuperSchemaBot. If you have older notes referring to a "SuperSchemaCrawler" user agent, that name is not used. The bot is SuperSchemaBot. ## What the crawler does - It is read-only. It requests your public pages and never logs in, submits forms, changes content, or touches your settings. - It respects robots.txt. If your rules disallow automated crawling of a path, SuperSchema will not read it. - It only reads pages you point it at or that it discovers from your sitemap and links, starting from your public site. - The same bot is used for the free Site Scan, for verifying that your schema is live, and for ongoing monitoring. Allow it once and all three work. ## Allowing SuperSchema through robots.txt The most reliable rule is to not block automated crawling of your public pages in the first place. A site-wide Disallow is the most common reason a scan or verification cannot read your content. If you keep robots.txt permissive for your public pages, SuperSchema can read them. If you want to be explicit, you can add an allow rule for the bot: ```text User-agent: SuperSchemaBot Allow: / ``` > **Warning:** Do not rely on a single robots.txt token as your only lever. Because of a known internal inconsistency, the site-scan check currently also keys on the legacy token AEO-Schema-Generator, while verification and monitoring key on SuperSchemaBot. The robust fix is to avoid a broad site-wide Disallow for your public pages, so no matter which token is evaluated, your pages remain readable. > **Tip:** Make sure your public pages are not behind a blanket "Disallow: /" for "User-agent: *". A global disallow blocks SuperSchema along with every other crawler. ## Firewalls, Cloudflare, and WAFs robots.txt only governs well-behaved crawlers. A firewall, CDN, or bot-protection service (Cloudflare, a WAF, or a security plugin) can block SuperSchema before it ever sees robots.txt. When that happens, a page that loads fine in your browser still fails to scan or verify. 1. Identify where automated traffic is being challenged or blocked: Cloudflare Bot Fight Mode, a WAF managed rule, or a security plugin. 2. Add an allow rule that permits the user agent SuperSchemaBot for your public pages. 3. If your firewall matches on the full user agent string, allow "Mozilla/5.0 (compatible; SuperSchemaBot/1.0; +https://superschema.ai/bot)". 4. Re-run the scan or verification after the rule is in place. > **Tip:** When you allowlist by user agent in a firewall or WAF, allow SuperSchemaBot. That is the token the bot presents on every request. > **Note:** If you cannot adjust your firewall or bot-protection rules, you can still deploy schema by copying it into your page head yourself and confirming it with an external tool such as Google’s Rich Results Test, rather than relying on automated verification. ## When a scan or verification is blocked If SuperSchema cannot read a page, the fix is almost always one of the two access layers above. Work through them in order: - Confirm the page is public and loads without a login. - Confirm robots.txt does not disallow the path, and that there is no site-wide Disallow blocking crawlers. - Confirm your firewall, CDN, or bot-protection is not challenging or blocking SuperSchemaBot. - Re-run the scan or verification once access is open. > **Tip:** For the full list of scan errors and how to fix each, see scan troubleshooting. For verification failures specifically, see verification troubleshooting. ## Questions this answers - What user agent does SuperSchema use? - Is the SuperSchema crawler read-only, or does it change my site? - How do I let the SuperSchema crawler through robots.txt? - My site is behind Cloudflare or a firewall. How do I let SuperSchema in? - How do I allowlist SuperSchema in a WAF? - Does the same bot handle scanning, verification, and monitoring? - Why was my scan or verification blocked? - Does SuperSchema respect robots.txt? ## Related - https://superschema.ai/docs/scan-troubleshooting - https://superschema.ai/docs/verification-troubleshooting - https://superschema.ai/docs/how-verification-works - https://superschema.ai/docs/running-a-free-scan