Reference

Crawler and access reference

How SuperSchema reads your pages: the exact bot user agent, what the crawler does and does not do, and how to let it through robots.txt, Cloudflare, and other firewalls so scans, verification, and monitoring all work.

View as plain text Updated 2026-07-07

Scanning, verification, and monitoring all work by reading your public pages the way a search engine or AI bot would. If your site blocks automated readers, SuperSchema cannot see your content and those features will fail. This page tells you exactly how the crawler identifies itself and how to let it through cleanly.

The SuperSchema crawler user agent #

SuperSchema identifies itself as SuperSchemaBot. Its full user agent string is:

text 
Mozilla/5.0 (compatible; SuperSchemaBot/1.0; +https://superschema.ai/bot)
Note The token to look for in your logs or firewall rules is SuperSchemaBot. If you have older notes referring to a "SuperSchemaCrawler" user agent, that name is not used. The bot is SuperSchemaBot.

What the crawler does #

  • It is read-only. It requests your public pages and never logs in, submits forms, changes content, or touches your settings.
  • It respects robots.txt. If your rules disallow automated crawling of a path, SuperSchema will not read it.
  • It only reads pages you point it at or that it discovers from your sitemap and links, starting from your public site.
  • The same bot is used for the free Site Scan, for verifying that your schema is live, and for ongoing monitoring. Allow it once and all three work.

Allowing SuperSchema through robots.txt #

The most reliable rule is to not block automated crawling of your public pages in the first place. A site-wide Disallow is the most common reason a scan or verification cannot read your content. If you keep robots.txt permissive for your public pages, SuperSchema can read them.

If you want to be explicit, you can add an allow rule for the bot:

text 
User-agent: SuperSchemaBot
Allow: /
Warning Do not rely on a single robots.txt token as your only lever. Because of a known internal inconsistency, the site-scan check currently also keys on the legacy token AEO-Schema-Generator, while verification and monitoring key on SuperSchemaBot. The robust fix is to avoid a broad site-wide Disallow for your public pages, so no matter which token is evaluated, your pages remain readable.
Tip Make sure your public pages are not behind a blanket "Disallow: /" for "User-agent: *". A global disallow blocks SuperSchema along with every other crawler.

Firewalls, Cloudflare, and WAFs #

robots.txt only governs well-behaved crawlers. A firewall, CDN, or bot-protection service (Cloudflare, a WAF, or a security plugin) can block SuperSchema before it ever sees robots.txt. When that happens, a page that loads fine in your browser still fails to scan or verify.

  1. Identify where automated traffic is being challenged or blocked: Cloudflare Bot Fight Mode, a WAF managed rule, or a security plugin.
  2. Add an allow rule that permits the user agent SuperSchemaBot for your public pages.
  3. If your firewall matches on the full user agent string, allow "Mozilla/5.0 (compatible; SuperSchemaBot/1.0; +https://superschema.ai/bot)".
  4. Re-run the scan or verification after the rule is in place.
Tip When you allowlist by user agent in a firewall or WAF, allow SuperSchemaBot. That is the token the bot presents on every request.
Note If you cannot adjust your firewall or bot-protection rules, you can still deploy schema by copying it into your page head yourself and confirming it with an external tool such as Google’s Rich Results Test, rather than relying on automated verification.

When a scan or verification is blocked #

If SuperSchema cannot read a page, the fix is almost always one of the two access layers above. Work through them in order:

  • Confirm the page is public and loads without a login.
  • Confirm robots.txt does not disallow the path, and that there is no site-wide Disallow blocking crawlers.
  • Confirm your firewall, CDN, or bot-protection is not challenging or blocking SuperSchemaBot.
  • Re-run the scan or verification once access is open.
Tip For the full list of scan errors and how to fix each, see scan troubleshooting. For verification failures specifically, see verification troubleshooting.
Related
Scan troubleshooting A reference for every scan error SuperSchema can show, what each one usually means, and how to fix it. Most failures come down to bot protection, a firewall, or a slow response, and the fix is almost always allowing SuperSchemaBot or trying another public page. Verification failed If SuperSchema cannot confirm your schema on the live page, it is almost always one of a handful of causes: the code is not published yet, a wrong URL, a blocked crawler, a redirect, or an older package on the page. Work through them in order using the exact status message SuperSchema showed you. How verification works After you deploy, SuperSchema re-crawls your live page, finds its own install block, and confirms the exact schema package you generated is really published. It matches by version id first, then by a content hash, then by schema type, and reports one clear status so you know whether you are done. Free scan Enter your domain and SuperSchema samples up to 10 key pages, measures how ready they are for AI and search, and hands you a SuperSchema Score. It is free, needs no account, and never changes your live site.

Still have questions?

Can't find what you're looking for? Our support team is here to help.

Contact Support