Robots.txt & AI Crawler Checker

Ingested from the official MCP Registry

FastDrop pulled this entry from the official MCP Registry as dev.workers.quietworklabs.robots-check/robots-check and probed the endpoint. The maker did not submit it, so the outbound link is not endorsed and the description is not theirs.

If this is your server, claim the listing to edit it and take ownership. The registry index lists everything else still awaiting a probe.

Tools this endpoint exposes

Read directly from the endpoint on 18 Oct 2026. Not supplied by the maker.

• robots_check

  Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ti

• ai_crawler_access

  AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot,

• sitemap_check

  Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, r

Is this tool yours?

Claim this listing and we’ll verify ownership before handing it over.