RobotsGate

Ingested from the official MCP Registry

FastDrop pulled this entry from the official MCP Registry as dev.workers.mike-tusa.robotsgate/robotsgate and probed the endpoint. The maker did not submit it, so the outbound link is not endorsed and the description is not theirs.

If this is your server, claim the listing to edit it and take ownership. The registry index lists everything else still awaiting a probe.

Tools this endpoint exposes

Read directly from the endpoint on 16 Oct 2026. Not supplied by the maker.

• list_crawlers

  Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a l

• generate_robots

  Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps a

• validate_robots

  Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same eng

• check_site

  Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, whic

Is this tool yours?

Claim this listing and we’ll verify ownership before handing it over.