Robots Context
Fetch and interpret one bounded public robots.txt for a target URL and user-agent token. Forest reports applicable rules, sitemap hints and a deterministic allowed/disallowed/unknown protocol observation; this is not authorization, legal advice or an anti-bot bypass.
$0.001 / callPOST /robots/contextx402 · Base
Use when
- can this agent path be crawled according to robots txt
- which robots rules apply to this URL
- find robots sitemap and crawl delay before fetching
Input example
{
"url": "https://example.com/docs/page",
"user_agent": "ForestBot/1.0"
}
Canonical paid endpoint
https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/robots/context