Robots Context
Interpret the robots.txt crawling rules that apply to a specific public URL and user-agent before an automated fetch or crawl. Returns allow/disallow decision evidence, matched rules, crawl delay and sitemap hints from one bounded robots.txt read. This is protocol guidance, not authorization or an anti-bot bypass.
Forest exposes this capability as a deterministic machine-readable API for software agents. Discovery is free; execution uses the canonical paid endpoint below. The live x402 challenge is the source of truth for payment terms.
Typical agent tasks
This capability is intended for bounded machine decisions such as:
- check whether robots.txt allows a crawler or agent to fetch a particular URL
- interpret allow and disallow rules for a user agent before crawling a site
- read matched robots rules crawl delay and sitemap hints for a target path
- preflight a web crawl against the site robots policy without fetching pages
What Forest guarantees
- one bounded same-origin robots.txt fetch with deterministic rule selection
- no crawling beyond robots.txt and no JavaScript rendering
Boundaries
- authorization to access content
- legal or terms-of-service compliance
- site behavior beyond the observed robots.txt response
Machine contract
The request and response schemas are published in the canonical OpenAPI document. This page shows a representative input from the same production manifest.
Input example
Canonical paid endpoint
Related capabilities
- Web Sitemap-find — Locate a website sitemap before crawling by reading sitemap declarations from robots.txt and, when none are declared, checking only the conventional same-origin sitemap.xml and sitemap_index.xml locations. Returns verified sitemap discovery evidence without crawling sitemap entries or site pages.
- Web Site-changed-since — Detect bounded changes across one static page surface using one HTML fetch plus at most one robots.txt fetch. Forest hashes HTTP identity, page metadata, normalized JSON-LD, downloadable links, form schema and robots context without storing snapshots, rendering JavaScript or submitting anything.
- Web Article-extract — Extract the main article/content body from one bounded public static HTML page as clean text and conservative Markdown. Forest deterministically selects article-like content, strips common navigation/footer/form/aside/consent chrome and returns bounded metadata. No JavaScript rendering, link following, authentication or model inference.
- Web Page-downloads — Extract PDF/CSV/XLS/XLSX/ZIP/JSON/XML links from static public HTML by suffix or download filename. Resolve relative/base links and deduplicate. Limits: 500 links, 8s, 5 redirects, 2 MiB. No JavaScript or file downloads; existence, MIME, size and file safety are unverified. Empty results are valid. 0.003 Base Sepolia test USDC per success; invalid inputs/errors are not settled.
- Web Page-forms — Extract bounded static HTML form structure without submitting anything. Forest returns field names/types/labels/constraints and submit-control structure, but never current values, textarea contents, hidden tokens, passwords, cookies or authentication material.
- Web Page-links — Extract a bounded normalized set of public HTTP(S) anchor links from one static HTML page. Forest resolves relative and base URLs, deduplicates by final absolute URL and classifies returned links as same-origin or cross-origin. One page fetch only; no linked URL is requested or crawled.