← capabilities

Robots Context

Interpret the robots.txt crawling rules that apply to a specific public URL and user-agent before an automated fetch or crawl. Returns allow/disallow decision evidence, matched rules, crawl delay and sitemap hints from one bounded robots.txt read. This is protocol guidance, not authorization or an anti-bot bypass.

Forest exposes this capability as a deterministic machine-readable API for software agents. Discovery is free; execution uses the canonical paid endpoint below. The live x402 challenge is the source of truth for payment terms.

$0.001 / callPOST /robots/contextx402 · Base

Typical agent tasks

This capability is intended for bounded machine decisions such as:

What Forest guarantees

Boundaries

Machine contract

The request and response schemas are published in the canonical OpenAPI document. This page shows a representative input from the same production manifest.

Input example

{ "url": "https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/fixtures/page", "user_agent": "ForestBot/1.0" }

Canonical paid endpoint

https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/robots/context

OpenAPI contract →

Related capabilities