Web Sitemap-find
Locate a website sitemap before crawling by reading sitemap declarations from robots.txt and, when none are declared, checking only the conventional same-origin sitemap.xml and sitemap_index.xml locations. Returns verified sitemap discovery evidence without crawling sitemap entries or site pages.
Forest exposes this capability as a deterministic machine-readable API for software agents. Discovery is free; execution uses the canonical paid endpoint below. The live x402 challenge is the source of truth for payment terms.
$0.002 / callPOST /sitemap/findx402 · Base
Typical agent tasks
This capability is intended for bounded machine decisions such as:
- locate the XML sitemap or sitemap index for a public website
- discover sitemap URLs from robots.txt before starting a crawler
- check conventional sitemap.xml and sitemap_index.xml locations when robots has no sitemap declaration
- find a verified website sitemap without crawling pages or sitemap entries
What Forest guarantees
- at most three bounded public HTTP requests: robots.txt plus at most two fixed same-origin sitemap probes
- conventional sitemap probes never follow cross-origin redirects
- no sitemap URL entries or site pages are crawled
Boundaries
- complete sitemap discovery when a site uses undisclosed nonstandard locations
- freshness or completeness of URLs listed by a discovered sitemap
- authorization to crawl discovered content
Machine contract
The request and response schemas are published in the canonical OpenAPI document. This page shows a representative input from the same production manifest.
Input example
{
"url": "https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/fixtures/page"
}
Canonical paid endpoint
https://http--forest-gas-station-mainnet--lcjl27p8lmjs.code.run/sitemap/find
Related capabilities
- Web Article-extract — Extract the main article/content body from one bounded public static HTML page as clean text and conservative Markdown. Forest deterministically selects article-like content, strips common navigation/footer/form/aside/consent chrome and returns bounded metadata. No JavaScript rendering, link following, authentication or model inference.
- Web Page-downloads — Extract PDF/CSV/XLS/XLSX/ZIP/JSON/XML links from static public HTML by suffix or download filename. Resolve relative/base links and deduplicate. Limits: 500 links, 8s, 5 redirects, 2 MiB. No JavaScript or file downloads; existence, MIME, size and file safety are unverified. Empty results are valid. 0.003 Base Sepolia test USDC per success; invalid inputs/errors are not settled.
- Web Page-forms — Extract bounded static HTML form structure without submitting anything. Forest returns field names/types/labels/constraints and submit-control structure, but never current values, textarea contents, hidden tokens, passwords, cookies or authentication material.
- Web Page-links — Extract a bounded normalized set of public HTTP(S) anchor links from one static HTML page. Forest resolves relative and base URLs, deduplicates by final absolute URL and classifies returned links as same-origin or cross-origin. One page fetch only; no linked URL is requested or crawled.
- Web Page-structured — Extract bounded generic metadata and normalized JSON-LD entities from one static public HTML page. Forest returns only compact fields and key/type summaries, never arbitrary raw JSON-LD; JavaScript rendering and domain-specific business interpretation are excluded.
- Web Page-structured-changed — Detect meaningful changes in normalized static page metadata and JSON-LD with a deterministic semantic hash. Transport noise and malformed JSON-LD counts are excluded; truncated structured output reports unknown when the observed prefix is unchanged.