Does your harness ever encounter a javascript rendering or bot detection issue when using standard curl or webfetch attempts?
If so, you can tell your harness to use firecrawl
in our data enrichment endpoints to get the scraping done!
Example harness instructions:
**Blocked web pages — escalation order:** when WebFetch or `curl` returns a bot-detection or "checking your browser" page instead of content, try in this order, stopping at the first that works:
1. Firecrawl through our own endpoint (`POST /v1/data/api/firecrawl/scrape`, body `{"url": …, "formats": ["markdown"]}`, ~$0.015/call). It renders JavaScript and handles standard Cloudflare-style challenges, so it is the right second attempt for any JS-rendered or bot-gated page.
2. The site's own alternative front door: a `noscript.` subdomain or link in the challenge page, an onion/I2P mirror, an RSS/sitemap/`llms.txt`, or its public API.
3. A cached copy: `curl` the Wayback Machine (`
https://web.archive.org/web/2026id_/<url>`, pass `--compressed`); WebFetch cannot reach
http://archive.org.
Do not loop on the same fetcher with longer waits — one retry is enough evidence that it will not pass.
Head to our data enrichment page to learn more!
https://ppq.ai/api-docs#data-enrichment
