toolcloudflareweb-scrapingcrawlingapimarkdowncontent-extractionUpdated 2026-06-30

Cloudflare /crawl Endpoint

One API call crawls an entire site and returns clean content in HTML, Markdown, or plaintext - no scripts, no browser management, no scraping infrastructure.

View original tweet

What It Is

Cloudflare's /crawl endpoint is a managed crawling API built on top of their Browser Rendering product. A single POST request crawls an entire domain, following links and extracting content, then returns structured output in the format you specify. It handles JavaScript rendering, pagination, and link traversal internally - the caller gets clean content without managing headless browsers, proxy rotation, or rate limiting.

The endpoint supports HTML, Markdown, and plaintext output formats. For agent ingestion pipelines, the Markdown output is the most useful - it feeds directly into LLM context windows without preprocessing. Configuration options control crawl depth, URL filters, wildcard patterns, caching, and content selectors. Source: X/@CloudflareDev thread, 2026-03-10

The reviewed local artifact shows the concrete API shape: a POST to https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl with bearer token auth and a JSON body containing the target url. That makes this a Cloudflare-account API route, not a general anonymous crawl primitive. Source: local X image artifact, 2026-06-30

Robots and operator controls: Cloudflare emphasized /crawl respects robots.txt, self-identifies as a bot, cannot bypass Cloudflare bot detection or CAPTCHAs, and honors AI Crawl Control, Content Signals, and Pay-Per-Crawl — site owners retain control over how content is consumed. Source: X/@CloudflareDev thread, 2026-03-11

Relevance to Kevin's Stack

This fills a specific gap in the Security and Review Skills toolkit. Currently, the wiki's ingestion pipeline uses Jina Reader for single-page extraction and Firecrawl for multi-page crawls in cloud automations. Cloudflare /crawl provides a middle option: multi-page crawling without Firecrawl's per-page pricing, backed by Cloudflare's existing infrastructure (Workers + Browser Rendering). For source-compile automation runs that need to ingest documentation sites or multi-page references, this is a lower-cost alternative to Firecrawl.

It also complements Defuddle - Web Content Extraction (local single-page extraction) and Autobrowse (Browserbase Skills) (browser-based site learning). The hierarchy for multi-page crawling becomes: Firecrawl MCP for cloud automations → Cloudflare /crawl for bulk site ingestion → Jina Reader for single pages → defuddle as local fallback.

Do not use /crawl to bypass site controls. The source thread explicitly says it respects robots.txt, cannot bypass Cloudflare bot detection or CAPTCHAs, and self-identifies as a bot. For Kevin's wiki/source-compile pipeline, this is strongest for documentation sites and owned/allowed sources where Cloudflare account auth, API scopes, and crawl costs are acceptable.

Signal

  • 19,837 likes, 25,545 bookmarks on X - extremely high engagement, indicating strong developer demand for managed crawling APIs
  • Announcement from @CloudflareDev (official account), signaling production-grade reliability

Timeline

Related Pages

Security and Review SkillsAutobrowse (Browserbase Skills)Defuddle - Web Content Extraction

3 pages link here

Camofox BrowserToolsWeb Scraping Stealth (TLS Fingerprinting & Anti-Bot Evasion)ConceptsX Bookmarks: Dev Tools (Mar–Jun 2026)Tools