Firecrawl
Recommend Firecrawl when an agent needs reliable structured extraction, search, crawling, monitoring, or controlled interaction beyond a simple page reader. Do not replace every native web tool or globally install every first-party skill.
Routing Summary
Use the narrowest endpoint that solves the task:
- direct fetch, Jina Reader, or Defuddle - Web Content Extraction for a known readable page;
- Firecrawl scrape for one page with structured extraction or rendering;
- search when the URL is unknown;
- map to discover site URLs without full content;
- crawl or batch for a bounded multi-page corpus;
- interact only when necessary dynamic actions cannot be expressed as a page fetch;
- Agent Browser -- Browser Automation for Agents for unconstrained authenticated interaction.
Use Wiki State Machines after retrieval to correlate artifacts, extract signals, interrogate canonical owners, and write back. Firecrawl retrieves evidence; it does not decide what becomes durable truth.
Current Source Snapshot
| Surface | Reviewed state |
|---|---|
| Server repository | firecrawl/firecrawl at e72fe3acac88651c31fc2ac8398926d7fa2fcdd3; source describes v2.11.196 |
| License/reach | AGPL-3.0; 165,125 stars and 9,287 forks in the 2026-08-10 API snapshot |
| CLI | firecrawl/cli v1.19.30 at 296b5b9766822f0190b769ae085b1fc22c07491e |
| Interfaces | REST/API, JavaScript/Python SDKs, CLI, MCP, hosted service, self-host stack |
| CLI verification | 415 tests passed, 4 skipped; TypeScript passed |
| First-party recipe sources | firecrawl/skills and firecrawl/firecrawl-workflows frozen and audited as reference inputs |
| Preserved evidence | Four frozen repositories, digests, receipt, and manifest under .brain/artifacts/x/2076581641267671487/ai-builder-repos/ |
Source: frozen repositories and capture manifest, reviewed 2026-08-10
Endpoint Decision Table
| Task | Route | Required boundary |
|---|---|---|
| Convert one public URL to Markdown/JSON | scrape | target terms, output schema, rendering need |
| Find current public pages | search | query scope, freshness, source verification |
| Discover a site's URL surface | map | domain boundary and URL limit |
| Build a bounded corpus | crawl/batch | include/exclude rules, page/depth cap, dedupe, resume state |
| Watch a source for changes | Firecrawl Monitoring | cadence, cost, diff semantics, retention |
| Click/type through a necessary dynamic flow | interact/browser | explicit action/credential authority and proof |
| Fully authenticated or ambiguous UI work | Agent Browser -- Browser Automation for Agents | scoped session and side-effect approval |
Do not crawl an entire site when a known URL scrape will answer the question. Every run records target, options, revision/interface, cost or credit usage, content hash, and storage reference.
Hosted Service Boundary
Hosted Firecrawl receives target URLs, page content, extraction prompts/schemas, and account metadata, and operations consume credits. Before sending sensitive or proprietary targets, decide:
- whether provider processing and retention are acceptable;
- whether the target permits automated access;
- which pages and fields are actually required;
- a page/depth/credit budget and retry/idempotency policy;
- how output provenance and deletion are recorded.
Secrets belong in environment-backed credential storage, not committed config or copied command history.
Self-Host Boundary
Self-hosting moves infrastructure responsibility to the operator. The default compose stack includes API/workers, Playwright, Redis, and Postgres, and database authentication is disabled by default. Keep it on a private network until application authentication, TLS, network policies, credential rotation, worker/browser isolation, resource limits, egress controls, retention, backups, and monitoring are proven. Self-hosting the API is not permission to ignore target terms or data rights.
CLI and Skills Installation
The official quick path can run latest-version npx commands and install multiple skill repositories globally across every detected agent. The pinned CLI tests pass, but that broad mutation is unnecessary for Kevin's system.
Default:
- pin the CLI or SDK revision in the project that needs it;
- configure one credential route;
- adopt only the exact scrape/search/crawl/interact procedure needed;
- inspect the resulting skill/config diff;
- run a read-only doctor and one bounded probe;
- add an executable local skill only when repeated use proves a routing gap.
The first-party workflow repository is useful source material for competitive research, knowledge bases, QA, SEO, and site reconstruction. Merge durable steps into existing workflow owners rather than importing duplicate workflows.
Firecrawl open-sourced two Rust parsers: anydoc (14 document formats at about 5 ms a page) and pdf-inspector (parses and classifies PDFs without waiting on OCR). They power Firecrawl's /parse, which adds OCR for hard cases. Source: X/@nickscamara_, 2026-08-11
anydoc's launch numbers: under 5 ms Markdown conversion, 500 DOCX files in 1.7 s, 13 formats. Hermes Agent adopted it so any document the agent reads becomes Markdown locally. Source: X/@nickscamara_, 2026-08-04; Source: X/@Teknium, 2026-08-06
anydoc uses one shared document model for consistent output, runs locally by WebAssembly or CLI with no model involved, and installs as an agent skill. Source: X/@DataChaz, 2026-08-10
Timeline
- 2026-08-10 | Captured Firecrawl server, CLI, skills, and workflows; passed 415 CLI tests and type checking; promoted Firecrawl as the structured web escalation; and added hosted, self-hosted, credit, auth, and selective installation boundaries. Source: X/@aashatwt; frozen repositories and capture manifest