The scrape call returns success but the markdown content is empty or a skeleton page — no error anywhere. Why?
The signature silent failure of rendering pipelines, with three root causes in order of probability. First: the target page hydrates content client-side after network idle, and the default wait was insufficient — the engine captured the HTML shell before React mounted; fix by passing a waitFor parameter or actions that wait for a selector. Second: you are self-hosting and the Playwright worker is dead — the API returns 200 from cache or a failed job payload while the worker queue silently backs up; check the worker logs, not the API logs. Third: the site serves a bot-detection interstitial (Cloudflare, DataDome) that renders as a challenge page — technically a successful scrape of the wrong document. The engineering fix: log the raw response length per call during your first week; any scrape under 500 characters on a content page is a suspect, not a result.
Firecrawl vs calling the Firecrawl REST API directly — what does MCP actually add?
Orchestration. Via REST you write the loop: map, then batch-scrape, then extract. Via MCP the agent IS the loop — it decides at runtime whether a page deserves a scrape, a crawl, or a schema extraction, chains the calls, and adapts when it hits a paywall or a redirect. You trade deterministic control for adaptive coverage.
Can it extract structured data without me writing selectors?
Yes — the extract tool accepts a natural-language schema (fields, types, descriptions) and uses an LLM internally to map page content onto it. It handles redesigns gracefully because it binds to semantics, not DOM paths. The cost: each extract call consumes both Firecrawl credits and LLM tokens.
How do I stop the agent from recursively crawling an entire domain?
Three layers: always pass limit and maxDepth on crawl calls, set includePaths and excludePaths to fence the URL graph, and use the cloud dashboard hard cap or a self-hosted queue concurrency limit as the final backstop. The crawl job is asynchronous — by the time you notice the quota drain, the job has already queued thousands of pages.