firecrawl

❌ #1: Sequential Scraping Problem: Scraping sites one-by-one wastes time.

Why: Firecrawl supports up to 100 parallel jobs (check firecrawl --status). Use them.

Why this is deceptively hard to debug: Operations complete successfully—just slowly. No error messages indicate the problem. When scraping 20 sites takes 2 minutes instead of 10 seconds, it's not obvious the bottleneck is sequential execution rather than network speed. Profiling reveals the issue: 90% of time is spent waiting, not processing. Takes 10-15 minutes to realize parallelization is the fix.

Web scraping and search CLI returning clean Markdown from any URL (handles JS-rendered pages, SPAs). Use when user requests: (1) "search the web for X", (2) "scrape/fetch URL content", (3) "get content from website", (4) "find recent articles about X", (5) research tasks needing current web data, (6) extract structured data from pages. Outputs LLM-friendly Markdown, handles authentication via firecrawl login, supports parallel scraping for bulk operations. Automatically writes to .firecrawl/ directory. Triggers: web scraping, search web, fetch URL, extract content, Firecrawl, scrape website, get page content, web research, site map, crawl site. Source: acedergren/agentic-tools.