firecrawl-crawl
installsforks
How to install firecrawl-crawl?
firecrawl/cli/firecrawl-crawl
$npx -y skills add firecrawl/cli --skill firecrawl-crawlBulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extrac
SKILL.md
| 1 | # firecrawl crawl |
| 2 | |
| 3 | Bulk extract content from a website. Crawls pages following links up to a depth/limit. |
| 4 | |
| 5 | ## When to use |
| 6 | |
| 7 | - You need content from many pages on a site (e.g., all `/docs/`) |
| 8 | - You want to extract an entire site section |
| 9 | - Step 4 in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → **crawl** → interact |
| 10 | |
| 11 | ## Quick start |
| 12 | |
| 13 | ```bash |
| 14 | # Crawl a docs section |
| 15 | firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json |
| 16 | |
| 17 | # Full crawl with depth limit |
| 18 | firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json |
| 19 | |
| 20 | # Check status of a running crawl |
| 21 | firecrawl crawl <job-id> |
| 22 | ``` |
| 23 | |
| 24 | ## Options |
| 25 | |
| 26 | | Option | Description | |
| 27 | | ------------------------- | ------------------------------------------- | |
| 28 | | `--wait` | Wait for crawl to complete before returning | |
| 29 | | `--progress` | Show progress while waiting | |
| 30 | | `--limit <n>` | Max pages to crawl | |
| 31 | | `--max-depth <n>` | Max link depth to follow | |
| 32 | | `--include-paths <paths>` | Only crawl URLs matching these paths | |
| 33 | | `--exclude-paths <paths>` | Skip URLs matching these paths | |
| 34 | | `--delay <ms>` | Delay between requests | |
| 35 | | `--max-concurrency <n>` | Max parallel crawl workers | |
| 36 | | `--pretty` | Pretty print JSON output | |
| 37 | | `-o, --output <path>` | Output file path | |
| 38 | |
| 39 | ## Tips |
| 40 | |
| 41 | - Always use `--wait` when you need the results immediately. Without it, crawl returns a job ID for async polling. |
| 42 | - Use `--include-paths` to scope the crawl — don't crawl an entire site when you only need one section. |
| 43 | - Crawl consumes credits per page. Check `firecrawl credit-usage` before large crawls. |
| 44 | |
| 45 | ## See also |
| 46 | |
| 47 | - [firecrawl-scrape](../firecrawl-scrape/SKILL.md) — scrape individual pages |
| 48 | - [firecrawl-map](../firecrawl-map/SKILL.md) — discover URLs before deciding to crawl |
| 49 | - [firecrawl-download](../firecrawl-download/SKILL.md) — download site to local files (uses map + scrape) |
People who liked this also viewed
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clustersSkillsJul 2026
- lark-base飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、workflow、角色权限；遇到 Base/多维表格/bitable 或 /base/ 链接时使用。文件导入转 lark-drive，认证/授权转 lark-shared。SkillsJul 2026
- lark-calendar飞书日历：管理日历日程和会议室。查看/搜索日程、创建/更新日程、管理参会人、查询忙闲和推荐时段、预定会议室。当用户需要查看日程安排、创建/修改会议、查询/预定会议室时使用。不负责：查询过去的视频会议记录（走 lark-vc）、待办任务（走 lark-task）。SkillsJul 2026
- shadcnManages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. AppliSkillsJul 2026
- xlsxUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.SkillsJul 2026
- firebase-data-connectAgent Skills for FirebaseSkillsJul 2026
