xcrawl-crawl
starsforks
How to install xcrawl-crawl?
xcrawl-api/xcrawl-skills/xcrawl-crawl
$npx -y skills add xcrawl-api/xcrawl-skills --skill xcrawl-crawlUse this skill for XCrawl crawl tasks, including bulk site crawling, crawler rule design, async status polling, and delivery of crawl output for downstream scrape and search workflows.
SKILL.md
| 1 | # XCrawl Crawl |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | This skill orchestrates full-site or scoped crawling with XCrawl Crawl APIs. |
| 6 | Default behavior is raw passthrough: return upstream API response bodies as-is. |
| 7 | |
| 8 | ## Required Local Config |
| 9 | |
| 10 | Before using this skill, the user must create a local config file and write `XCRAWL_API_KEY` into it. |
| 11 | |
| 12 | Path: `~/.xcrawl/config.json` |
| 13 | |
| 14 | ```json |
| 15 | { |
| 16 | "XCRAWL_API_KEY": "<your_api_key>" |
| 17 | } |
| 18 | ``` |
| 19 | |
| 20 | Read API key from local config file only. Do not require global environment variables. |
| 21 | |
| 22 | ## Credits and Account Setup |
| 23 | |
| 24 | Using XCrawl APIs consumes credits. |
| 25 | If the user does not have an account or available credits, guide them to register at `https://dash.xcrawl.com/`. |
| 26 | After registration, they can activate the free `1000` credits plan before running requests. |
| 27 | |
| 28 | ## Tool Permission Policy |
| 29 | |
| 30 | Request runtime permissions for `curl` and `node` only. |
| 31 | Do not request Python, shell helper scripts, or other runtime permissions. |
| 32 | |
| 33 | ## API Surface |
| 34 | |
| 35 | - Start crawl: `POST /v1/crawl` |
| 36 | - Read result: `GET /v1/crawl/{crawl_id}` |
| 37 | - Base URL: `https://run.xcrawl.com` |
| 38 | - Required header: `Authorization: Bearer <XCRAWL_API_KEY>` |
| 39 | |
| 40 | ## Usage Examples |
| 41 | |
| 42 | ### cURL (create + result) |
| 43 | |
| 44 | ```bash |
| 45 | API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")" |
| 46 | |
| 47 | CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/crawl" \ |
| 48 | -H "Content-Type: application/json" \ |
| 49 | -H "Authorization: Bearer ${API_KEY}" \ |
| 50 | -d '{"url":"https://example.com","crawler":{"limit":100,"max_depth":2},"output":{"formats":["markdown","links"]}}')" |
| 51 | |
| 52 | echo "$CREATE_RESP" |
| 53 | |
| 54 | CRAWL_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.crawl_id||"")' "$CREATE_RESP")" |
| 55 | |
| 56 | curl -sS -X GET "https://run.xcrawl.com/v1/crawl/${CRAWL_ID}" \ |
| 57 | -H "Authorization: Bearer ${API_KEY}" |
| 58 | ``` |
| 59 | |
| 60 | ### Node |
| 61 | |
| 62 | ```bash |
| 63 | node -e ' |
| 64 | const fs=require("fs"); |
| 65 | const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY; |
| 66 | const body={url:"https://example.com",crawler:{limit:300,max_depth:3,include:["/docs/.*"],exclude:["/blog/.*"]},request:{locale:"ja-JP"},output:{formats:["markdown","links","json"]}}; |
| 67 | fetch("https://run.xcrawl.com/v1/crawl",{ |
| 68 | method:"POST", |
| 69 | headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`}, |
| 70 | body:JSON.stringify(body) |
| 71 | }).then(async r=>{console.log(await r.text());}); |
| 72 | ' |
| 73 | ``` |
| 74 | |
| 75 | ## Request Parameters |
| 76 | |
| 77 | ### Request endpoint and headers |
| 78 | |
| 79 | - Endpoint: `POST https://run.xcrawl.com/v1/crawl` |
| 80 | - Headers: |
| 81 | - `Content-Type: application/json` |
| 82 | - `Authorization: Bearer <api_key>` |
| 83 | |
| 84 | ### Request body: top-level fields |
| 85 | |
| 86 | | Field | Type | Required | Default | Description | |
| 87 | |---|---|---:|---|---| |
| 88 | | `url` | string | Yes | - | Site entry URL | |
| 89 | | `crawler` | object | No | - | Crawler config | |
| 90 | | `proxy` | object | No | - | Proxy config | |
| 91 | | `request` | object | No | - | Request config | |
| 92 | | `js_render` | object | No | - | JS rendering config | |
| 93 | | `output` | object | No | - | Output config | |
| 94 | | `webhook` | object | No | - | Async callback config | |
| 95 | |
| 96 | ### `crawler` |
| 97 | |
| 98 | | Field | Type | Required | Default | Description | |
| 99 | |---|---|---:|---|---| |
| 100 | | `limit` | integer | No | `100` | Max pages | |
| 101 | | `include` | string[] | No | - | Include only matching URLs (regex supported) | |
| 102 | | `exclude` | string[] | No | - | Exclude matching URLs (regex supported) | |
| 103 | | `max_depth` | integer | No | `3` | Max depth from entry URL | |
| 104 | | `include_entire_domain` | boolean | No | `false` | Crawl full site instead of only subpaths | |
| 105 | | `include_subdomains` | boolean | No | `false` | Include subdomains | |
| 106 | | `include_external_links` | boolean | No | `false` | Include external links | |
| 107 | | `sitemaps` | boolean | No | `true` | Use site sitemap | |
| 108 | |
| 109 | ### `proxy` |
| 110 | |
| 111 | | Field | Type | Required | Default | Description | |
| 112 | |---|---|---:|---|---| |
| 113 | | `location` | string | No | `US` | ISO-3166-1 alpha-2 country code, e.g. `US` / `JP` / `SG` | |
| 114 | | `sticky_session` | string | No | Auto-generated | Sticky session ID; same ID attempts to reuse exit | |
| 115 | |
| 116 | ### `request` |
| 117 | |
| 118 | | Field | Type | Required | Default | Description | |
| 119 | |---|---|---:|---|---| |
| 120 | | `locale` | string | No | `en-US,en;q=0.9` | Affects `Accept-Language` | |
| 121 | | `device` | string | No | `desktop` | `desktop` / `mobile`; affects UA and viewport | |
| 122 | | `cookies` | object map | No | - | Cookie key/value pairs | |
| 123 | | `headers` | object map | No | - | Header key/value pairs | |
| 124 | | `only_main_content` | boolean | No | `true` | Return main content only | |
| 125 | | `block_ads` |
People who liked this also viewed
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clustersSkillsJul 2026
- lark-base飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、workflow、角色权限；遇到 Base/多维表格/bitable 或 /base/ 链接时使用。文件导入转 lark-drive，认证/授权转 lark-shared。SkillsJul 2026
- lark-calendar飞书日历：管理日历日程和会议室。查看/搜索日程、创建/更新日程、管理参会人、查询忙闲和推荐时段、预定会议室。当用户需要查看日程安排、创建/修改会议、查询/预定会议室时使用。不负责：查询过去的视频会议记录（走 lark-vc）、待办任务（走 lark-task）。SkillsJul 2026
- shadcnManages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. AppliSkillsJul 2026
- xlsxUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.SkillsJul 2026
- firebase-data-connectAgent Skills for FirebaseSkillsJul 2026
