xcrawl-scrape
starsforks
How to install xcrawl-scrape?
xcrawl-api/xcrawl-skills/xcrawl-scrape
$npx -y skills add xcrawl-api/xcrawl-skills --skill xcrawl-scrapeUse this skill for XCrawl scrape tasks, including single-URL fetch, format selection, sync or async execution, and JSON extraction with prompt or json_schema.
SKILL.md
| 1 | # XCrawl Scrape |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | This skill handles single-page extraction with XCrawl Scrape APIs. |
| 6 | Default behavior is raw passthrough: return upstream API response bodies as-is. |
| 7 | |
| 8 | ## Required Local Config |
| 9 | |
| 10 | Before using this skill, the user must create a local config file and write `XCRAWL_API_KEY` into it. |
| 11 | |
| 12 | Path: `~/.xcrawl/config.json` |
| 13 | |
| 14 | ```json |
| 15 | { |
| 16 | "XCRAWL_API_KEY": "<your_api_key>" |
| 17 | } |
| 18 | ``` |
| 19 | |
| 20 | Read API key from local config file only. Do not require global environment variables. |
| 21 | |
| 22 | ## Credits and Account Setup |
| 23 | |
| 24 | Using XCrawl APIs consumes credits. |
| 25 | If the user does not have an account or available credits, guide them to register at `https://dash.xcrawl.com/`. |
| 26 | After registration, they can activate the free `1000` credits plan before running requests. |
| 27 | |
| 28 | ## Tool Permission Policy |
| 29 | |
| 30 | Request runtime permissions for `curl` and `node` only. |
| 31 | Do not request Python, shell helper scripts, or other runtime permissions. |
| 32 | |
| 33 | ## API Surface |
| 34 | |
| 35 | - Start scrape: `POST /v1/scrape` |
| 36 | - Read async result: `GET /v1/scrape/{scrape_id}` |
| 37 | - Base URL: `https://run.xcrawl.com` |
| 38 | - Required header: `Authorization: Bearer <XCRAWL_API_KEY>` |
| 39 | |
| 40 | ## Usage Examples |
| 41 | |
| 42 | ### cURL (sync) |
| 43 | |
| 44 | ```bash |
| 45 | API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")" |
| 46 | |
| 47 | curl -sS -X POST "https://run.xcrawl.com/v1/scrape" \ |
| 48 | -H "Content-Type: application/json" \ |
| 49 | -H "Authorization: Bearer ${API_KEY}" \ |
| 50 | -d '{"url":"https://example.com","mode":"sync","output":{"formats":["markdown","links"]}}' |
| 51 | ``` |
| 52 | |
| 53 | ### cURL (async create + result) |
| 54 | |
| 55 | ```bash |
| 56 | API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")" |
| 57 | |
| 58 | CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/scrape" \ |
| 59 | -H "Content-Type: application/json" \ |
| 60 | -H "Authorization: Bearer ${API_KEY}" \ |
| 61 | -d '{"url":"https://example.com/product/1","mode":"async","output":{"formats":["json"]},"json":{"prompt":"Extract title and price."}}')" |
| 62 | |
| 63 | echo "$CREATE_RESP" |
| 64 | |
| 65 | SCRAPE_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.scrape_id||"")' "$CREATE_RESP")" |
| 66 | |
| 67 | curl -sS -X GET "https://run.xcrawl.com/v1/scrape/${SCRAPE_ID}" \ |
| 68 | -H "Authorization: Bearer ${API_KEY}" |
| 69 | ``` |
| 70 | |
| 71 | ### Node |
| 72 | |
| 73 | ```bash |
| 74 | node -e ' |
| 75 | const fs=require("fs"); |
| 76 | const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY; |
| 77 | const body={url:"https://example.com",mode:"sync",output:{formats:["markdown","json"]},json:{prompt:"Extract title and publish date."}}; |
| 78 | fetch("https://run.xcrawl.com/v1/scrape",{ |
| 79 | method:"POST", |
| 80 | headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`}, |
| 81 | body:JSON.stringify(body) |
| 82 | }).then(async r=>{console.log(await r.text());}); |
| 83 | ' |
| 84 | ``` |
| 85 | |
| 86 | ## Request Parameters |
| 87 | |
| 88 | ### Request endpoint and headers |
| 89 | |
| 90 | - Endpoint: `POST https://run.xcrawl.com/v1/scrape` |
| 91 | - Headers: |
| 92 | - `Content-Type: application/json` |
| 93 | - `Authorization: Bearer <api_key>` |
| 94 | |
| 95 | ### Request body: top-level fields |
| 96 | |
| 97 | | Field | Type | Required | Default | Description | |
| 98 | |---|---|---:|---|---| |
| 99 | | `url` | string | Yes | - | Target URL | |
| 100 | | `mode` | string | No | `sync` | `sync` or `async` | |
| 101 | | `proxy` | object | No | - | Proxy config | |
| 102 | | `request` | object | No | - | Request config | |
| 103 | | `js_render` | object | No | - | JS rendering config | |
| 104 | | `output` | object | No | - | Output config | |
| 105 | | `webhook` | object | No | - | Async webhook config (`mode=async`) | |
| 106 | |
| 107 | ### `proxy` |
| 108 | |
| 109 | | Field | Type | Required | Default | Description | |
| 110 | |---|---|---:|---|---| |
| 111 | | `location` | string | No | `US` | ISO-3166-1 alpha-2 country code, e.g. `US` / `JP` / `SG` | |
| 112 | | `sticky_session` | string | No | Auto-generated | Sticky session ID; same ID attempts to reuse exit | |
| 113 | |
| 114 | ### `request` |
| 115 | |
| 116 | | Field | Type | Required | Default | Description | |
| 117 | |---|---|---:|---|---| |
| 118 | | `locale` | string | No | `en-US,en;q=0.9` | Affects `Accept-Language` | |
| 119 | | `device` | string | No | `desktop` | `desktop` / `mobile`; affects UA and viewport | |
| 120 | | `cookies` | object map | No | - | Cookie key/value pairs | |
| 121 | | `headers` | object map | No | - | Header key/value pairs | |
| 122 | | `only_main_content` | boolean | No | `true` | Return main content only | |
| 123 | | `block_ads` | boolean | No | `true` | Attempt to block ad resources | |
| 124 | | `skip_tls_verification` | boolean | No | `true` | Skip TLS verification | |
| 125 | |
| 126 | ### `js_render` |
| 127 | |
| 128 | | Field | Type | Required | Default | Description | |
| 129 | |---|---|---:|---|---| |
| 130 | | `enabled` | |
People who liked this also viewed
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clustersSkillsJul 2026
- lark-base飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、workflow、角色权限；遇到 Base/多维表格/bitable 或 /base/ 链接时使用。文件导入转 lark-drive，认证/授权转 lark-shared。SkillsJul 2026
- lark-calendar飞书日历：管理日历日程和会议室。查看/搜索日程、创建/更新日程、管理参会人、查询忙闲和推荐时段、预定会议室。当用户需要查看日程安排、创建/修改会议、查询/预定会议室时使用。不负责：查询过去的视频会议记录（走 lark-vc）、待办任务（走 lark-task）。SkillsJul 2026
- shadcnManages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. AppliSkillsJul 2026
- xlsxUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.SkillsJul 2026
- firebase-data-connectAgent Skills for FirebaseSkillsJul 2026
