xcrawl-map
starsforks
How to install xcrawl-map?
xcrawl-api/xcrawl-skills/xcrawl-map
$npx -y skills add xcrawl-api/xcrawl-skills --skill xcrawl-mapUse this skill for XCrawl map tasks, including site URL discovery, regex filtering, scope estimation, and crawl planning before full-site crawling.
SKILL.md
| 1 | # XCrawl Map |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | This skill uses XCrawl Map API to discover URLs for a site. |
| 6 | Default behavior is raw passthrough: return upstream API response bodies as-is. |
| 7 | |
| 8 | ## Required Local Config |
| 9 | |
| 10 | Before using this skill, the user must create a local config file and write `XCRAWL_API_KEY` into it. |
| 11 | |
| 12 | Path: `~/.xcrawl/config.json` |
| 13 | |
| 14 | ```json |
| 15 | { |
| 16 | "XCRAWL_API_KEY": "<your_api_key>" |
| 17 | } |
| 18 | ``` |
| 19 | |
| 20 | Read API key from local config file only. Do not require global environment variables. |
| 21 | |
| 22 | ## Credits and Account Setup |
| 23 | |
| 24 | Using XCrawl APIs consumes credits. |
| 25 | If the user does not have an account or available credits, guide them to register at `https://dash.xcrawl.com/`. |
| 26 | After registration, they can activate the free `1000` credits plan before running requests. |
| 27 | |
| 28 | ## Tool Permission Policy |
| 29 | |
| 30 | Request runtime permissions for `curl` and `node` only. |
| 31 | Do not request Python, shell helper scripts, or other runtime permissions. |
| 32 | |
| 33 | ## API Surface |
| 34 | |
| 35 | - Start map task: `POST /v1/map` |
| 36 | - Base URL: `https://run.xcrawl.com` |
| 37 | - Required header: `Authorization: Bearer <XCRAWL_API_KEY>` |
| 38 | |
| 39 | ## Usage Examples |
| 40 | |
| 41 | ### cURL |
| 42 | |
| 43 | ```bash |
| 44 | API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")" |
| 45 | |
| 46 | curl -sS -X POST "https://run.xcrawl.com/v1/map" \ |
| 47 | -H "Content-Type: application/json" \ |
| 48 | -H "Authorization: Bearer ${API_KEY}" \ |
| 49 | -d '{"url":"https://example.com","filter":"/docs/.*","limit":2000,"include_subdomains":true,"ignore_query_parameters":false}' |
| 50 | ``` |
| 51 | |
| 52 | ### Node |
| 53 | |
| 54 | ```bash |
| 55 | node -e ' |
| 56 | const fs=require("fs"); |
| 57 | const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY; |
| 58 | const body={url:"https://example.com",filter:"/docs/.*",limit:3000,include_subdomains:true,ignore_query_parameters:false}; |
| 59 | fetch("https://run.xcrawl.com/v1/map",{ |
| 60 | method:"POST", |
| 61 | headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`}, |
| 62 | body:JSON.stringify(body) |
| 63 | }).then(async r=>{console.log(await r.text());}); |
| 64 | ' |
| 65 | ``` |
| 66 | |
| 67 | ## Request Parameters |
| 68 | |
| 69 | ### Request endpoint and headers |
| 70 | |
| 71 | - Endpoint: `POST https://run.xcrawl.com/v1/map` |
| 72 | - Headers: |
| 73 | - `Content-Type: application/json` |
| 74 | - `Authorization: Bearer <api_key>` |
| 75 | |
| 76 | ### Request body: top-level fields |
| 77 | |
| 78 | | Field | Type | Required | Default | Description | |
| 79 | |---|---|---:|---|---| |
| 80 | | `url` | string | Yes | - | Site entry URL | |
| 81 | | `filter` | string | No | - | Regex filter for URLs | |
| 82 | | `limit` | integer | No | `5000` | Max URLs (up to `100000`) | |
| 83 | | `include_subdomains` | boolean | No | `true` | Include subdomains | |
| 84 | | `ignore_query_parameters` | boolean | No | `true` | Ignore URLs with query parameters | |
| 85 | |
| 86 | ## Response Parameters |
| 87 | |
| 88 | | Field | Type | Description | |
| 89 | |---|---|---| |
| 90 | | `map_id` | string | Task ID | |
| 91 | | `endpoint` | string | Always `map` | |
| 92 | | `version` | string | Version | |
| 93 | | `status` | string | `completed` | |
| 94 | | `url` | string | Entry URL | |
| 95 | | `data` | object | URL list data | |
| 96 | | `started_at` | string | Start time (ISO 8601) | |
| 97 | | `ended_at` | string | End time (ISO 8601) | |
| 98 | | `total_credits_used` | integer | Total credits used | |
| 99 | |
| 100 | `data` fields: |
| 101 | |
| 102 | - `links`: URL list |
| 103 | - `total_links`: URL count |
| 104 | - `credits_used`: credits used |
| 105 | - `credits_detail`: credit breakdown |
| 106 | |
| 107 | ## Workflow |
| 108 | |
| 109 | 1. Restate mapping objective. |
| 110 | - Discovery only, selective crawl planning, or structure analysis. |
| 111 | |
| 112 | 2. Build and execute `POST /v1/map`. |
| 113 | - Keep filters explicit and reproducible. |
| 114 | |
| 115 | 3. Return raw API response directly. |
| 116 | - Do not synthesize URL-family summaries unless requested. |
| 117 | |
| 118 | ## Output Contract |
| 119 | |
| 120 | Return: |
| 121 | |
| 122 | - Endpoint used (`POST /v1/map`) |
| 123 | - `request_payload` used for the request |
| 124 | - Raw response body from map call |
| 125 | - Error details when request fails |
| 126 | |
| 127 | Do not generate summaries unless the user explicitly requests a summary. |
| 128 | |
| 129 | ## Guardrails |
| 130 | |
| 131 | - Do not claim full site coverage if `limit` is reached. |
| 132 | - Do not mix inferred URLs with returned URLs. |
| 133 | - Do not hardcode provider-specific tool schemas in core logic. |
People who liked this also viewed
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clustersSkillsJul 2026
- lark-base飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、workflow、角色权限；遇到 Base/多维表格/bitable 或 /base/ 链接时使用。文件导入转 lark-drive，认证/授权转 lark-shared。SkillsJul 2026
- lark-calendar飞书日历：管理日历日程和会议室。查看/搜索日程、创建/更新日程、管理参会人、查询忙闲和推荐时段、预定会议室。当用户需要查看日程安排、创建/修改会议、查询/预定会议室时使用。不负责：查询过去的视频会议记录（走 lark-vc）、待办任务（走 lark-task）。SkillsJul 2026
- shadcnManages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. AppliSkillsJul 2026
- xlsxUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.SkillsJul 2026
- firebase-data-connectAgent Skills for FirebaseSkillsJul 2026
