scrape-web
starsforks
How to install scrape-web?
omnigentx/jarvis/scrape-web
$npx -y skills add omnigentx/jarvis --skill scrape-webAccess and extract content from websites. Use when you need to read URL content, scrape data, or fetch information from blocked pages. Priority: serpapi → ScraplingServer get → fetch → stealthy_fetch.
SKILL.md
| 1 | # Web Content Retrieval |
| 2 | |
| 3 | <tool_priority> |
| 4 | Escalation chain — follow this order strictly: |
| 5 | 1. serpapi (search/get) → Fast, stable, for most search queries. Always use `gl=vn`, `hl=vi`. |
| 6 | 2. ScraplingServer get → When accessing a specific URL directly |
| 7 | 3. ScraplingServer fetch → If get returns 403/empty |
| 8 | 4. ScraplingServer stealthy_fetch → If fetch is still blocked |
| 9 | </tool_priority> |
| 10 | |
| 11 | <rule> |
| 12 | DO NOT use ScraplingServer for search queries. Only use it when you need direct URL access that serpapi cannot provide. |
| 13 | </rule> |
| 14 | |
| 15 | ### When to Use What |
| 16 | |
| 17 | | Scenario | Tool | |
| 18 | |----------|------| |
| 19 | | General information search | `serpapi` → search | |
| 20 | | Google/Bing results | `serpapi` | |
| 21 | | Access specific user-provided URL | ScraplingServer `get` | |
| 22 | | URL returns 403/blocked | ScraplingServer `fetch` | |
| 23 | | Cloudflare/anti-bot protection | ScraplingServer `stealthy_fetch` | |
| 24 | | JS-rendered SPA | ScraplingServer `fetch` + `network_idle: true` | |
| 25 | |
| 26 | --- |
| 27 | |
| 28 | ## ScraplingServer — Fallback Details |
| 29 | |
| 30 | ### Per-Site Recommendations |
| 31 | |
| 32 | | Site | Tool | |
| 33 | |------|------| |
| 34 | | LinkedIn, Glassdoor, GitHub, Reddit | `get` | |
| 35 | | Telegram public channel | `get` + URL: `https://t.me/s/CHANNEL` | |
| 36 | | Twitter/X, Twitch, Facebook | `fetch` + `network_idle: true` | |
| 37 | | TikTok profile/page | `stealthy_fetch` + `network_idle: true`, `wait: 4000` | |
| 38 | | TikTok comments | `get` via internal API (see below) | |
| 39 | | Cloudflare-protected | `stealthy_fetch` + `solve_cloudflare: true` | |
| 40 | |
| 41 | ### Key Parameters |
| 42 | |
| 43 | - `extraction_type`: `"text"` · `"markdown"` · `"html"` |
| 44 | - `main_content_only: true` — strips nav/footer noise |
| 45 | - `css_selector` — target specific elements |
| 46 | - `network_idle: true` + `wait: 2000` — for lazy-loaded JS content |
| 47 | - `disable_resources: true` — faster (fetch/stealthy_fetch only) |
| 48 | |
| 49 | ### Bulk Fetching |
| 50 | |
| 51 | ``` |
| 52 | tool: bulk_get / bulk_fetch |
| 53 | urls: ["https://...", "https://..."] |
| 54 | ``` |
| 55 | |
| 56 | --- |
| 57 | |
| 58 | ## TikTok Comments API |
| 59 | |
| 60 | Comments don't render in DOM — call the internal API directly (no login needed): |
| 61 | |
| 62 | ``` |
| 63 | tool: get |
| 64 | url: https://www.tiktok.com/api/comment/list/ |
| 65 | params: { aweme_id: "VIDEO_ID", count: "10", cursor: "0", aid: "1988" } |
| 66 | headers: { Referer: "https://www.tiktok.com/@USER/video/VIDEO_ID" } |
| 67 | extraction_type: "html" |
| 68 | main_content_only: false |
| 69 | ``` |
| 70 | |
| 71 | Response JSON fields: `comments[].text`, `comments[].user.nickname`, `comments[].digg_count`, `total`, `has_more`. Paginate by incrementing `cursor` by 10. |
| 72 | |
| 73 | To get VIDEO_ID: `stealthy_fetch` the profile page as `html`, extract with regex `/@[\w.]+/video/(\d+)`. |
People who liked this also viewed
6 picksType
- agent-browserBrowser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web aSkillsJul 2026
- browser-act--- name: browser-act description: "Browser automation CLI for AI agents. NEVER run browser-act commands directly via Bash — always invoke this skill first. Use browser-act when a user mentions it bSkillsJul 2026
- playwright-cliAutomate browser interactions, test web pages and work with Playwright tests.SkillsJul 2026
- accessibilityAudit and improve web accessibility following WCAG 2.2 guidelines. Use when asked to "improve accessibility", "a11y audit", "WCAG compliance", "screen reader support", "keyboard navigation", or "make SkillsJun 2026
- anti-detect-browserLaunch and manage anti-detect browsers with unique real-device fingerprints for multi-account operations, web scraping, ad verification, and AI agent automation. Use when the user needs to run multiplSkillsMar 2026
- web-access所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。SkillsMay 2026
