web-scraper
starsforks
How to install web-scraper?
shobcoder/shob/web-scraper
$npx -y skills add shobcoder/shob --skill web-scraperScrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
SKILL.md
| 1 | # Web Scraper |
| 2 | |
| 3 | ## Overview |
| 4 | Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction. |
| 5 | |
| 6 | ## When to Use |
| 7 | - Extract text content from web pages |
| 8 | - Crawl entire websites |
| 9 | - Collect structured data |
| 10 | - Research and gather information |
| 11 | - Monitor website changes |
| 12 | - Extract tables and lists |
| 13 | |
| 14 | ## Tools Available |
| 15 | |
| 16 | ### Content Extraction |
| 17 | ```javascript |
| 18 | // Use extract_content_from_websites for structured extraction |
| 19 | // Supports batch processing of multiple URLs |
| 20 | // Returns JSON format with extracted content |
| 21 | ``` |
| 22 | |
| 23 | ### Task Format |
| 24 | ```javascript |
| 25 | { |
| 26 | tasks: [ |
| 27 | { |
| 28 | url: "https://example.com", |
| 29 | prompt: "Extract specific information", |
| 30 | task_name: "optional_name" |
| 31 | } |
| 32 | ] |
| 33 | } |
| 34 | ``` |
| 35 | |
| 36 | ## Usage Patterns |
| 37 | |
| 38 | ### Simple Content Extraction |
| 39 | ```javascript |
| 40 | // Extract main content from a page |
| 41 | const result = await extract_content_from_websites({ |
| 42 | tasks: [{ |
| 43 | url: "https://news.example.com/article", |
| 44 | prompt: "Extract the title, author, date, and main content" |
| 45 | }] |
| 46 | }); |
| 47 | ``` |
| 48 | |
| 49 | ### Batch URL Processing |
| 50 | ```javascript |
| 51 | // Process multiple URLs in parallel |
| 52 | const urls = [ |
| 53 | "https://site.com/page1", |
| 54 | "https://site.com/page2", |
| 55 | "https://site.com/page3" |
| 56 | ]; |
| 57 | |
| 58 | const results = await extract_content_from_websites({ |
| 59 | tasks: urls.map((url, i) => ({ |
| 60 | url, |
| 61 | prompt: "Extract all product information, prices, and descriptions", |
| 62 | task_name: `product_${i}` |
| 63 | })) |
| 64 | }); |
| 65 | ``` |
| 66 | |
| 67 | ### Data Mining |
| 68 | ```javascript |
| 69 | // Extract structured data like prices, reviews, specifications |
| 70 | const data = await extract_content_from_websites({ |
| 71 | tasks: [{ |
| 72 | url: "https://ecommerce.example.com/products", |
| 73 | prompt: "Extract product name, price, rating, and availability for all products listed" |
| 74 | }] |
| 75 | }); |
| 76 | ``` |
| 77 | |
| 78 | ## Extraction Modes |
| 79 | |
| 80 | ### Auto Mode (Default) |
| 81 | - Attempts HTTP GET first |
| 82 | - Falls back to browser rendering for CSR pages |
| 83 | - Best for most websites |
| 84 | |
| 85 | ### Curl Only Mode |
| 86 | - Fast direct HTTP requests |
| 87 | - Best for static HTML pages |
| 88 | - May fail on JavaScript-heavy sites |
| 89 | |
| 90 | ### Browser Only Mode |
| 91 | - Full browser rendering |
| 92 | - Handles dynamic content |
| 93 | - Slower but more comprehensive |
| 94 | |
| 95 | ## Best Practices |
| 96 | 1. Start with simpler extraction before complex patterns |
| 97 | 2. Use specific prompts for targeted data |
| 98 | 3. Respect website terms of service |
| 99 | 4. Add delays between requests when scraping multiple pages |
| 100 | 5. Handle errors gracefully with try/catch |
| 101 | |
| 102 | ## Data Handling |
| 103 | - Returns JSON format for easy processing |
| 104 | - Handles batch operations efficiently |
| 105 | - Supports pagination when needed |
| 106 | - Maintains data structure in results |
People who liked this also viewed
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clustersSkillsJul 2026
- lark-base飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、workflow、角色权限；遇到 Base/多维表格/bitable 或 /base/ 链接时使用。文件导入转 lark-drive，认证/授权转 lark-shared。SkillsJul 2026
- lark-calendar飞书日历：管理日历日程和会议室。查看/搜索日程、创建/更新日程、管理参会人、查询忙闲和推荐时段、预定会议室。当用户需要查看日程安排、创建/修改会议、查询/预定会议室时使用。不负责：查询过去的视频会议记录（走 lark-vc）、待办任务（走 lark-task）。SkillsJul 2026
- shadcnManages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. AppliSkillsJul 2026
- xlsxUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.SkillsJul 2026
- firebase-data-connectAgent Skills for FirebaseSkillsJul 2026
