playwright-scraper-skill
starsforks
How to install playwright-scraper-skill?
szsip239/teamclaw/playwright-scraper-skill
$npx -y skills add szsip239/teamclaw --skill playwright-scraper-skillPlaywright-based web scraping OpenClaw Skill with anti-bot protection. Successfully tested on complex sites like Discuss.com.hk.
SKILL.md
| 1 | # Playwright Scraper Skill |
| 2 | |
| 3 | A Playwright-based web scraping OpenClaw Skill with anti-bot protection. Choose the best approach based on the target website's anti-bot level. |
| 4 | |
| 5 | --- |
| 6 | |
| 7 | ## 🎯 Use Case Matrix |
| 8 | |
| 9 | | Target Website | Anti-Bot Level | Recommended Method | Script | |
| 10 | |---------------|----------------|-------------------|--------| |
| 11 | | **Regular Sites** | Low | web_fetch tool | N/A (built-in) | |
| 12 | | **Dynamic Sites** | Medium | Playwright Simple | `scripts/playwright-simple.js` | |
| 13 | | **Cloudflare Protected** | High | **Playwright Stealth** ⭐ | `scripts/playwright-stealth.js` | |
| 14 | | **YouTube** | Special | deep-scraper | Install separately | |
| 15 | | **Reddit** | Special | reddit-scraper | Install separately | |
| 16 | |
| 17 | --- |
| 18 | |
| 19 | ## 📦 Installation |
| 20 | |
| 21 | ```bash |
| 22 | cd playwright-scraper-skill |
| 23 | npm install |
| 24 | npx playwright install chromium |
| 25 | ``` |
| 26 | |
| 27 | --- |
| 28 | |
| 29 | ## 🚀 Quick Start |
| 30 | |
| 31 | ### 1️⃣ Simple Sites (No Anti-Bot) |
| 32 | |
| 33 | Use OpenClaw's built-in `web_fetch` tool: |
| 34 | |
| 35 | ```bash |
| 36 | # Invoke directly in OpenClaw |
| 37 | Hey, fetch me the content from https://example.com |
| 38 | ``` |
| 39 | |
| 40 | --- |
| 41 | |
| 42 | ### 2️⃣ Dynamic Sites (Requires JavaScript) |
| 43 | |
| 44 | Use **Playwright Simple**: |
| 45 | |
| 46 | ```bash |
| 47 | node scripts/playwright-simple.js "https://example.com" |
| 48 | ``` |
| 49 | |
| 50 | **Example output:** |
| 51 | ```json |
| 52 | { |
| 53 | "url": "https://example.com", |
| 54 | "title": "Example Domain", |
| 55 | "content": "...", |
| 56 | "elapsedSeconds": "3.45" |
| 57 | } |
| 58 | ``` |
| 59 | |
| 60 | --- |
| 61 | |
| 62 | ### 3️⃣ Anti-Bot Protected Sites (Cloudflare etc.) |
| 63 | |
| 64 | Use **Playwright Stealth**: |
| 65 | |
| 66 | ```bash |
| 67 | node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot" |
| 68 | ``` |
| 69 | |
| 70 | **Features:** |
| 71 | - Hide automation markers (`navigator.webdriver = false`) |
| 72 | - Realistic User-Agent (iPhone, Android) |
| 73 | - Random delays to mimic human behavior |
| 74 | - Screenshot and HTML saving support |
| 75 | |
| 76 | --- |
| 77 | |
| 78 | ### 4️⃣ YouTube Video Transcripts |
| 79 | |
| 80 | Use **deep-scraper** (install separately): |
| 81 | |
| 82 | ```bash |
| 83 | # Install deep-scraper skill |
| 84 | npx clawhub install deep-scraper |
| 85 | |
| 86 | # Use it |
| 87 | cd skills/deep-scraper |
| 88 | node assets/youtube_handler.js "https://www.youtube.com/watch?v=VIDEO_ID" |
| 89 | ``` |
| 90 | |
| 91 | --- |
| 92 | |
| 93 | ## 📖 Script Descriptions |
| 94 | |
| 95 | ### `scripts/playwright-simple.js` |
| 96 | - **Use Case:** Regular dynamic websites |
| 97 | - **Speed:** Fast (3-5 seconds) |
| 98 | - **Anti-Bot:** None |
| 99 | - **Output:** JSON (title, content, URL) |
| 100 | |
| 101 | ### `scripts/playwright-stealth.js` ⭐ |
| 102 | - **Use Case:** Sites with Cloudflare or anti-bot protection |
| 103 | - **Speed:** Medium (5-20 seconds) |
| 104 | - **Anti-Bot:** Medium-High (hides automation, realistic UA) |
| 105 | - **Output:** JSON + Screenshot + HTML file |
| 106 | - **Verified:** 100% success on Discuss.com.hk |
| 107 | |
| 108 | --- |
| 109 | |
| 110 | ## 🎓 Best Practices |
| 111 | |
| 112 | ### 1. Try web_fetch First |
| 113 | If the site doesn't have dynamic loading, use OpenClaw's `web_fetch` tool—it's fastest. |
| 114 | |
| 115 | ### 2. Need JavaScript? Use Playwright Simple |
| 116 | If you need to wait for JavaScript rendering, use `playwright-simple.js`. |
| 117 | |
| 118 | ### 3. Getting Blocked? Use Stealth |
| 119 | If you encounter 403 or Cloudflare challenges, use `playwright-stealth.js`. |
| 120 | |
| 121 | ### 4. Special Sites Need Specialized Skills |
| 122 | - YouTube → deep-scraper |
| 123 | - Reddit → reddit-scraper |
| 124 | - Twitter → bird skill |
| 125 | |
| 126 | --- |
| 127 | |
| 128 | ## 🔧 Customization |
| 129 | |
| 130 | All scripts support environment variables: |
| 131 | |
| 132 | ```bash |
| 133 | # Set screenshot path |
| 134 | SCREENSHOT_PATH=/path/to/screenshot.png node scripts/playwright-stealth.js URL |
| 135 | |
| 136 | # Set wait time (milliseconds) |
| 137 | WAIT_TIME=10000 node scripts/playwright-simple.js URL |
| 138 | |
| 139 | # Enable headful mode (show browser) |
| 140 | HEADLESS=false node scripts/playwright-stealth.js URL |
| 141 | |
| 142 | # Save HTML |
| 143 | SAVE_HTML=true node scripts/playwright-stealth.js URL |
| 144 | |
| 145 | # Custom User-Agent |
| 146 | USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js URL |
| 147 | ``` |
| 148 | |
| 149 | --- |
| 150 | |
| 151 | ## 📊 Performance Comparison |
| 152 | |
| 153 | | Method | Speed | Anti-Bot | Success Rate (Discuss.com.hk) | |
| 154 | |--------|-------|----------|-------------------------------| |
| 155 | | web_fetch | ⚡ Fastest | ❌ None | 0% | |
| 156 | | Playwright Simple | 🚀 Fast | ⚠️ Low | 20% | |
| 157 | | **Playwright Stealth** | ⏱️ Medium | ✅ Medium | **100%** ✅ | |
| 158 | | Puppeteer Stealth | ⏱️ Medium | ✅ Medium-High | ~80% | |
| 159 | | Crawlee (deep-scraper) | 🐢 Slow | ❌ Detected | 0% | |
| 160 | | Chaser (Rust) | ⏱️ Medium | ❌ Detected | 0% | |
| 161 | |
| 162 | --- |
| 163 | |
| 164 | ## 🛡️ Anti-Bot Techniques Summary |
| 165 | |
| 166 | Lessons learned from our testing: |
| 167 | |
| 168 | ### ✅ Effective Anti-Bot Measures |
| 169 | 1. **Hide `navigator.webdriver`** — Essential |
| 170 | 2. **Realistic User-Agent** — Use real devices (iPhone, Android) |
| 171 | 3. **Mimic Human Behavior** — Random delays, scrolling |
| 172 | 4. **Avoid Framework Signatures** — Crawlee, Selenium are easily detected |
| 173 | 5. **Use `addInitScript` (Playwright)** — Inject before page load |
| 174 | |
| 175 | ### ❌ Ineffective Anti-Bot Measures |
| 176 | 1. **Only changing User-Agent** — Not enough |
| 177 | 2. **Using high-level frameworks (Crawlee)** — More easily detected |
| 178 | 3. **Docker isolation** — Doesn't help with Cloudflare |
| 179 | |
| 180 | --- |
| 181 | |
| 182 | ## 🔍 Troubleshooting |
| 183 | |
| 184 | ### Issue: 403 Forbidden |
| 185 | **Solution:** Use `playwright-stealth.js` |
| 186 | |
| 187 | ### Issue: Cloudflare Challenge Page |
| 188 | **Solution:** |
| 189 | 1. Increase wait time (10-15 seconds) |
| 190 | 2. Try `headless: false` (headful mode sometimes has higher success rate) |
| 191 | 3. Con |
People who liked this also viewed
6 picksType
- agent-browserBrowser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web aSkillsJul 2026
- browser-act--- name: browser-act description: "Browser automation CLI for AI agents. NEVER run browser-act commands directly via Bash — always invoke this skill first. Use browser-act when a user mentions it bSkillsJul 2026
- playwright-cliAutomate browser interactions, test web pages and work with Playwright tests.SkillsJul 2026
- accessibilityAudit and improve web accessibility following WCAG 2.2 guidelines. Use when asked to "improve accessibility", "a11y audit", "WCAG compliance", "screen reader support", "keyboard navigation", or "make SkillsJun 2026
- anti-detect-browserLaunch and manage anti-detect browsers with unique real-device fingerprints for multi-account operations, web scraping, ad verification, and AI agent automation. Use when the user needs to run multiplSkillsMar 2026
- web-access所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。SkillsMay 2026
