webclaw
starsforks
How to install webclaw?
0xmassi/webclaw
$git clone https://github.com/0xMassi/webclaw---
README.md
| 1 | <p align="center"><strong>English</strong> | <a href="./README_zh-CN.md">简体中文</a></p> |
| 2 | |
| 3 | <p align="center"> |
| 4 | <a href="https://webclaw.io"> |
| 5 | <img src=".github/banner.png" alt="webclaw" width="760" /> |
| 6 | </a> |
| 7 | </p> |
| 8 | |
| 9 | <h1 align="center">webclaw</h1> |
| 10 | |
| 11 | <p align="center"> |
| 12 | <strong>Turn websites into clean markdown, JSON, and LLM-ready context.</strong><br/> |
| 13 | <sub>CLI, MCP server, REST API, and SDKs for AI agents and RAG pipelines.</sub> |
| 14 | </p> |
| 15 | |
| 16 | <p align="center"> |
| 17 | <a href="https://github.com/0xMassi/webclaw/stargazers"><img src="https://shieldcn.dev/github/stars/0xMassi/webclaw.svg?variant=branded&logo=github" alt="Stars" /></a> |
| 18 | <a href="https://github.com/0xMassi/webclaw/releases"><img src="https://shieldcn.dev/github/tag/0xMassi/webclaw.svg?variant=branded&logo=rust" alt="Version" /></a> |
| 19 | <a href="https://github.com/0xMassi/webclaw/blob/main/LICENSE"><img src="https://shieldcn.dev/github/license/0xMassi/webclaw.svg?variant=branded" alt="License" /></a> |
| 20 | <a href="https://www.npmjs.com/package/create-webclaw"><img src="https://shieldcn.dev/npm/dt/create-webclaw.svg?variant=branded" alt="npm installs" /></a> |
| 21 | </p> |
| 22 | |
| 23 | <p align="center"> |
| 24 | <a href="https://discord.gg/KDfd48EpnW"><img src="https://shieldcn.dev/badge/Discord-Join.svg?variant=branded&logo=discord" alt="Discord" /></a> |
| 25 | <a href="https://x.com/webclaw_io"><img src="https://shieldcn.dev/badge/Follow-@webclaw__io.svg?variant=branded&logo=x" alt="X / Twitter" /></a> |
| 26 | <a href="https://webclaw.io"><img src="https://shieldcn.dev/badge/Hosted-webclaw.io.svg?variant=branded&logo=safari" alt="Hosted webclaw" /></a> |
| 27 | <a href="https://webclaw.io/docs"><img src="https://shieldcn.dev/badge/Docs-Read.svg?variant=branded&logo=readthedocs" alt="Docs" /></a> |
| 28 | </p> |
| 29 | |
| 30 | <p align="center"> |
| 31 | <img src="assets/demo.gif" alt="webclaw extracting clean markdown from a page" width="760" /> |
| 32 | </p> |
| 33 | |
| 34 | --- |
| 35 | |
| 36 | Most web scraping tools give your agent one of two bad outputs: |
| 37 | |
| 38 | - a blocked page, login wall, or empty app shell |
| 39 | - raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate |
| 40 | |
| 41 | [webclaw.io](https://webclaw.io) is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server. |
| 42 | |
| 43 | webclaw turns a URL into clean content your tools can actually use. |
| 44 | |
| 45 | ```bash |
| 46 | webclaw https://example.com --format markdown |
| 47 | ``` |
| 48 | |
| 49 | ```md |
| 50 | # Example Domain |
| 51 | |
| 52 | This domain is for use in illustrative examples in documents. |
| 53 | |
| 54 | You may use this domain in literature without prior coordination or asking for permission. |
| 55 | ``` |
| 56 | |
| 57 | Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server. |
| 58 | |
| 59 | --- |
| 60 | |
| 61 | ## Install |
| 62 | |
| 63 | ### Agent setup |
| 64 | |
| 65 | The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools: |
| 66 | |
| 67 | ```bash |
| 68 | npx create-webclaw |
| 69 | ``` |
| 70 | |
| 71 | The installer detects supported clients and configures the MCP server for you. |
| 72 | |
| 73 | ### Homebrew |
| 74 | |
| 75 | ```bash |
| 76 | brew tap 0xMassi/webclaw |
| 77 | brew install webclaw |
| 78 | ``` |
| 79 | |
| 80 | ### Prebuilt binaries |
| 81 | |
| 82 | Download macOS, Linux, and Windows binaries from [GitHub Releases](https://github.com/0xMassi/webclaw/releases). |
| 83 | |
| 84 | ### Docker |
| 85 | |
| 86 | ```bash |
| 87 | docker run --rm ghcr.io/0xmassi/webclaw https://example.com |
| 88 | ``` |
| 89 | |
| 90 | ### Cargo |
| 91 | |
| 92 | ```bash |
| 93 | cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli |
| 94 | cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp |
| 95 | ``` |
| 96 | |
| 97 | If building from source fails because native build tools are missing, install the platform prerequisites: |
| 98 | |
| 99 | | OS | Command | |
| 100 | | --- | --- | |
| 101 | | Debian / Ubuntu | `sudo apt install -y pkg-config libssl-dev cmake clang git build-essential` | |
| 102 | | Fedora / RHEL | `sudo dnf install -y pkg-config openssl-devel cmake clang git make gcc` | |
| 103 | | Arch | `sudo pacman -S pkg-config openssl cmake clang git base-devel` | |
| 104 | | macOS | `xcode-select --install` | |
| 105 | |
| 106 | --- |
| 107 | |
| 108 | ## Quick Start |
| 109 | |
| 110 | ### Scrape one page |
| 111 | |
| 112 | ```bash |
| 113 | webclaw https://stripe.com --format markdown |
| 114 | ``` |
| 115 | |
| 116 | ### Return LLM-optimized text |
| 117 | |
| 118 | ```bash |
| 119 | webclaw |
People who liked this also viewed
6 picksType
- ddgsA metasearch library that aggregates results from diverse web search servicesMCP ServersMay 2026
- firecrawl-mcp-server🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.MCP ServersJul 2026
- fliGoogle Flights MCP, CLI and Python LibraryMCP ServersJul 2026
- sembleFast and Accurate Code Search for Agents. Uses ~98% fewer tokens than grep+readMCP ServersJul 2026
- duckduckgo-mcp-serverA Model Context Protocol (MCP) server that provides web search capabilities through DuckDuckGo, with additional features for content fetching and parsing.MCP ServersJul 2026
- wikipedia-mcpA Model Context Protocol (MCP) server that retrieves information from Wikipedia to provide context to LLMs.MCP ServersMay 2026
