HandsOn
starsforks
How to install HandsOn?
3spky5u-oss/handson
$git clone https://github.com/3spky5u-oss/HandsOnGive Claude eyes and hands — screen capture and desktop automation plugin for Claude Code
README.md
| 1 | # HandsOn |
| 2 | |
| 3 | [](https://github.com/3spky5u-oss/HandsOn/releases) |
| 4 | [](LICENSE) |
| 5 | []() |
| 6 | [](https://docs.anthropic.com/en/docs/claude-code) |
| 7 | [](https://modelcontextprotocol.io) |
| 8 | []() |
| 9 | |
| 10 | Give Claude eyes and hands. A Claude Code plugin that lets Claude see your screen and interact with any application on your desktop. |
| 11 | |
| 12 | <!-- TODO:  --> |
| 13 | |
| 14 | > **Alpha software.** HandsOn works and is genuinely useful, but Claude will sometimes stumble through tasks — misclicking sidebar links, scrolling the wrong container, needing multiple attempts to find the right element. Complex multi-step workflows (like filling out a Reddit post) may take several retries to get right. It's getting better with each release, but set your expectations accordingly. |
| 15 | |
| 16 | ## What's New in v0.3.0 |
| 17 | |
| 18 | - **Window-cropped visual detection** — When a target window is set, visual analysis is cropped to that window's bounds, eliminating noise from the terminal and other background apps |
| 19 | - **30-50% token reduction** — Compact output across all tools: shorter class names, removed redundant formatting, tighter coordinate notation |
| 20 | - **Unnamed element filtering** — `list_elements` now skips unnamed Pane/Group/Custom elements that add noise without value (common in Steam, Electron apps, etc.) |
| 21 | - **Process-tree terminal refocus** — `set_target_window("")` walks the process tree to find and refocus the host terminal instead of guessing by title |
| 22 | - **Immediate window focus** — `set_target_window()` now focuses the target window immediately on set, not just before the next input action |
| 23 | - **Visual detect module** — New CV-based UI region detection for apps with limited accessibility support |
| 24 | - **Self-correcting click** — `click_text` automatically retries at offset positions when the initial center click produces no visual change |
| 25 | |
| 26 | ## Why HandsOn? |
| 27 | |
| 28 | Claude Code is powerful, but it's blind — it can't see your screen or interact with anything outside the terminal. HandsOn changes that. It gives Claude eyes to see what's on your desktop and hands to interact with it, turning it into a general-purpose desktop assistant. |
| 29 | |
| 30 | **The core idea:** Claude can now do anything you can do at a keyboard and mouse — navigate apps, fill out forms, click through wizards, manage windows, launch programs, and verify its own work visually. |
| 31 | |
| 32 | **What this enables:** |
| 33 | |
| 34 | - **Desktop automation** — Automate any application on your computer, even legacy apps with no API |
| 35 | - **Visual verification** — Claude can see the result of its own work, catch visual bugs, and fix them without you describing what's wrong |
| 36 | - **App interaction** — Navigate settings, install software, fill out forms, click through dialogs — anything you'd normally alt-tab to do yourself |
| 37 | - **Accessibility-first targeting** — Uses the platform accessibility tree — Windows UI Automation (UIA) or macOS AXUIElement — not just pixel coordinates, so element targeting is DPI-aware and resolution-independent |
| 38 | - **Automatic fallbacks** — When accessibility doesn't work (custom widgets, canvas apps, games), it falls back to OCR, then to visual analysis. `smart_find` picks the right approach automatically. |
| 39 | - **Framework-aware** — Detects Qt, WPF, Electron, WinForms, etc., and tells Claude exactly what will and won't work |
| 40 | |
| 41 | ## Quick Start |
| 42 | |
| 43 | ### Install |
| 44 | |
| 45 | ```bash |
| 46 | # From GitHub (recommended) |
| 47 | claude plugin marketplace add 3spky5u-oss/HandsOn |
| 48 | claude plugin install handson |
| 49 | |
| 50 | # From GitLab |
| 51 | claude plugin marketplace add git@gitlab.com:3spky5u/HandsOn.git |
| 52 | claude plugin install handson |
| 53 | |
| 54 | # From Codeberg |
| 55 | claude plugin marketplace add git@codeberg.org:3spky5u/HandsOn.git |
| 56 | claude plugin install handson |
| 57 | ``` |
| 58 | |
| 59 | Restart Claude Code after installing. Python dependencies are installed automatically on first run. |
| 60 | |
| 61 | To update: |
| 62 | ```bash |
| 63 | claude plugin marketplace update handson |
| 64 | claude plugin update handson |
| 65 | ``` |
| 66 | |
| 67 | **Requirements:** Python 3.10+, Windows 10/11 or macOS 12+ (Monterey or later). |
| 68 | |
| 69 | **Optional (Playwright bridge for browser tasks):** |
| 70 | If you also have the [Playwright MCP plugin](https://www.npmjs.com/package/@anthropic-ai/claude-code-playwright) installed, HandsOn automatically defers to Playwright for in-browser interactions (DOM access, form filling, page navigation) and reserves itself for desktop apps, native dialogs, and visual verification. For the best experience, connect Playwright to your existing browser using the `--extension` flag or `--cdp-endpoint` — this lets Playwright work on the same browser you see, with HandsOn verifying the result on you |
People who liked this also viewed
6 picksType
- Scrapling🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!MCP ServersJul 2026
- excel-mcp-serverA Model Context Protocol server for Excel file manipulationMCP ServersApr 2026
- Excel-MCP-Server-MasterExcel MCP Server - Manipulate Excel files without Microsoft Excel. Model Context Protocol for XLSX, XLSM with Claude AI integrationMCP ServersJun 2025
- agent-deviceCLI to control iOS and Android devices for AI agentsMCP ServersJul 2026
- n8n-mcpA MCP for Claude Desktop / Claude Code / Windsurf / Cursor to build n8n workflows for you MCP ServersJul 2026
- ha-mcpThe Unofficial and Awesome Home Assistant MCP ServerMCP ServersJul 2026
