agent-desktop
starsforks
How to install agent-desktop?
lahfir/agent-desktop
$git clone https://github.com/lahfir/agent-desktopagent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
README.md
| 1 | <h1 align="center">AGENT DESKTOP</h1> |
| 2 | |
| 3 | <p align="center"> |
| 4 | <strong>OBSERVE. DECIDE. ACT.</strong> |
| 5 | </p> |
| 6 | |
| 7 | <p align="center"> |
| 8 | <a href="https://github.com/lahfir/agent-desktop/actions/workflows/ci.yml?query=branch%3Amain"><img src="https://img.shields.io/github/actions/workflow/status/lahfir/agent-desktop/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a> |
| 9 | <a href="https://github.com/lahfir/agent-desktop/releases"><img src="https://img.shields.io/github/v/release/lahfir/agent-desktop?include_prereleases&style=for-the-badge" alt="GitHub release"></a> |
| 10 | <a href="https://www.npmjs.com/package/agent-desktop"><img src="https://img.shields.io/npm/v/agent-desktop?label=npm&style=for-the-badge" alt="npm version"></a> |
| 11 | <a href="https://clawhub.ai/lahfir/agent-desktop"><img src="https://img.shields.io/badge/ClawHub-skill-f97316?style=for-the-badge" alt="ClawHub skill"></a> |
| 12 | <a href="https://skills.sh/lahfir/agent-desktop/agent-desktop"><img src="https://img.shields.io/badge/skills.sh-listed-8b5cf6?style=for-the-badge" alt="skills.sh listing"></a> |
| 13 | <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache--2.0-blue.svg?style=for-the-badge" alt="Apache-2.0 License"></a> |
| 14 | </p> |
| 15 | |
| 16 | <p align="center"> |
| 17 | <img src="assets/Tutorial.gif" alt="agent-desktop tutorial demo" width="800" /> |
| 18 | </p> |
| 19 | |
| 20 | **agent-desktop** is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required. |
| 21 | |
| 22 | ## Architecture |
| 23 | |
| 24 | <p align="center"> |
| 25 | <img src="docs/architecture.png" alt="agent-desktop architecture diagram" width="900" /> |
| 26 | </p> |
| 27 | |
| 28 | <p align="center"> |
| 29 | <img src="docs/example.png" alt="agent-desktop real-world example — Slack accessibility tree with 97% token savings" width="900" /> |
| 30 | </p> |
| 31 | |
| 32 | <a href="https://star-history.com/#lahfir/agent-desktop&Date"> |
| 33 | <picture> |
| 34 | <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=lahfir/agent-desktop&type=Date&theme=dark"> |
| 35 | <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=lahfir/agent-desktop&type=Date"> |
| 36 | <img alt="Star history for lahfir/agent-desktop" src="https://api.star-history.com/svg?repos=lahfir/agent-desktop&type=Date"> |
| 37 | </picture> |
| 38 | </a> |
| 39 | |
| 40 | ## Key Features |
| 41 | |
| 42 | - **Native Rust CLI**: Fast, single binary, no runtime dependencies |
| 43 | - **C-ABI cdylib** (`libagent_desktop_ffi`): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call |
| 44 | - **58 command names, 54 operational commands**: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, plus a bundled `skills` doc loader. The four held-input names are reserved for a stateful daemon and fail closed in the stateless CLI. |
| 45 | - **Progressive skeleton traversal**: 78–96% token reduction on dense apps via shallow overview + targeted drill-down |
| 46 | - **Snapshot & refs**: AI-optimized workflow using compact snapshot IDs and qualified element references (`@s8f3k2p9:e1`, `@s8f3k2p9:e2`) |
| 47 | - **Headless-by-default interactions**: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects |
| 48 | - **Structured JSON output**: Machine-readable responses with error codes and recovery hints |
| 49 | - **Works with any app**: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree |
| 50 | |
| 51 | ## Installation |
| 52 | |
| 53 | ### npm (recommended) |
| 54 | |
| 55 | ```bash |
| 56 | npm install -g agent-desktop # downloads prebuilt binary automatically |
| 57 | ``` |
| 58 | |
| 59 | Or without installing: |
| 60 | |
| 61 | ```bash |
| 62 | npx agent-desktop snapshot --app Finder -i |
| 63 | ``` |
| 64 | |
| 65 | ### From source |
| 66 | |
| 67 | ```bash |
| 68 | git clone https://github.com/lahfir/agent-desktop |
| 69 | cd agent-desktop |
| 70 | cargo build --release |
| 71 | cp target/release/agent-desktop /usr/local/bin/ |
| 72 | ``` |
| 73 | |
| 74 | Requires Rust 1.89+ and macOS 13.0+. |
| 75 | |
| 76 | ### Permissions |
| 77 | |
| 78 | macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and th |
People who liked this also viewed
6 picksType
- Scrapling🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!MCP ServersJul 2026
- excel-mcp-serverA Model Context Protocol server for Excel file manipulationMCP ServersApr 2026
- Excel-MCP-Server-MasterExcel MCP Server - Manipulate Excel files without Microsoft Excel. Model Context Protocol for XLSX, XLSM with Claude AI integrationMCP ServersJun 2025
- agent-deviceCLI to control iOS and Android devices for AI agentsMCP ServersJul 2026
- n8n-mcpA MCP for Claude Desktop / Claude Code / Windsurf / Cursor to build n8n workflows for you MCP ServersJul 2026
- ha-mcpThe Unofficial and Awesome Home Assistant MCP ServerMCP ServersJul 2026
