phone-use
starsforks
How to install phone-use?
niki914/agentic-nexus/phone-use
$npx -y skills add niki914/agentic-nexus --skill phone-useMUST Load this skill for ANY task that involves GUI interaction or device control —
SKILL.md
| 1 | ## Overview |
| 2 | |
| 3 | Phone Use gives you the ability to see and interact with the Android device screen. It uses Android AccessibilityService (with root-assisted auto-setup) to read the UI tree and simulate user actions. |
| 4 | |
| 5 | This is the ONLY way to control the device — do NOT attempt to use non-existent APIs or platform tools. |
| 6 | |
| 7 | **When to load:** any GUI task. App launching, tapping, typing, scrolling, swiping, navigation keys, screen reading. Even when the user doesn't say "on the phone" — infer it from the task. |
| 8 | |
| 9 | **When NOT to retry:** non-native apps (Flutter, Unity, WebView, games). If `screen_content` returns empty/root-only, stop and report immediately. |
| 10 | |
| 11 | Five core tools: |
| 12 | |
| 13 | | Tool | Purpose | |
| 14 | |:-----|:--------| |
| 15 | | `screen_content` | Read the current screen's accessibility tree as YAML | |
| 16 | | `search_nodes` | Search the current screen for nodes matching keywords. Returns matched indices for use with `node_action` | |
| 17 | | `node_action` | Click, long-click, set text, or scroll on a UI node by its index | |
| 18 | | `gesture` | Swipe/drag by screen coordinates | |
| 19 | | `key_event` | Inject system key events (Back, Home, Recents, etc.) | |
| 20 | |
| 21 | Two auxiliary tools for app discovery and launching: |
| 22 | |
| 23 | | Tool | Purpose | |
| 24 | |:-----|:--------| |
| 25 | | `search_apps` | Search installed apps by name or package name | |
| 26 | | `launch_app` | Launch an app by package name or fuzzy app name | |
| 27 | |
| 28 | ## Tool Reference |
| 29 | |
| 30 | ### screen_content |
| 31 | |
| 32 | ``` |
| 33 | screen_content() |
| 34 | ``` |
| 35 | |
| 36 | Returns the current screen's accessibility tree as a YAML string. No arguments. |
| 37 | |
| 38 | **Response format:** The tool returns raw YAML output. On failure, the string starts with `error: `. |
| 39 | |
| 40 | **Non-native app detection:** If the tree is empty or contains only a root node, the current app likely uses a non-native UI framework (Flutter, Unity, WebView, game engine) that does not expose standard Android accessibility node trees. No amount of retrying will help. **Stop immediately and report to the user.** |
| 41 | |
| 42 | ### search_nodes |
| 43 | |
| 44 | ``` |
| 45 | search_nodes(keywords: string[], match_mode?: string, limit?: integer) |
| 46 | ``` |
| 47 | |
| 48 | | Param | Type | Required | Description | |
| 49 | |:------|:-----|:---------|:------------| |
| 50 | | `keywords` | string[] | yes | Keywords for case-insensitive substring matching on node text (`txt`) and content description (`h`) | |
| 51 | | `match_mode` | string | no | `"any"` (default, match any keyword) or `"all"` (match all keywords) | |
| 52 | | `limit` | integer | no | Maximum results to return (default 10) | |
| 53 | |
| 54 | ```json |
| 55 | {"keywords": ["收藏", "我的"]} |
| 56 | {"keywords": ["settings"], "match_mode": "all", "limit": 5} |
| 57 | ``` |
| 58 | |
| 59 | Use `search_nodes` when the screen has many nodes and you need to locate specific UI elements by their text or description. It returns a concise YAML list of matching nodes with their indices, types, bounds, and positions — use the returned indices directly with `node_action`. This is faster than parsing the full `screen_content` tree. |
| 60 | |
| 61 | **Response format:** The tool returns raw YAML output: |
| 62 | ```yaml |
| 63 | matched: 3 |
| 64 | nodes: |
| 65 | - {i: 12, t: tab, b: [0,2160,360,2400], pos: bottom-left, txt: 收藏, tap: true} |
| 66 | - {i: 15, t: tab, b: [360,2160,720,2400], pos: bottom, txt: 我的, tap: true} |
| 67 | ``` |
| 68 | |
| 69 | If no nodes match, `matched: 0` with an empty `nodes:` section. On failure, the string starts with `error: `. |
| 70 | |
| 71 | ### node_action |
| 72 | |
| 73 | ``` |
| 74 | node_action(action: string, index: integer, text?: string, method?: string) |
| 75 | ``` |
| 76 | |
| 77 | | Param | Type | Required | Description | |
| 78 | |:------|:-----|:---------|:------------| |
| 79 | | `action` | string | yes | `click`, `long_click`, `set_text`, `scroll_forward`, `scroll_backward` | |
| 80 | | `index` | integer | yes | Node index from the latest `screen_content` output (the `i` field) | |
| 81 | | `text` | string | only `set_text` | Text to type into an input field | |
| 82 | | `method` | string | no | `accessibility` (default) or `shell` | |
| 83 | |
| 84 | ```json |
| 85 | {"action": "click", "index": 42} |
| 86 | {"action": "set_text", "index": 7, "text": "hello"} |
| 87 | {"action": "scroll_forward", "index": 15} |
| 88 | {"action": "long_click", "index": 3, "method": "shell"} |
| 89 | ``` |
| 90 | |
| 91 | **Method rules:** |
| 92 | - `set_text` **only works with `accessibility`** (the default). Shell cannot type into text fields — the tool rejects it with `METHOD_NOT_SUPPORTED`. |
| 93 | - `accessibility` tries the accessibility action first; non-set_text actions auto-fall-back to shell on failure. |
| 94 | - `shell` uses `su -c input tap/swipe` directly, bypassing the accessibility service. Use it when the service is unavailable or as an explicit escape hatch. |
| 95 | |
| 96 | **Scroll limitation:** `scroll_forward` and `scroll_backward` issue one accessibility scroll action per call, but the resulting distance is app-defined. Use them only for single-step increments in pickers or small widgets. For list traversal, follow Wo |
People who liked this also viewed
6 picksType
- paperclip-create-agentCreate new agents in Paperclip with governance-aware hiring. Use when you need to inspect adapter configuration options, compare existing agent configs, draft a new agent prompt/config, and submit a hSkillsJul 2026
- subagent-driven-developmentUse when executing implementation plans with independent tasks in the current sessionSkillsJul 2026
- dispatching-parallel-agentsUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesSkillsJul 2026
- browser-act-skill-forgeForges reusable Skill packages (SKILL.md + scripts) from website exploration via browser-act — no re-exploration later. Use when: user wants a reusable Skill for any website, needs to understand a sitSkillsJul 2026
- wizardGenerate an interactive bash wizard that walks a human through a manual procedure — third-party setup, a one-off migration, an A→B state transition — opening URLs, capturing values, confirming each stSkillsJul 2026
- wonda-cliUsing the Wonda CLI to generate images, videos, music, and audio from the terminal — plus LinkedIn, Reddit, and X/Twitter research and automationSkillsJul 2026
