computer-use

Name: computer-use
Author: tykimos

59stars11forks

How to install computer-use?

tykimos/cheliped-skills/computer-use

$npx -y skills add tykimos/cheliped-skills --skill computer-use

Installs into the current project.

Summary

Control the local macOS machine — move/click the mouse, type and press key combos, take screenshots, and launch/quit/activate apps via AppleScript and native CLI tools. Use this skill when Claude needs to: (1) take a screenshot of the Mac screen, (2) move or click the mouse at co

Files · 1

View on GitHub

SKILL.md

1	# Cheliped Computer Use (macOS)
2
3	Drive the local Mac from the terminal. One CLI takes a JSON array of commands and returns JSON — the same shape as the `browser` skill.
4
5	## Output Envelope
6
7	Output is always a JSON array, one envelope per command, compact by default:
8
9	```json
10	[{ "cmd": "screenshot", "ok": true, "result": { "path": "...", "scale": 2 } },
11	{ "cmd": "type", "ok": false, "error": { "code": "E_NO_PERMISSION", "message": "..." } }]
12	```
13
14	Each command runs even if a prior one fails (per-command isolation). Error `code` is one of `E_NO_PERMISSION` (Accessibility off), `E_NO_CLICLICK`, `E_WRONG_FOCUS`, `E_DISABLED` (escape hatch not enabled), `E_BAD_ARG`, `E_UNKNOWN_CMD`, `E_TIMEOUT`, `E_UNKNOWN`.
15
16	Flags (before the JSON arg): `--pretty` indents output; `--allow-shell` enables the `run-shell` / `run-applescript` escape hatches (off by default).
17
18	## Setup
19
20	Built-ins (`osascript`, `screencapture`) need no install. Mouse control needs cliclick:
21
22	```bash
23	brew install cliclick
24	```
25
26	Permissions: the terminal/agent process must be granted Accessibility and Screen Recording in System Settings → Privacy & Security. The CLI now preflights Accessibility and returns `E_NO_PERMISSION` (instead of a silent no-op) when keyboard/mouse input would not be delivered; screenshots that come back near-uniform get a `warning` hinting at missing Screen Recording. Grant the permissions, then restart the terminal (permissions only apply to a freshly launched process).
27
28	## Core Workflow: Observe-Act Loop
29
30	```bash
31	# 1. See the screen
32	node scripts/computer-cli.mjs '[{"cmd":"screenshot","args":["/tmp/now.png"]}]'
33
34	# 2. Act on what you saw (read the screenshot, then click / type)
35	node scripts/computer-cli.mjs '[{"cmd":"click","args":[640,400]},{"cmd":"type","args":["hello"]}]'
36	```
37
38	Read the screenshot with the Read tool, decide coordinates, then act. Re-screenshot to confirm.
39
40	## Commands
41
42	Each row's `Returns` is the `result` object inside the envelope.
43
44	\| Command \| Args \| Returns \|
45	\|---------\|------\|---------\|
46	\| `screenshot` \| `["path"]` (optional); fields `region:[x,y,w,h]`, `maxWidth:n` \| `{ path, pixelWidth, pixelHeight, pointWidth, pointHeight, scale, cropOffset?, warning? }` \|
47	\| `screen-size` \| none \| `{ width, height, unit:"point", scale }` \|
48	\| `mouse-pos` \| none \| `{ x, y, unit:"point" }` (needs cliclick) \|
49	\| `move` \| `[x, y]` \| `{ x, y }` (cliclick) \|
50	\| `click` \| `[x, y]` (optional → current pos) \| `{}` (cliclick) \|
51	\| `double-click` \| `[x, y]` (optional) \| `{}` (cliclick) \|
52	\| `right-click` \| `[x, y]` (optional) \| `{}` (cliclick) \|
53	\| `drag` \| `[x1, y1, x2, y2]` \| `{}` (cliclick) \|
54	\| `scroll` \| `[amount]` (+ up / − down; `\|amount\|/5` pages) \| `{ pages }` \|
55	\| `type` \| `["text"]`; fields `target:"App"`, `noClipboard:true` \| `{ method:"keystroke"\\|"paste" }` \|
56	\| `paste` \| `["text"]` (optional → current clipboard); field `target` \| `{ method:"paste" }` \|
57	\| `key` \| `["combo"]` e.g. `"cmd c"`; field `target:"App"` \| `{ combo }` \|
58	\| `launch` \| `["AppName"]` \| `{ app }` \|
59	\| `activate` \| `["AppName"]` \| `{ app }` \|
60	\| `quit` \| `["AppName"]` \| `{ app }` \|
61	\| `open` \| `["url-or-path"]` \| `{ target }` \|
62	\| `list-apps` \| none \| `{ apps: [...] }` — visible running apps \|
63	\| `frontmost` \| none \| `{ app }` — active app \|
64	\| `notify` \| `["text", "title"]` \| `{}` — banner notification \|
65	\| `run-applescript` \| `["script"]` (needs `--allow-shell`) \| `{ result, audited }` \|
66	\| `run-shell` \| `["command"]` (needs `--allow-shell`) \| `{ stdout, audited }` \|
67	\| `wait` \| `[ms]` \| `{}` \|
68
69	Typing (IME-safe): `type` sends ASCII as keystrokes but routes non-ASCII / multiline text (한글, 漢字, emoji, newlines) through a clipboard paste so it lands intact in CJK/IME fields — it saves and restores your clipboard (pass `noClipboard:true` to opt out). Pass `target:"AppName"` on `type`/`key`/`paste` to verify that app is frontmost first (re-activates once, else `E_WRONG_FOCUS`) — prevents typing into the wrong window.
70
71	Coordinates are in points everywhere (input and the `point*` fields); `scale` tells you the pixel↔point ratio. `screenshot` `region`/`maxWidth` crop and downscale to cut image tokens; add `cropOffset` to coordinates read off a cropped image to map back to the full sc

People who liked this also viewed

6 picks

Type

computer-use

59stars11forks

View on GitHub

How to install computer-use?

tykimos/cheliped-skills/computer-use

$npx -y skills add tykimos/cheliped-skills --skill computer-use

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Cheliped Computer Use (macOS)
2
3	Drive the local Mac from the terminal. One CLI takes a JSON array of commands and returns JSON — the same shape as the `browser` skill.
4
5	## Output Envelope
6
7	Output is always a JSON array, one envelope per command, compact by default:
8
9	```json
10	[{ "cmd": "screenshot", "ok": true, "result": { "path": "...", "scale": 2 } },
11	{ "cmd": "type", "ok": false, "error": { "code": "E_NO_PERMISSION", "message": "..." } }]
12	```
13
14	Each command runs even if a prior one fails (per-command isolation). Error `code` is one of `E_NO_PERMISSION` (Accessibility off), `E_NO_CLICLICK`, `E_WRONG_FOCUS`, `E_DISABLED` (escape hatch not enabled), `E_BAD_ARG`, `E_UNKNOWN_CMD`, `E_TIMEOUT`, `E_UNKNOWN`.
15
16	Flags (before the JSON arg): `--pretty` indents output; `--allow-shell` enables the `run-shell` / `run-applescript` escape hatches (off by default).
17
18	## Setup
19
20	Built-ins (`osascript`, `screencapture`) need no install. Mouse control needs cliclick:
21
22	```bash
23	brew install cliclick
24	```
25
26	Permissions: the terminal/agent process must be granted Accessibility and Screen Recording in System Settings → Privacy & Security. The CLI now preflights Accessibility and returns `E_NO_PERMISSION` (instead of a silent no-op) when keyboard/mouse input would not be delivered; screenshots that come back near-uniform get a `warning` hinting at missing Screen Recording. Grant the permissions, then restart the terminal (permissions only apply to a freshly launched process).
27
28	## Core Workflow: Observe-Act Loop
29
30	```bash
31	# 1. See the screen
32	node scripts/computer-cli.mjs '[{"cmd":"screenshot","args":["/tmp/now.png"]}]'
33
34	# 2. Act on what you saw (read the screenshot, then click / type)
35	node scripts/computer-cli.mjs '[{"cmd":"click","args":[640,400]},{"cmd":"type","args":["hello"]}]'
36	```
37
38	Read the screenshot with the Read tool, decide coordinates, then act. Re-screenshot to confirm.
39
40	## Commands
41
42	Each row's `Returns` is the `result` object inside the envelope.
43
44	\| Command \| Args \| Returns \|
45	\|---------\|------\|---------\|
46	\| `screenshot` \| `["path"]` (optional); fields `region:[x,y,w,h]`, `maxWidth:n` \| `{ path, pixelWidth, pixelHeight, pointWidth, pointHeight, scale, cropOffset?, warning? }` \|
47	\| `screen-size` \| none \| `{ width, height, unit:"point", scale }` \|
48	\| `mouse-pos` \| none \| `{ x, y, unit:"point" }` (needs cliclick) \|
49	\| `move` \| `[x, y]` \| `{ x, y }` (cliclick) \|
50	\| `click` \| `[x, y]` (optional → current pos) \| `{}` (cliclick) \|
51	\| `double-click` \| `[x, y]` (optional) \| `{}` (cliclick) \|
52	\| `right-click` \| `[x, y]` (optional) \| `{}` (cliclick) \|
53	\| `drag` \| `[x1, y1, x2, y2]` \| `{}` (cliclick) \|
54	\| `scroll` \| `[amount]` (+ up / − down; `\|amount\|/5` pages) \| `{ pages }` \|
55	\| `type` \| `["text"]`; fields `target:"App"`, `noClipboard:true` \| `{ method:"keystroke"\\|"paste" }` \|
56	\| `paste` \| `["text"]` (optional → current clipboard); field `target` \| `{ method:"paste" }` \|
57	\| `key` \| `["combo"]` e.g. `"cmd c"`; field `target:"App"` \| `{ combo }` \|
58	\| `launch` \| `["AppName"]` \| `{ app }` \|
59	\| `activate` \| `["AppName"]` \| `{ app }` \|
60	\| `quit` \| `["AppName"]` \| `{ app }` \|
61	\| `open` \| `["url-or-path"]` \| `{ target }` \|
62	\| `list-apps` \| none \| `{ apps: [...] }` — visible running apps \|
63	\| `frontmost` \| none \| `{ app }` — active app \|
64	\| `notify` \| `["text", "title"]` \| `{}` — banner notification \|
65	\| `run-applescript` \| `["script"]` (needs `--allow-shell`) \| `{ result, audited }` \|
66	\| `run-shell` \| `["command"]` (needs `--allow-shell`) \| `{ stdout, audited }` \|
67	\| `wait` \| `[ms]` \| `{}` \|
68
69	Typing (IME-safe): `type` sends ASCII as keystrokes but routes non-ASCII / multiline text (한글, 漢字, emoji, newlines) through a clipboard paste so it lands intact in CJK/IME fields — it saves and restores your clipboard (pass `noClipboard:true` to opt out). Pass `target:"AppName"` on `type`/`key`/`paste` to verify that app is frontmost first (re-activates once, else `E_WRONG_FOCUS`) — prevents typing into the wrong window.
70
71	Coordinates are in points everywhere (input and the `point*` fields); `scale` tells you the pixel↔point ratio. `screenshot` `region`/`maxWidth` crop and downscale to cut image tokens; add `cropOffset` to coordinates read off a cropped image to map back to the full sc

People who liked this also viewed

6 picks

Type