device-interaction
starsforks
How to install device-interaction?
superagents-lab/xcode27-skills/device-interaction
$npx -y skills add superagents-lab/xcode27-skills --skill device-interactionVerify iOS app behavior on device or simulator via screenshots, UI hierarchy, and touch interactions.
SKILL.md
| 1 | # Device Interaction |
| 2 | |
| 3 | TRIGGER when: user asks to verify/test/check if the app works on device, after implementing a UI-affecting feature that needs device verification, user says "does it work", "test this", "check on device", user reports UI doesn't work as expected, need to debug touch/interaction issues. |
| 4 | DO NOT TRIGGER when: user asks about unit tests only, build-only requests without device testing, code review without device testing, simulator configuration questions, changes that don't affect UI (e.g. comments, refactors, non-UI logic). |
| 5 | |
| 6 | --- |
| 7 | |
| 8 | # For the Main Agent |
| 9 | |
| 10 | **This is a SUBAGENT skill.** Invoke it via the Agent tool when device verification is needed. |
| 11 | |
| 12 | ``` |
| 13 | Agent tool: |
| 14 | - subagent_type: "general-purpose" |
| 15 | - description: "Verify login feature works" |
| 16 | - prompt: "Using the device-interaction skill, verify that the login feature works correctly on session <device-interaction-session>. Launch the app, capture screenshot and UI hierarchy, check that the login button is visible and tappable, and report if the implementation is working correctly." |
| 17 | ``` |
| 18 | |
| 19 | **After implementing a UI-affecting feature, invoke this skill to verify the implementation works on a device.** |
| 20 | |
| 21 | ## Session Lifecycle |
| 22 | |
| 23 | ``` |
| 24 | DeviceInteractionStartSession (do this early, runs in the background) |
| 25 | → DeviceInteractionInstallAndRun (after each code change; includes building) |
| 26 | → DeviceEventSynthesize (interact + observe, repeatable) |
| 27 | → DeviceInteractionEndSession (when done — keeping sessions open is resource-heavy) |
| 28 | ``` |
| 29 | |
| 30 | ## DeviceInteractionStartSession tool |
| 31 | |
| 32 | ### Device Discovery |
| 33 | |
| 34 | When opening a new device interaction session, pass a device identifier to select a device, or omit it to use the current destination. Pass any non-matching value to get a list of available targets. |
| 35 | |
| 36 | ## DeviceInteractionInstallAndRun tool |
| 37 | |
| 38 | ### Optional Parameters |
| 39 | |
| 40 | - `commandLineArguments` — arguments passed to the app at launch. Use `$(inherited)` as a token to preserve the scheme's existing arguments (e.g. `["$(inherited)", "--reset-state"]` to add an extra argument at the end). |
| 41 | - `environmentVariables` — key/value pairs set in the app's environment at launch. Use `"$(inherited)"` as a key to preserve the scheme's existing environment variables (e.g. `{"$(inherited)": "", "DEBUG_MODE": "1"}`). |
| 42 | |
| 43 | Omit both parameters to leave the scheme's arguments and environment unchanged. |
| 44 | |
| 45 | **Prefer these parameters over editing the scheme directly.** They are applied only for that one run and have no lasting effect on the user's configuration. |
| 46 | |
| 47 | --- |
| 48 | |
| 49 | # For the Subagent |
| 50 | |
| 51 | **ALWAYS** report UI issues that might be caused by code: overlapping or unreadable text, unexpectedly cropped image/text, wrong colors etc. |
| 52 | |
| 53 | ## DeviceEventSynthesize tool |
| 54 | |
| 55 | This tool allows performing an interaction and observing the state of a device. |
| 56 | |
| 57 | ## Reading Hierarchy Files |
| 58 | |
| 59 | The hierarchy files include calculated center positions for each element: |
| 60 | |
| 61 | ``` |
| 62 | UIView {{100, 200}, {50, 30}}, center: {125.0, 215.0} |
| 63 | UIButton "Login" {{110, 205}, {30, 20}}, center: {125.0, 215.0} |
| 64 | ``` |
| 65 | |
| 66 | - `{100, 200}` - origin position |
| 67 | - `{50, 30}` - width and height |
| 68 | - `center: {125.0, 215.0}` - calculated center point (best for tapping) |
| 69 | |
| 70 | **Always prefer the center coordinates for touch events.** |
| 71 | |
| 72 | ## Interaction Command Syntax |
| 73 | |
| 74 | The `interactionCommand` parameter accepts a command syntax: |
| 75 | |
| 76 | | Command | Description | |
| 77 | |---|---| |
| 78 | | `t <x> <y> [duration]` | Tap at coordinates with optional hold duration | |
| 79 | | `d <x> <y>` | Double tap | |
| 80 | | `t <x1> <y1> f <x2> <y2> [duration]` | Swipe from (x1,y1) to (x2,y2) | |
| 81 | | `b h/p/u/d [duration]` | Hardware button: h=Home, p=Power, u=VolUp, d=VolDown | |
| 82 | | `sender keyboard kbd <text>` | Type text; **must be the last command in the chain** — all content after `kbd ` is taken verbatim (multiple spaces preserved). For special characters use `\u{XXXX}` Unicode escapes: `\u{000A}` (return/newline), `\u{0009}` (tab) | |
| 83 | | `w duration` | Wait for a duration without any work | |
| 84 | | `orientation faceDown/faceUp/landscapeLeft/landscapeRight/portrait/portraitUpsideDown` | Set device orientation | |
| 85 | |
| 86 | **Examples:** |
| 87 | - `"t 100 200"` - Tap at (100, 200) |
| 88 | - `"d 200 300"` - Double tap at (200, 300) |
| 89 | - `"t 200 600 f 200 200 0.3"` - Swipe up (scroll to the content below) |
| 90 | - `"t 200 200 f 200 600 0.3"` - Swipe down (scroll to the content above) |
| 91 | - `"b h"` - Press home button |
| 92 | - `"b h b h"` - Press home button twice to go to the app switcher |
| 93 | - `"b h w 0 b h"` - Wake and unlock a device (non-passcode devices only) |
| 94 | - `"sender keyboard kbd hello world"` - Type text with spaces |
| 95 | - `"sender keyboard kbd hello world"` - Type text preserving multiple spaces |
| 96 | - `"sender keyboard kbd submit\u{000A}"` - Type text then press Return/submit |
| 97 | - `"w 0.3"` - Wait for 0.3s |
| 98 | - `"orientation landscapeLeft"` - Rotate device to landscape |
| 99 | |
| 100 | ## Standard Subagent Workflow |
| 101 | |
| 102 | Before any interaction, always capture and read the hie |
People who liked this also viewed
6 picksType
- hyperframes-cliUse the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes, single or batch render, publish, cloud, SkillsJul 2026
- request-refactor-planSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- analyze-projectRigor Analyze / Rigor Audit read-only skill for deep learning research repositories. Use when the user wants to read and understand a repository, inspect model structure and training or inference entrSkillsMay 2026
- explore-codeRigor Improve implementation leaf skill for auditable candidate implementation in deep learning research repositories. Use when the researcher explicitly authorizes exploratory work on an isolated braSkillsMay 2026
- ubiquitous-languageSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- scaffold-exercisesSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
