hyperpod-ssm

Name: hyperpod-ssm
Author: awslabs

843stars132forks

How to install hyperpod-ssm?

awslabs/agent-plugins/hyperpod-ssm

$npx -y skills add awslabs/agent-plugins --skill hyperpod-ssm

Installs into the current project.

Summary

Remote command execution and file transfer on SageMaker HyperPod cluster nodes via AWS Systems Manager (SSM). This is the primary interface for accessing HyperPod nodes — direct SSH is not available. Use when any skill, workflow, or user request needs to execute commands on clust

Files · 1

View on GitHub

SKILL.md

1	# HyperPod SSM Access
2
3	## Prerequisites
4
5	- `aws` CLI v2, authenticated for the target account/Region.
6	- `session-manager-plugin` — installed alongside the AWS CLI.
7	- `jq` — the scripts build JSON payloads with it.
8	- `unbuffer` (from the `expect` package) — wraps `aws ssm start-session` with a PTY so the session-manager-plugin flushes stdout instead of racing to close. Without it, calls intermittently return empty output with `Cannot perform start session: EOF` even when the command ran. Install with `sudo yum install expect`, `sudo apt install expect`, or `brew install expect`. `ssm-exec.sh` detects and uses it automatically; falls back with a warning if missing.
9
10	## SSM Target Format
11
12	Target: `sagemaker-cluster:<CLUSTER_ID>_<GROUP_NAME>-<INSTANCE_ID>`
13
14	- `CLUSTER_ID`: Last segment of cluster ARN (NOT the cluster name). Extract via `get-cluster-info.sh`.
15	- `GROUP_NAME`: Instance group name — retrieve via `list-nodes.sh`.
16	- `INSTANCE_ID`: EC2 instance ID (e.g., `i-0123456789abcdef0`)
17
18	## Scripts
19
20	Three scripts under `scripts/`. Resolve cluster info and nodes once, then execute per node.
21
22	### get-cluster-info.sh — Resolve cluster name → ID (call once)
23
24	```bash
25	scripts/get-cluster-info.sh CLUSTER_NAME [--region REGION]
26	# Output: {"cluster_id":"...","cluster_arn":"...","cluster_name":"...","region":"..."}
27	```
28
29	### list-nodes.sh — List all nodes with pagination (call once)
30
31	```bash
32	scripts/list-nodes.sh CLUSTER_NAME [--region REGION] [--instance-group GROUP] [--instance-id ID]
33	# Output: JSON array of ClusterNodeSummaries (InstanceId, InstanceGroupName, InstanceStatus, etc.)
34	```
35
36	`list-cluster-nodes` paginates at 100 nodes. This script handles pagination automatically.
37
38	### ssm-exec.sh — Execute command on a node (call per node)
39
40	```bash
41	# Execute — with pre-built target
42	scripts/ssm-exec.sh --target "sagemaker-cluster:CLUSTERID_GROUP-INSTANCEID" 'command' [--region REGION]
43
44	# Execute — with parts
45	scripts/ssm-exec.sh --cluster-id ID --group GROUP --instance-id INSTANCE_ID 'command' [--region REGION]
46
47	# Upload
48	scripts/ssm-exec.sh --target TARGET --upload LOCAL_PATH REMOTE_PATH [--region REGION]
49
50	# Read remote file
51	scripts/ssm-exec.sh --target TARGET --read REMOTE_PATH [--region REGION]
52	```
53
54	## Running Commands Across Many Nodes
55
56	SSM `start-session` rate limit: 3 TPS per account. Plan batch size and delay accordingly.
57
58	`aws ssm send-command` does NOT support `sagemaker-cluster:` targets — only `start-session` works.
59
60	## Manual SSM Commands
61
62	When the scripts aren't suitable, use `aws ssm start-session` directly with `AWS-StartNonInteractiveCommand`. Wrap every invocation in `unbuffer` — without it, stdout is intermittently empty (see Prerequisites).
63
64	```bash
65	cat > /tmp/cmd.json << 'EOF'
66	{"command": ["bash -c 'echo hello && whoami'"]}
67	EOF
68
69	unbuffer aws ssm start-session \
70	--target sagemaker-cluster:{CLUSTER_ID}_{GROUP_NAME}-{INSTANCE_ID} \
71	--region REGION \
72	--document-name AWS-StartNonInteractiveCommand \
73	--parameters file:///tmp/cmd.json
74	```
75
76	- Always use a JSON file for `--parameters` — inline parameters break with special characters.
77	- The document's `command` parameter is argv, not shell input. Wrap multi-statement scripts in `bash -c '...'` so pipes, semicolons, and redirects evaluate.
78
79	## Common Diagnostic Commands
80
81	\| Task \| Command \|
82	\| ---------------- \| -------------------------------------------------------------- \|
83	\| Lifecycle logs \| `cat /var/log/provision/provisioning.log` \|
84	\| Memory \| `free -h` \|
85	\| Disk/mounts \| `df -h && lsblk` \|
86	\| GPU status \| `nvidia-smi` \|
87	\| GPU memory \| `nvidia-smi --query-gpu=memory.used,memory.total --format=csv` \|
88	\| EFA/network \| `fi_info -p efa` \|
89	\| CloudWatch agent \| `sudo systemctl status amazon-cloudwatch-agent` \|
90	\| Top processes \| `ps aux --sort=-%mem \\| head -20` \|
91
92	## Key Details
93
94	- Default SSM non-interactive user is `root`.
95	- SSM rate limit: 3 TPS per account.
96	- For interactive sessions (rare), omit `--document-name` to get a shell.
97	- Interactive commands (vim, top) are not supported via `AWS-Sta

People who liked this also viewed

6 picks

Type

hyperpod-ssm

843stars132forks

View on GitHub

How to install hyperpod-ssm?

awslabs/agent-plugins/hyperpod-ssm

$npx -y skills add awslabs/agent-plugins --skill hyperpod-ssm

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# HyperPod SSM Access
2
3	## Prerequisites
4
5	- `aws` CLI v2, authenticated for the target account/Region.
6	- `session-manager-plugin` — installed alongside the AWS CLI.
7	- `jq` — the scripts build JSON payloads with it.
8	- `unbuffer` (from the `expect` package) — wraps `aws ssm start-session` with a PTY so the session-manager-plugin flushes stdout instead of racing to close. Without it, calls intermittently return empty output with `Cannot perform start session: EOF` even when the command ran. Install with `sudo yum install expect`, `sudo apt install expect`, or `brew install expect`. `ssm-exec.sh` detects and uses it automatically; falls back with a warning if missing.
9
10	## SSM Target Format
11
12	Target: `sagemaker-cluster:<CLUSTER_ID>_<GROUP_NAME>-<INSTANCE_ID>`
13
14	- `CLUSTER_ID`: Last segment of cluster ARN (NOT the cluster name). Extract via `get-cluster-info.sh`.
15	- `GROUP_NAME`: Instance group name — retrieve via `list-nodes.sh`.
16	- `INSTANCE_ID`: EC2 instance ID (e.g., `i-0123456789abcdef0`)
17
18	## Scripts
19
20	Three scripts under `scripts/`. Resolve cluster info and nodes once, then execute per node.
21
22	### get-cluster-info.sh — Resolve cluster name → ID (call once)
23
24	```bash
25	scripts/get-cluster-info.sh CLUSTER_NAME [--region REGION]
26	# Output: {"cluster_id":"...","cluster_arn":"...","cluster_name":"...","region":"..."}
27	```
28
29	### list-nodes.sh — List all nodes with pagination (call once)
30
31	```bash
32	scripts/list-nodes.sh CLUSTER_NAME [--region REGION] [--instance-group GROUP] [--instance-id ID]
33	# Output: JSON array of ClusterNodeSummaries (InstanceId, InstanceGroupName, InstanceStatus, etc.)
34	```
35
36	`list-cluster-nodes` paginates at 100 nodes. This script handles pagination automatically.
37
38	### ssm-exec.sh — Execute command on a node (call per node)
39
40	```bash
41	# Execute — with pre-built target
42	scripts/ssm-exec.sh --target "sagemaker-cluster:CLUSTERID_GROUP-INSTANCEID" 'command' [--region REGION]
43
44	# Execute — with parts
45	scripts/ssm-exec.sh --cluster-id ID --group GROUP --instance-id INSTANCE_ID 'command' [--region REGION]
46
47	# Upload
48	scripts/ssm-exec.sh --target TARGET --upload LOCAL_PATH REMOTE_PATH [--region REGION]
49
50	# Read remote file
51	scripts/ssm-exec.sh --target TARGET --read REMOTE_PATH [--region REGION]
52	```
53
54	## Running Commands Across Many Nodes
55
56	SSM `start-session` rate limit: 3 TPS per account. Plan batch size and delay accordingly.
57
58	`aws ssm send-command` does NOT support `sagemaker-cluster:` targets — only `start-session` works.
59
60	## Manual SSM Commands
61
62	When the scripts aren't suitable, use `aws ssm start-session` directly with `AWS-StartNonInteractiveCommand`. Wrap every invocation in `unbuffer` — without it, stdout is intermittently empty (see Prerequisites).
63
64	```bash
65	cat > /tmp/cmd.json << 'EOF'
66	{"command": ["bash -c 'echo hello && whoami'"]}
67	EOF
68
69	unbuffer aws ssm start-session \
70	--target sagemaker-cluster:{CLUSTER_ID}_{GROUP_NAME}-{INSTANCE_ID} \
71	--region REGION \
72	--document-name AWS-StartNonInteractiveCommand \
73	--parameters file:///tmp/cmd.json
74	```
75
76	- Always use a JSON file for `--parameters` — inline parameters break with special characters.
77	- The document's `command` parameter is argv, not shell input. Wrap multi-statement scripts in `bash -c '...'` so pipes, semicolons, and redirects evaluate.
78
79	## Common Diagnostic Commands
80
81	\| Task \| Command \|
82	\| ---------------- \| -------------------------------------------------------------- \|
83	\| Lifecycle logs \| `cat /var/log/provision/provisioning.log` \|
84	\| Memory \| `free -h` \|
85	\| Disk/mounts \| `df -h && lsblk` \|
86	\| GPU status \| `nvidia-smi` \|
87	\| GPU memory \| `nvidia-smi --query-gpu=memory.used,memory.total --format=csv` \|
88	\| EFA/network \| `fi_info -p efa` \|
89	\| CloudWatch agent \| `sudo systemctl status amazon-cloudwatch-agent` \|
90	\| Top processes \| `ps aux --sort=-%mem \\| head -20` \|
91
92	## Key Details
93
94	- Default SSM non-interactive user is `root`.
95	- SSM rate limit: 3 TPS per account.
96	- For interactive sessions (rare), omit `--document-name` to get a shell.
97	- Interactive commands (vim, top) are not supported via `AWS-Sta

People who liked this also viewed

6 picks

Type