hyperpod-performance-debugger

Name: hyperpod-performance-debugger
Author: awslabs

843stars132forks

How to install hyperpod-performance-debugger?

awslabs/agent-plugins/hyperpod-performance-debugger

$npx -y skills add awslabs/agent-plugins --skill hyperpod-performance-debugger

Installs into the current project.

Summary

Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-

Files · 1

View on GitHub

SKILL.md

1	# HyperPod Performance Debugger
2
3	1. Uneven NCCL performance across nodes — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
4	2. Poor filesystem performance — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.
5
6	## Scope and delegation
7
8	Route findings outside the two in-scope scenarios to the owner skill below.
9
10	\| Concern observed \| Route to \|
11	\| ---------------------------------------------------------------------- \| ------------------------------------------------------------ \|
12	\| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace \| `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) \|
13	\| `Cannot allocate memory` at `os.fork()`, root volume exhausted \| `hyperpod-node-debugger` (§ I Resource Exhaustion) \|
14	\| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock \| `hyperpod-nccl` \|
15	\| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes \| `hyperpod-version-checker` \|
16	\| EFA self-referencing security-group rule missing — single node \| `hyperpod-node-debugger` § A (EFA / Security Group) \|
17	\| EFA self-referencing security-group rule missing — cluster-wide \| `hyperpod-cluster-debugger` § A (EFA Health Checks) \|
18	\| Slurm node state changes (drain / resume / reboot) \| `hyperpod-slurm-debugger` \|
19	\| Diagnostic bundle for AWS Support \| `hyperpod-issue-report` \|
20	\| Shell access on a node \| `hyperpod-ssm` \|
21
22	## Operating policy
23
24	- Read-only. Print commands the customer runs; do not execute commands that modify state.
25	- Container vs host version comparisons go through `hyperpod-version-checker`.
26	- Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in `hyperpod-node-debugger` § G.
27
28	## Workflow
29
30	1. Confirm the symptom is uneven NCCL or poor filesystem performance. If neither, route to the matching sibling skill above.
31	2. Run `scripts/perf-snapshot.sh` (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.
32	3. For each `[CONCERN]` line in the script output, open the matching section below and read the supporting reference.
33	4. After the per-incident diagnosis, recommend the HyperPod platform health features in [§ Continuous health coverage](#continuous-health-coverage) so the customer gets ongoing protection.
34
35	## Step 1: Run the snapshot
36
37	```bash
38	bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
39
40	# Scope to one suspect node:
41	bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>
42	```
43
44	The script samples one node by default. It collects host-side data via `hyperpod-ssm`: `nvidia-smi` output (temperature, SM clocks, PCIe link width, ECC, NVLink, `topo -m`), recent `dmesg` Xid lines, EFA port state and `fi_info` provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, `df -h` / `lfs df -h` per mount, host iowait, `/dev/shm` size, and root-volume usage. All read-only.
45
46	Tags: `[OK]` healthy · `[CONCERN]` signal worth investigating (carries a `→` pointer to the owner skill) · `[INFO]` informational.
47
48	Host vs container scope. The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of `unknown` is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via `hyperpod-version-checker` before drawing conclusions.
49
50	## Step 2: Match signal → section
51
52	\| Observation \| Section

People who liked this also viewed

6 picks

Type

hyperpod-performance-debugger

843stars132forks

View on GitHub

How to install hyperpod-performance-debugger?

awslabs/agent-plugins/hyperpod-performance-debugger

$npx -y skills add awslabs/agent-plugins --skill hyperpod-performance-debugger

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# HyperPod Performance Debugger
2
3	1. Uneven NCCL performance across nodes — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
4	2. Poor filesystem performance — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.
5
6	## Scope and delegation
7
8	Route findings outside the two in-scope scenarios to the owner skill below.
9
10	\| Concern observed \| Route to \|
11	\| ---------------------------------------------------------------------- \| ------------------------------------------------------------ \|
12	\| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace \| `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) \|
13	\| `Cannot allocate memory` at `os.fork()`, root volume exhausted \| `hyperpod-node-debugger` (§ I Resource Exhaustion) \|
14	\| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock \| `hyperpod-nccl` \|
15	\| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes \| `hyperpod-version-checker` \|
16	\| EFA self-referencing security-group rule missing — single node \| `hyperpod-node-debugger` § A (EFA / Security Group) \|
17	\| EFA self-referencing security-group rule missing — cluster-wide \| `hyperpod-cluster-debugger` § A (EFA Health Checks) \|
18	\| Slurm node state changes (drain / resume / reboot) \| `hyperpod-slurm-debugger` \|
19	\| Diagnostic bundle for AWS Support \| `hyperpod-issue-report` \|
20	\| Shell access on a node \| `hyperpod-ssm` \|
21
22	## Operating policy
23
24	- Read-only. Print commands the customer runs; do not execute commands that modify state.
25	- Container vs host version comparisons go through `hyperpod-version-checker`.
26	- Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in `hyperpod-node-debugger` § G.
27
28	## Workflow
29
30	1. Confirm the symptom is uneven NCCL or poor filesystem performance. If neither, route to the matching sibling skill above.
31	2. Run `scripts/perf-snapshot.sh` (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.
32	3. For each `[CONCERN]` line in the script output, open the matching section below and read the supporting reference.
33	4. After the per-incident diagnosis, recommend the HyperPod platform health features in [§ Continuous health coverage](#continuous-health-coverage) so the customer gets ongoing protection.
34
35	## Step 1: Run the snapshot
36
37	```bash
38	bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
39
40	# Scope to one suspect node:
41	bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>
42	```
43
44	The script samples one node by default. It collects host-side data via `hyperpod-ssm`: `nvidia-smi` output (temperature, SM clocks, PCIe link width, ECC, NVLink, `topo -m`), recent `dmesg` Xid lines, EFA port state and `fi_info` provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, `df -h` / `lfs df -h` per mount, host iowait, `/dev/shm` size, and root-volume usage. All read-only.
45
46	Tags: `[OK]` healthy · `[CONCERN]` signal worth investigating (carries a `→` pointer to the owner skill) · `[INFO]` informational.
47
48	Host vs container scope. The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of `unknown` is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via `hyperpod-version-checker` before drawing conclusions.
49
50	## Step 2: Match signal → section
51
52	\| Observation \| Section

People who liked this also viewed

6 picks

Type