hyperpod-slurm-debugger

Name: hyperpod-slurm-debugger
Author: awslabs

843stars132forks

How to install hyperpod-slurm-debugger?

awslabs/agent-plugins/hyperpod-slurm-debugger

$npx -y skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger

Installs into the current project.

Summary

Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd no

Files · 1

View on GitHub

SKILL.md

1	# HyperPod Slurm Debugger
2
3	Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on
4	HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.
5	For remediation, link to the official AWS or Slurm documentation.
6
7	## When to invoke
8
9	Invoke when the user reports any of the symptoms in the [decision table](#decision-table).
10
11	## When NOT to invoke
12
13	- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
14	- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.
15	- NCCL training-hang investigation — invoke `hyperpod-nccl`.
16	- Node unreachable via SSM — invoke `hyperpod-ssm`.
17
18	## Constraints
19
20	- Read-only. Do not run, recommend, or print state-mutating commands.
21	- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
22	- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation
23	drifts the live state from the IaC plan.
24
25	Canonical recovery URLs:
26	[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).
27
28	## Prerequisites
29
30	- AWS CLI v2, authenticated for the target account and region with permissions:
31	- `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
32	- `ssm:StartSession` on the HyperPod-created SSM document
33	- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
34	installed locally.
35	- `jq` ≥ 1.6.
36	- `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`
37	returns empty stdout intermittently with `Cannot perform start session: EOF` and every
38	check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian /
39	Ubuntu / macOS. Script exits at prerequisite check if missing.
40
41	## Procedure
42
43	### Step 1 — Collect inputs
44
45	Ask the user for:
46
47	1. HyperPod cluster name (not Slurm partition name).
48	2. AWS region.
49	3. Optional: a specific Slurm node name.
50
51	### Step 2 — Confirm orchestrator
52
53	```bash
54	aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
55	--query 'Orchestrator' --output json
56	```
57
58	If `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).
59
60	### Step 3 — Run the diagnostic script
61
62	```bash
63	bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
64	# Scope to a node:
65	bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>
66	```
67
68	Relay the script output to the user verbatim.
69
70	### Step 4 — Map findings → docs
71
72	For each finding, look up the section in the [decision table](#decision-table) and link
73	the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.
74
75	## Decision table
76
77	\| Symptom (`sinfo -o "%N %T %30E"` or script finding) \| Section \|
78	\| ----------------------------------------------------------- \| ------------------------------------------------------ \|
79	\| Node state = `down` or `down*`, reason other than below \| [A: Node Down](#a-node-down) \|
80	\| Node state = `down*`, Reason = `Node unexpectedly rebooted` \| [B: Unexpected Reboot](#b-unexpected-reboot) \|
81	\| Jobs `PENDING` with `REASON=Resources` while nodes are idle \| [C: Controller State](#c-controller-state) \|
82	\| Jobs stuck `COMPLETING` after node replacement \| [C: Controller State](#c-controller-state) \|
83	\| `scontrol ping` returns `DOWN` for the controller \| [C: Controller State](#c-controller-state) \|
84	\| GRES (GPU) counts incorrect or not released \| [C: Controller State](#c-controller-state) \|
85	\| `state=fail` issued but no recovery occurred \| [D: Action Reason Mismatch](#d-action-reason-mismatch) \|
86	\| Accounting errors or RPC errors mentioning `dbd` \| [C: Controller State](#c-controller-state) (slurmdbd) \|
87	\| `slurm.conf` edited; new partitions or nodes not visible \| [C: Controller State](#c-controller-state) (config) \|
88	\| Job exited on a hardware failure but did not restart \| [E: Auto-resume](#e-auto-resume) \|
89
90	## Defaults
91
92	\| Beh

People who liked this also viewed

6 picks

Type

hyperpod-slurm-debugger

843stars132forks

View on GitHub

How to install hyperpod-slurm-debugger?

awslabs/agent-plugins/hyperpod-slurm-debugger

$npx -y skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# HyperPod Slurm Debugger
2
3	Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on
4	HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.
5	For remediation, link to the official AWS or Slurm documentation.
6
7	## When to invoke
8
9	Invoke when the user reports any of the symptoms in the [decision table](#decision-table).
10
11	## When NOT to invoke
12
13	- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
14	- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.
15	- NCCL training-hang investigation — invoke `hyperpod-nccl`.
16	- Node unreachable via SSM — invoke `hyperpod-ssm`.
17
18	## Constraints
19
20	- Read-only. Do not run, recommend, or print state-mutating commands.
21	- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
22	- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation
23	drifts the live state from the IaC plan.
24
25	Canonical recovery URLs:
26	[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).
27
28	## Prerequisites
29
30	- AWS CLI v2, authenticated for the target account and region with permissions:
31	- `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
32	- `ssm:StartSession` on the HyperPod-created SSM document
33	- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
34	installed locally.
35	- `jq` ≥ 1.6.
36	- `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`
37	returns empty stdout intermittently with `Cannot perform start session: EOF` and every
38	check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian /
39	Ubuntu / macOS. Script exits at prerequisite check if missing.
40
41	## Procedure
42
43	### Step 1 — Collect inputs
44
45	Ask the user for:
46
47	1. HyperPod cluster name (not Slurm partition name).
48	2. AWS region.
49	3. Optional: a specific Slurm node name.
50
51	### Step 2 — Confirm orchestrator
52
53	```bash
54	aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
55	--query 'Orchestrator' --output json
56	```
57
58	If `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).
59
60	### Step 3 — Run the diagnostic script
61
62	```bash
63	bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
64	# Scope to a node:
65	bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>
66	```
67
68	Relay the script output to the user verbatim.
69
70	### Step 4 — Map findings → docs
71
72	For each finding, look up the section in the [decision table](#decision-table) and link
73	the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.
74
75	## Decision table
76
77	\| Symptom (`sinfo -o "%N %T %30E"` or script finding) \| Section \|
78	\| ----------------------------------------------------------- \| ------------------------------------------------------ \|
79	\| Node state = `down` or `down*`, reason other than below \| [A: Node Down](#a-node-down) \|
80	\| Node state = `down*`, Reason = `Node unexpectedly rebooted` \| [B: Unexpected Reboot](#b-unexpected-reboot) \|
81	\| Jobs `PENDING` with `REASON=Resources` while nodes are idle \| [C: Controller State](#c-controller-state) \|
82	\| Jobs stuck `COMPLETING` after node replacement \| [C: Controller State](#c-controller-state) \|
83	\| `scontrol ping` returns `DOWN` for the controller \| [C: Controller State](#c-controller-state) \|
84	\| GRES (GPU) counts incorrect or not released \| [C: Controller State](#c-controller-state) \|
85	\| `state=fail` issued but no recovery occurred \| [D: Action Reason Mismatch](#d-action-reason-mismatch) \|
86	\| Accounting errors or RPC errors mentioning `dbd` \| [C: Controller State](#c-controller-state) (slurmdbd) \|
87	\| `slurm.conf` edited; new partitions or nodes not visible \| [C: Controller State](#c-controller-state) (config) \|
88	\| Job exited on a hardware failure but did not restart \| [E: Auto-resume](#e-auto-resume) \|
89
90	## Defaults
91
92	\| Beh

People who liked this also viewed

6 picks

Type