hyperpod-nccl
starsforks
How to install hyperpod-nccl?
awslabs/agent-plugins/hyperpod-nccl
$npx -y skills add awslabs/agent-plugins --skill hyperpod-ncclDiagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or memlock issues, NCCL version mismatch across pods, c
SKILL.md
| 1 | # HyperPod NCCL Debugger |
| 2 | |
| 3 | **Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation. |
| 4 | |
| 5 | Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts/nccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references/<file>.md § <section>`. Read-only. |
| 6 | |
| 7 | **Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health. |
| 8 | |
| 9 | --- |
| 10 | |
| 11 | ## Workflow |
| 12 | |
| 13 | 1. Collect cluster name, region, namespace/job (EKS), exact NCCL error string. |
| 14 | 2. Run the diagnostic (always — the output drives everything else). |
| 15 | 3. For every `[FAIL]` line, `Read` the referenced section. |
| 16 | 4. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval. |
| 17 | 5. Re-run the diagnostic to confirm. |
| 18 | |
| 19 | If a finding has no matching section, report it as a bug — do not invent a fix. |
| 20 | |
| 21 | ## Step 1: Authenticate kubectl (EKS) |
| 22 | |
| 23 | ```bash |
| 24 | EKS_ARN=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD-NAME> --region <REGION> \ |
| 25 | --query 'Orchestrator.Eks.ClusterArn' --output text) |
| 26 | EKS_NAME=$(echo "$EKS_ARN" | awk -F'/' '{print $NF}') |
| 27 | aws eks update-kubeconfig --name "$EKS_NAME" --region <REGION> |
| 28 | kubectl get nodes |
| 29 | ``` |
| 30 | |
| 31 | ## Step 2: Run the diagnostic |
| 32 | |
| 33 | ```bash |
| 34 | # Basic: |
| 35 | bash scripts/nccl-diagnose.sh --cluster <HYPERPOD-NAME> --region <REGION> |
| 36 | |
| 37 | # Scope to an EKS job/namespace: |
| 38 | bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --namespace <NS> --job <JOB> |
| 39 | |
| 40 | # Force orchestrator: |
| 41 | bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --orchestrator slurm |
| 42 | |
| 43 | # Larger hardware sample (default 3): |
| 44 | bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --sample-nodes 10 |
| 45 | |
| 46 | # Specific node only: |
| 47 | bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --node i-0abc123def456 |
| 48 | ``` |
| 49 | |
| 50 | Tags: `[PASS]` · `[FAIL]` (counted in `Issues Found`, has reference pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks training · **P1** degraded · **P2** informational. |
| 51 | |
| 52 | --- |
| 53 | |
| 54 | ## Remediation index |
| 55 | |
| 56 | Each `[FAIL]` line in the script already points directly at the right section. This table is a lookup for manual triage. |
| 57 | |
| 58 | | Finding | Section | |
| 59 | | ------------------------------------------ | --------------------------------------------------------------------------------------------------- | |
| 60 | | SG missing inbound/outbound self-reference | [operations.md § 8](references/operations.md) | |
| 61 | | Blocking NetworkPolicy / allow-all missing | [operations.md § 8](references/operations.md) | |
| 62 | | Slurm node DOWN / DRAINING / RemoveIPC | [operations.md § 7](references/operations.md) | |
| 63 | | GPU XID / SYSTEM_ERROR / hardware fault | [hyperpod-node-debugger § F / § G](../hyperpod-node-debugger/references/node-diagnostics-detail.md) | |
| 64 | | GPU row-remap / DCGM Fail / silent NaNs | [hyperpod-node-debugger § G.1.a/b](../hyperpod-node-debugger/references/node-diagnostics-detail.md) | |
| 65 | | NCCL timeout / rendezvous / straggler | [debugging-guide.md § 1](references/debugging-guide.md) | |
| 66 | | EFA configuration / not used | [debugging-guide.md § 6](references/debugging-guide.md) | |
| 67 | | EFA TCP fallback (`NET/OFI Using TCP`) | [debugging-guide.md § 13](references/debugging-guide.md) | |
| 68 | | NC |
People who liked this also viewed
6 picksType
- azure-diagnosticsDebug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deploySkillsJul 2026
- systematic-debuggingUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesSkillsJul 2026
- safe-debugRigor Debug / Rigor Audit skill for deep learning research work. Use when the user pastes a traceback, terminal error, CUDA OOM, checkpoint load failure, shape mismatch, NaN loss symptom, or training SkillsMay 2026
- explore-runRigor Improve / Rigor Explore run leaf skill for bounded exploratory evidence in deep learning research repositories. Use when the researcher explicitly authorizes exploratory runs such as small-subseSkillsMay 2026
- minimal-run-and-auditRigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation SkillsMay 2026
- insforge-debugUse when diagnosing problems in an InsForge project — reactive failures (SDK error object, HTTP 4xx/5xx, gateway timeout 502/503/504, edge function failure or timeout, login/OAuth/auth errors, RLS denSkillsJul 2026
