hyperpod-cluster-debugger
starsforks
How to install hyperpod-cluster-debugger?
awslabs/agent-plugins/hyperpod-cluster-debugger
$npx -y skills add awslabs/agent-plugins --skill hyperpod-cluster-debuggerDiagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes
SKILL.md
| 1 | # HyperPod Cluster Debugger |
| 2 | |
| 3 | **Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). |
| 4 | |
| 5 | **Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review). |
| 6 | |
| 7 | `scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/<file>.md § <section>`. |
| 8 | |
| 9 | | Reference | Open when | |
| 10 | | ------------------------------------------------------------------------- | ------------------------------------------------------------------- | |
| 11 | | [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L) | |
| 12 | | [cluster-operations.md](references/cluster-operations.md) | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) | |
| 13 | | [cloudformation-errors.md](references/cloudformation-errors.md) | § H needs the full per-resource CFN error catalog | |
| 14 | | [capacity-planning.md](references/capacity-planning.md) | § B or `--validate` flags capacity / subnet sizing | |
| 15 | | [lifecycle-scripts.md](references/lifecycle-scripts.md) | § C points at a specific lifecycle failure | |
| 16 | | [iam-permissions.md](references/iam-permissions.md) | Full IAM policy for the diagnostic | |
| 17 | |
| 18 | --- |
| 19 | |
| 20 | ## Workflow |
| 21 | |
| 22 | 1. Collect HyperPod cluster name (not EKS name), region, exact error string. |
| 23 | 2. Run `scripts/diagnose-cluster.sh` (or `--validate` for pre-create). |
| 24 | 3. For every `[FAIL]` line, `Read` the referenced section. |
| 25 | 4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval. |
| 26 | 5. Re-run the diagnostic to confirm. |
| 27 | |
| 28 | --- |
| 29 | |
| 30 | ## Step 1: Run diagnostics |
| 31 | |
| 32 | ```bash |
| 33 | # Diagnose an existing cluster: |
| 34 | bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION> |
| 35 | |
| 36 | # Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints, |
| 37 | # optionally S3 lifecycle scripts and per-AZ capacity: |
| 38 | bash scripts/diagnose-cluster.sh --validate --region <REGION> \ |
| 39 | --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \ |
| 40 | [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge] |
| 41 | ``` |
| 42 | |
| 43 | Pass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time). |
| 44 | |
| 45 | Tags: `[PASS]` · `[FAIL]` (counted, has `→ references/...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational. |
| 46 | |
| 47 | --- |
| 48 | |
| 49 | ## Step 2: Match signal → section |
| 50 | |
| 51 | **Error messages / events:** |
| 52 | |
| 53 | | Signal | Section | |
| 54 | | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | |
| 55 | | `"EFA health checks did not run successfully"` (public-doc verbatim signal) | **[A: EFA Health Checks](#a-efa-health-checks)** | |
| 56 | | Insufficient-capacity or AZ-mismatch failure at creation | **[B: Capacity & AZ](#b-capacity--az)** | |
| 57 | | Lifecycle-script failure or timeout during provisioning | **[C: Lifecycle Scripts](#c-lifecycle-scripts)** | |
| 58 | | kubectl auth error (server asks for credentials / no API group list) | **[D: EKS Access](#d-eks-access--kubectl)** | |
| 59 | | `InService` but not all instances visible | **[E: Cluster Provisioning](#e-cluster-provisioning)** | |
| 60 | | `"Target is not connected"` / SSM errors |
People who liked this also viewed
6 picksType
- azure-diagnosticsDebug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deploySkillsJul 2026
- systematic-debuggingUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesSkillsJul 2026
- safe-debugRigor Debug / Rigor Audit skill for deep learning research work. Use when the user pastes a traceback, terminal error, CUDA OOM, checkpoint load failure, shape mismatch, NaN loss symptom, or training SkillsMay 2026
- explore-runRigor Improve / Rigor Explore run leaf skill for bounded exploratory evidence in deep learning research repositories. Use when the researcher explicitly authorizes exploratory runs such as small-subseSkillsMay 2026
- minimal-run-and-auditRigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation SkillsMay 2026
- insforge-debugUse when diagnosing problems in an InsForge project — reactive failures (SDK error object, HTTP 4xx/5xx, gateway timeout 502/503/504, edge function failure or timeout, login/OAuth/auth errors, RLS denSkillsJul 2026
