$npx -y skills add awslabs/agent-plugins --skill hyperpod-version-checkerCheck and compare software component versions on SageMaker HyperPod cluster nodes - NVIDIA drivers, CUDA toolkit, cuDNN, NCCL, EFA, AWS OFI NCCL, GDRCopy, MPI, Neuron SDK (Trainium/Inferentia), Python, and PyTorch. Use when checking component versions, verifying CUDA/driver compa
| 1 | # HyperPod Version Checker |
| 2 | |
| 3 | Upload to cluster nodes via `hyperpod-ssm` skill, then execute. |
| 4 | |
| 5 | ## Usage |
| 6 | |
| 7 | ```bash |
| 8 | # Text report to console + file |
| 9 | bash hyperpod_check_versions.sh |
| 10 | |
| 11 | # JSON only to stdout (text report still saved to file) — best for piping/parsing |
| 12 | bash hyperpod_check_versions.sh --json |
| 13 | |
| 14 | # Custom output file |
| 15 | bash hyperpod_check_versions.sh --output /tmp/versions.txt |
| 16 | |
| 17 | # No color (for logging) |
| 18 | bash hyperpod_check_versions.sh --no-color |
| 19 | ``` |
| 20 | |
| 21 | Output file: `component_versions_<hostname>_<timestamp>.txt` (default) |
| 22 | |
| 23 | ## What It Checks |
| 24 | |
| 25 | | Component | Detection Method | Applicable When | |
| 26 | | ----------------- | ----------------------------------------------- | --------------------------------------------- | |
| 27 | | NVIDIA Driver | `nvidia-smi` | GPU instances (p3/p4/p5/g5) | |
| 28 | | CUDA Toolkit | `nvcc`, `/usr/local/cuda` symlink | GPU instances | |
| 29 | | cuDNN | Header file, packages | GPU instances doing deep learning | |
| 30 | | NCCL | Library filename, header, packages | Distributed GPU training | |
| 31 | | EFA | `/opt/amazon/efa_installed_packages`, `fi_info` | EFA-capable instances (p4d/p4de/p5/trn1/trn2) | |
| 32 | | AWS OFI NCCL | `efa_installed_packages`, library search | EFA + NCCL workloads | |
| 33 | | GDRCopy | rpm/dpkg, kernel module | GPU instances with RDMA (p4d+/p5) | |
| 34 | | MPI | `mpirun`, `/opt/amazon/openmpi` | Distributed training | |
| 35 | | Neuron SDK | `neuronx-cc`, `neuron-ls`, packages | Trainium/Inferentia (trn1/trn2/inf1/inf2) | |
| 36 | | Python/PyTorch | `python3`, `torch` import | ML workloads | |
| 37 | | Container runtime | `docker`, `containerd`, `kubectl`, `nvidia-ctk` | EKS clusters | |
| 38 | |
| 39 | ## Multi-Node Comparison |
| 40 | |
| 41 | Run on each node individually via the `hyperpod-ssm` skill. With `--json`, stdout is clean JSON for easy diffing. |
| 42 | |
| 43 | ## Compatibility Reference |
| 44 | |
| 45 | The script automatically analyzes CUDA/driver compatibility. For reference: |
| 46 | |
| 47 | | Driver Series | Supported CUDA | |
| 48 | | ------------- | ----------------------------- | |
| 49 | | 580+ | 13.x, 12.x, 11.x | |
| 50 | | 570+ | 12.8+ (Blackwell), 12.x, 11.x | |
| 51 | | 545+ | 12.3-12.7, 11.x | |
| 52 | | 525-535 | 12.0-12.2, 11.x | |
| 53 | | 450+ | 11.x only | |
| 54 | |
| 55 | NCCL: Use 2.18+ for CUDA 12.x, 2.12+ for CUDA 11.x. Must be consistent across all nodes. |
| 56 | |
| 57 | | EFA Installer | AWS OFI NCCL | |
| 58 | | ------------- | --------------------- | |
| 59 | | 1.29+ | v1.7.3+ (recommended) | |
| 60 | | 1.26-1.28 | v1.7.0-v1.7.2 | |
| 61 | | 1.20-1.25 | v1.6.0+ | |