together-volcano

Name: together-volcano
Author: togethercomputer

32stars4forks

How to install together-volcano?

togethercomputer/skills/together-volcano

$npx -y skills add togethercomputer/skills --skill together-volcano

Installs into the current project.

Summary

Install and use the Volcano batch scheduler on a Together AI Kubernetes GPU cluster for gang scheduling. Covers installing Volcano, creating queues, submitting all-or-nothing gang-scheduled jobs (vcjobs), and verifying placement. Reach for it when a job on a Together cluster need

Files · 1

View on GitHub

SKILL.md

1	# Volcano on Together GPU clusters
2
3	Volcano is a Kubernetes-native batch scheduler. Its key property is gang scheduling: a group of pods is placed all-at-once or not at all, so distributed training never starts with only some workers running. Use it when a job needs N pods to run together or not at all.
4
5	Public cookbook: https://docs.together.ai/docs/volcano-on-gpu-clusters. Pair this skill with the `together-gpu-clusters` skill, which covers creating the cluster and configuring `kubectl`.
6
7	## Preconditions
8
9	- A Together Kubernetes GPU cluster in the `Ready` state, with `kubectl` pointed at it (`tg beta clusters get-credentials <cluster_id> --set-default-context`).
10	- GPU nodes expose `nvidia.com/gpu`; the NVIDIA device plugin is preinstalled on Together clusters. Confirm with:
11
12	```bash
13	kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
14	```
15
16	## Install
17
18	Pin the version. Do not use `master`.
19
20	```bash
21	kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
22	kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
23	```
24
25	A `default` queue is created automatically (`kubectl get queue`). Helm is an alternative: `helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace`.
26
27	## Create a queue
28
29	A queue caps and weights the resources a set of jobs can use.
30
31	```yaml
32	apiVersion: scheduling.volcano.sh/v1beta1
33	kind: Queue
34	metadata:
35	name: research
36	spec:
37	reclaimable: true # give idle capacity back to other queues
38	weight: 1 # relative share under contention
39	capability: # hard ceiling for this queue
40	nvidia.com/gpu: 8
41	```
42
43	## Attach shared storage (optional)
44
45	Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a `ReadWriteMany` PVC to it so every worker shares datasets and checkpoints, then mount it in the job (below).
46
47	```yaml
48	apiVersion: v1
49	kind: PersistentVolumeClaim
50	metadata:
51	name: shared-pvc
52	spec:
53	accessModes: ["ReadWriteMany"]
54	storageClassName: shared-wekafs # Together default shared storage class
55	volumeName: <shared-volume-name> # the static PV named after your shared volume
56	resources:
57	requests:
58	storage: 100Gi
59	```
60
61	## Run a gang-scheduled job
62
63	The gang guarantee comes from `minAvailable` on a Volcano `Job` (`vcjob`). Set `schedulerName: volcano` and a `queue`. The `volumes`/`volumeMounts` blocks are optional; drop them if the job needs no shared storage.
64
65	```yaml
66	apiVersion: batch.volcano.sh/v1alpha1
67	kind: Job
68	metadata:
69	name: gpu-gang
70	spec:
71	minAvailable: 4 # schedule all 4 pods together or none
72	schedulerName: volcano
73	queue: research
74	policies:
75	- event: PodEvicted
76	action: RestartJob # restart the whole gang if a pod is evicted
77	tasks:
78	- replicas: 4
79	name: worker
80	template:
81	spec:
82	restartPolicy: OnFailure
83	containers:
84	- name: worker
85	image: nvidia/cuda:12.4.0-base-ubuntu22.04
86	command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
87	resources:
88	limits:
89	nvidia.com/gpu: 2 # 4 x 2 = 8 GPUs
90	volumeMounts:
91	- name: shared
92	mountPath: /mnt/shared
93	volumes:
94	- name: shared
95	persistentVolumeClaim:
96	claimName: shared-pvc
97	```
98
99	## Verify
100
101	- `kubectl get vcjob <name>` should show `STATUS Running` and `RUNNINGS` equal to `minAvailable`.
102	- `kubectl get podgroup` shows the gang; a `Running` phase means the whole group is placed.
103	- A gang that cannot fit stays `Inqueue` with zero pods running (all-or-nothing). This is the signal Volcano is working, not a failure. Never "fix" it by lowering `minAvailable` below the job's real parallelism.
104
105	## Rules and gotchas
106
107	- Always pin the Volcano version in the install URL.
108	- `schedulerName: volcano` must be on the pod template (on a `vcjob` it goes under `spec`). Without it, the default scheduler grabs the pods and there is no gang guarantee.
109	- A job whose total request exceeds the queue's `capability` is rejected. Raise the cap or split the job.
110	- To gang-schedule non-vcjob workloads (for example MPIJobs from the preinstalled MPI Operator), set `schedulerName: volcano` and attach a `PodGroup`. See https://volcano.sh/en/docs/podgroup/.
111	- Volcano an

People who liked this also viewed

6 picks

Type

together-volcano

32stars4forks

View on GitHub

How to install together-volcano?

togethercomputer/skills/together-volcano

$npx -y skills add togethercomputer/skills --skill together-volcano

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Volcano on Together GPU clusters
2
3	Volcano is a Kubernetes-native batch scheduler. Its key property is gang scheduling: a group of pods is placed all-at-once or not at all, so distributed training never starts with only some workers running. Use it when a job needs N pods to run together or not at all.
4
5	Public cookbook: https://docs.together.ai/docs/volcano-on-gpu-clusters. Pair this skill with the `together-gpu-clusters` skill, which covers creating the cluster and configuring `kubectl`.
6
7	## Preconditions
8
9	- A Together Kubernetes GPU cluster in the `Ready` state, with `kubectl` pointed at it (`tg beta clusters get-credentials <cluster_id> --set-default-context`).
10	- GPU nodes expose `nvidia.com/gpu`; the NVIDIA device plugin is preinstalled on Together clusters. Confirm with:
11
12	```bash
13	kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
14	```
15
16	## Install
17
18	Pin the version. Do not use `master`.
19
20	```bash
21	kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
22	kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s
23	```
24
25	A `default` queue is created automatically (`kubectl get queue`). Helm is an alternative: `helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace`.
26
27	## Create a queue
28
29	A queue caps and weights the resources a set of jobs can use.
30
31	```yaml
32	apiVersion: scheduling.volcano.sh/v1beta1
33	kind: Queue
34	metadata:
35	name: research
36	spec:
37	reclaimable: true # give idle capacity back to other queues
38	weight: 1 # relative share under contention
39	capability: # hard ceiling for this queue
40	nvidia.com/gpu: 8
41	```
42
43	## Attach shared storage (optional)
44
45	Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a `ReadWriteMany` PVC to it so every worker shares datasets and checkpoints, then mount it in the job (below).
46
47	```yaml
48	apiVersion: v1
49	kind: PersistentVolumeClaim
50	metadata:
51	name: shared-pvc
52	spec:
53	accessModes: ["ReadWriteMany"]
54	storageClassName: shared-wekafs # Together default shared storage class
55	volumeName: <shared-volume-name> # the static PV named after your shared volume
56	resources:
57	requests:
58	storage: 100Gi
59	```
60
61	## Run a gang-scheduled job
62
63	The gang guarantee comes from `minAvailable` on a Volcano `Job` (`vcjob`). Set `schedulerName: volcano` and a `queue`. The `volumes`/`volumeMounts` blocks are optional; drop them if the job needs no shared storage.
64
65	```yaml
66	apiVersion: batch.volcano.sh/v1alpha1
67	kind: Job
68	metadata:
69	name: gpu-gang
70	spec:
71	minAvailable: 4 # schedule all 4 pods together or none
72	schedulerName: volcano
73	queue: research
74	policies:
75	- event: PodEvicted
76	action: RestartJob # restart the whole gang if a pod is evicted
77	tasks:
78	- replicas: 4
79	name: worker
80	template:
81	spec:
82	restartPolicy: OnFailure
83	containers:
84	- name: worker
85	image: nvidia/cuda:12.4.0-base-ubuntu22.04
86	command: ["bash", "-c", "nvidia-smi -L; sleep 300"]
87	resources:
88	limits:
89	nvidia.com/gpu: 2 # 4 x 2 = 8 GPUs
90	volumeMounts:
91	- name: shared
92	mountPath: /mnt/shared
93	volumes:
94	- name: shared
95	persistentVolumeClaim:
96	claimName: shared-pvc
97	```
98
99	## Verify
100
101	- `kubectl get vcjob <name>` should show `STATUS Running` and `RUNNINGS` equal to `minAvailable`.
102	- `kubectl get podgroup` shows the gang; a `Running` phase means the whole group is placed.
103	- A gang that cannot fit stays `Inqueue` with zero pods running (all-or-nothing). This is the signal Volcano is working, not a failure. Never "fix" it by lowering `minAvailable` below the job's real parallelism.
104
105	## Rules and gotchas
106
107	- Always pin the Volcano version in the install URL.
108	- `schedulerName: volcano` must be on the pod template (on a `vcjob` it goes under `spec`). Without it, the default scheduler grabs the pods and there is no gang guarantee.
109	- A job whose total request exceeds the queue's `capability` is rejected. Raise the cap or split the job.
110	- To gang-schedule non-vcjob workloads (for example MPIJobs from the preinstalled MPI Operator), set `schedulerName: volcano` and attach a `PodGroup`. See https://volcano.sh/en/docs/podgroup/.
111	- Volcano an

People who liked this also viewed

6 picks

Type