spark-memory-thermal-ops

Name: spark-memory-thermal-ops
Author: wshobson

38kstars4.1kforks

How to install spark-memory-thermal-ops?

wshobson/agents/spark-memory-thermal-ops

$npx -y skills add wshobson/agents --skill spark-memory-thermal-ops

Installs into the current project.

Summary

Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.

Files · 1

View on GitHub

SKILL.md

1	# Spark Memory & Thermal Ops
2
3	DGX Spark's GB10 chip has one 128GB unified
4	memory (UMA) pool shared by CPU and GPU, and a
5	sustained power ceiling well below its rated
6	figure. Both break discrete-GPU assumptions:
7	headroom isn't what `nvidia-smi` reports, and a
8	run that starts fast will slow down mid-job
9	with nothing misconfigured. This skill covers
10	planning memory headroom, working an actual
11	OOM, and watching thermals across a long job.
12	For launch-time failure modes (ABI mismatches,
13	flash-attn, playbook breakage), see
14	`spark-training-gotchas` — this skill assumes
15	the job starts.
16
17	## Common Issues Quick Reference
18
19	\| Situation \| Do this \|
20	\|---\|---\|
21	\| Planning headroom before launch \| Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model \|
22	\| Job OOMs on unified memory \| Work the OOM Ladder in order: flush, then batch/pack, then method downgrade \|
23	\| Throughput drops mid-run \| Check the power/temp log before assuming a config bug — see Thermal Monitoring \|
24	\| Trainer + inference server both wanted \| Run one at a time — see Concurrent Workloads \|
25
26	## When to Use This Skill
27
28	- Sizing a training run against the 128GB pool
29	before launch — will this model, method, and
30	batch/pack combination fit.
31	- A run OOMs mid-load or mid-step and the
32	remediation order matters — what to try first,
33	second, third.
34	- Watching temperature and power during a
35	multi-hour job, deciding whether a slowdown is
36	thermal throttling or something else.
37	- Planning to run a trainer alongside an
38	inference server (vLLM, Ollama) on the same box.
39
40	## UMA Memory Model
41
42	Spark has no separate GPU VRAM — the GPU and
43	CPU share one 128GB pool. Two consequences:
44
45	- **`nvidia-smi` and `cudaMemGetInfo`
46	underreport pressure — or report nothing at
47	all.** Both report CUDA-allocator-visible
48	memory, not the pool's actual state — a box can
49	show headroom in `nvidia-smi` and still OOM,
50	because page-cache and mmap'd pages the
51	allocator doesn't see consume the same pool. On
52	some driver/setups, the memory query returns
53	`[N/A], [N/A]` outright instead of a number — a
54	script grepping for a numeric value there gets
55	nothing, not a misleading undercount (see
56	`spark-training-gotchas` gotcha G3).
57
58	- **Model load is a transient peak, not the
59	steady state.** Loading safetensors weights
60	mmaps the file, then copies into CUDA
61	tensors — for a window during load, both the
62	mmap'd pages and the CUDA copy count against
63	the pool at once. A model that fits while
64	training can still OOM during load if headroom
65	was sized for the post-load footprint instead
66	of this doubled transient.
67
68	Plan and diagnose with `free -g`, not
69	`nvidia-smi`:
70
71	```bash
72	free -g \| awk 'NR==2 {print "free:", $4, "GB"}'
73	```
74
75	Rule of thumb: take that free figure, subtract a
76	few GB for OS/driver overhead, and budget against
77	the result — not the 128GB spec number.
78	The worksheet in `references/uma-accounting.md`
79	accepts parameter count, dtype, and method as
80	input, and returns a memory estimate to compare
81	against known anchors.
82
83	### Planning Sequence
84
85	Before launch, work through these in order:
86
87	1. Read `free -g`; subtract OS/driver overhead
88	for the budget.
89	2. Estimate weights + optimizer + gradients +
90	activations from `references/uma-accounting.md`.
91	3. Compare against the closest anchor (70B
92	QLoRA, 27B LoRA, 9B full FT), not the
93	estimate alone.
94	4. If the estimate is close to the budget, start
95	with shorter packing or a smaller batch —
96	cheaper than hitting the OOM Ladder mid-run.
97
98	### Example: Sizing a 70B QLoRA Run
99
100	A sanity check of the worksheet formula against
101	the ≈40GB anchor:
102
103	```python
104	params = 70e9
105	weights_gb = params * 0.5 / 1e9 # NF4, step 1
106	adapter_gb = 0.5 # step 5, negligible
107	total_gb = weights_gb + adapter_gb # + activations
108	print(f"{total_gb:.0f}GB before activations")
109	```
110
111	Weights alone land near the ≈40GB anchor — a plan
112	estimating far above that for the same model
113	class is a signal to recheck dtype and method.
114
115	## The OOM Ladder
116
117	When a job OOMs on unified memory, work this
118	ladder in order. Each step is more disruptive
119	than the last — don't skip ahead:
120	reducing batch size is never step 1.
121
122	1. Flush the buffer cache. Page cache from a
123	previous run or a large dataset read often
124	accounts for GB of the "missing" headroom.
125	This costs nothing but a rerun and doesn't
126	touch the job's configuration:
127
128	```bash
129	sync; echo 3 > /proc/sys/vm/drop_caches
130	```
131
132	Needs root; a between-run reset, not a
133	mid-training step. See
134	`spark-training-gotchas` (gotcha G3) for the
135	full diagnostic behind this step.
136
137	2. Reduce batch size or packing length. Only
138	after a flush fails to free enough he

People who liked this also viewed

6 picks

Type

spark-memory-thermal-ops

38kstars4.1kforks

View on GitHub

How to install spark-memory-thermal-ops?

wshobson/agents/spark-memory-thermal-ops

$npx -y skills add wshobson/agents --skill spark-memory-thermal-ops

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Spark Memory & Thermal Ops
2
3	DGX Spark's GB10 chip has one 128GB unified
4	memory (UMA) pool shared by CPU and GPU, and a
5	sustained power ceiling well below its rated
6	figure. Both break discrete-GPU assumptions:
7	headroom isn't what `nvidia-smi` reports, and a
8	run that starts fast will slow down mid-job
9	with nothing misconfigured. This skill covers
10	planning memory headroom, working an actual
11	OOM, and watching thermals across a long job.
12	For launch-time failure modes (ABI mismatches,
13	flash-attn, playbook breakage), see
14	`spark-training-gotchas` — this skill assumes
15	the job starts.
16
17	## Common Issues Quick Reference
18
19	\| Situation \| Do this \|
20	\|---\|---\|
21	\| Planning headroom before launch \| Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model \|
22	\| Job OOMs on unified memory \| Work the OOM Ladder in order: flush, then batch/pack, then method downgrade \|
23	\| Throughput drops mid-run \| Check the power/temp log before assuming a config bug — see Thermal Monitoring \|
24	\| Trainer + inference server both wanted \| Run one at a time — see Concurrent Workloads \|
25
26	## When to Use This Skill
27
28	- Sizing a training run against the 128GB pool
29	before launch — will this model, method, and
30	batch/pack combination fit.
31	- A run OOMs mid-load or mid-step and the
32	remediation order matters — what to try first,
33	second, third.
34	- Watching temperature and power during a
35	multi-hour job, deciding whether a slowdown is
36	thermal throttling or something else.
37	- Planning to run a trainer alongside an
38	inference server (vLLM, Ollama) on the same box.
39
40	## UMA Memory Model
41
42	Spark has no separate GPU VRAM — the GPU and
43	CPU share one 128GB pool. Two consequences:
44
45	- **`nvidia-smi` and `cudaMemGetInfo`
46	underreport pressure — or report nothing at
47	all.** Both report CUDA-allocator-visible
48	memory, not the pool's actual state — a box can
49	show headroom in `nvidia-smi` and still OOM,
50	because page-cache and mmap'd pages the
51	allocator doesn't see consume the same pool. On
52	some driver/setups, the memory query returns
53	`[N/A], [N/A]` outright instead of a number — a
54	script grepping for a numeric value there gets
55	nothing, not a misleading undercount (see
56	`spark-training-gotchas` gotcha G3).
57
58	- **Model load is a transient peak, not the
59	steady state.** Loading safetensors weights
60	mmaps the file, then copies into CUDA
61	tensors — for a window during load, both the
62	mmap'd pages and the CUDA copy count against
63	the pool at once. A model that fits while
64	training can still OOM during load if headroom
65	was sized for the post-load footprint instead
66	of this doubled transient.
67
68	Plan and diagnose with `free -g`, not
69	`nvidia-smi`:
70
71	```bash
72	free -g \| awk 'NR==2 {print "free:", $4, "GB"}'
73	```
74
75	Rule of thumb: take that free figure, subtract a
76	few GB for OS/driver overhead, and budget against
77	the result — not the 128GB spec number.
78	The worksheet in `references/uma-accounting.md`
79	accepts parameter count, dtype, and method as
80	input, and returns a memory estimate to compare
81	against known anchors.
82
83	### Planning Sequence
84
85	Before launch, work through these in order:
86
87	1. Read `free -g`; subtract OS/driver overhead
88	for the budget.
89	2. Estimate weights + optimizer + gradients +
90	activations from `references/uma-accounting.md`.
91	3. Compare against the closest anchor (70B
92	QLoRA, 27B LoRA, 9B full FT), not the
93	estimate alone.
94	4. If the estimate is close to the budget, start
95	with shorter packing or a smaller batch —
96	cheaper than hitting the OOM Ladder mid-run.
97
98	### Example: Sizing a 70B QLoRA Run
99
100	A sanity check of the worksheet formula against
101	the ≈40GB anchor:
102
103	```python
104	params = 70e9
105	weights_gb = params * 0.5 / 1e9 # NF4, step 1
106	adapter_gb = 0.5 # step 5, negligible
107	total_gb = weights_gb + adapter_gb # + activations
108	print(f"{total_gb:.0f}GB before activations")
109	```
110
111	Weights alone land near the ≈40GB anchor — a plan
112	estimating far above that for the same model
113	class is a signal to recheck dtype and method.
114
115	## The OOM Ladder
116
117	When a job OOMs on unified memory, work this
118	ladder in order. Each step is more disruptive
119	than the last — don't skip ahead:
120	reducing batch size is never step 1.
121
122	1. Flush the buffer cache. Page cache from a
123	previous run or a large dataset read often
124	accounts for GB of the "missing" headroom.
125	This costs nothing but a rerun and doesn't
126	touch the job's configuration:
127
128	```bash
129	sync; echo 3 > /proc/sys/vm/drop_caches
130	```
131
132	Needs root; a between-run reset, not a
133	mid-training step. See
134	`spark-training-gotchas` (gotcha G3) for the
135	full diagnostic behind this step.
136
137	2. Reduce batch size or packing length. Only
138	after a flush fails to free enough he

People who liked this also viewed

6 picks

Type