optimize-musa-training

Name: optimize-musa-training
Author: open-infra-skills

137stars1forks

How to install optimize-musa-training?

open-infra-skills/infra-skills/optimize-musa-training

$npx -y skills add open-infra-skills/infra-skills --skill optimize-musa-training

Installs into the current project.

Summary

Profile, benchmark, debug, and optimize AI training workloads on Moore Threads MUSA GPUs while preserving numerical behavior. Use for Torch MUSA, MTT GPUs, mthreads-gmi, Moore Perf System (msys), Moore Perf Compute (mcu), MFU/HFU, FlashAttention/FA2, SDPA, torch.compile, FSDP/FSD

Files · 1

View on GitHub

SKILL.md

1	# Optimize MUSA Training
2
3	Use a measurement-first workflow to improve MUSA training throughput without changing model semantics. Treat framework timing, system traces, and kernel counters as different layers of evidence.
4
5	## Guardrails
6
7	- Preserve model architecture, data semantics, optimizer math, precision policy, and checkpoint compatibility unless the user explicitly authorizes a change.
8	- Establish a versioned baseline before editing code. Compare forward outputs, loss, gradients, memory, and steady-state throughput after every retained change.
9	- Keep profiler overhead out of the throughput denominator. Measure FLOPs in a profiled run and steady step time in an otherwise equivalent non-profiled run.
10	- Never label profiler-attributed executed FLOPs as model MFU without stating the FLOP definition and coverage. Distinguish useful model FLOPs, executed hardware FLOPs, and profiler-attributed FLOPs.
11	- Do not infer MUSA behavior from CUDA behavior. Feature-detect the installed driver, SDK, Torch MUSA, muDNN, muBLAS, MCCL, attention backend, and profiler versions.
12	- Keep cluster transport separate from profiling logic. Do not require PowerShell, VS Code, a jump host, a specific scheduler, or a particular client operating system.
13	- Keep credentials, internal hostnames, private image registries, dataset paths, and proprietary reports out of public artifacts.
14
15	## Route The Task
16
17	1. For environment, import, device-selection, or container failures, read [environment-and-preflight.md](references/environment-and-preflight.md).
18	2. For MFU/HFU, PyTorch profiling, timeline analysis, transfers, or kernel counters, read [measurement-and-profiling.md](references/measurement-and-profiling.md).
19	3. For FA2, GEMM, compile, launch, memory, dataloader, FSDP, or MCCL optimization, read [optimization-playbook.md](references/optimization-playbook.md).
20	4. Before retaining any optimization, read [correctness-and-experiments.md](references/correctness-and-experiments.md).
21	5. For a real low-batch S5000 case and negative results worth avoiding, read [s5000-case-study.md](references/s5000-case-study.md).
22
23	## Workflow
24
25	### 1. Define The Metric And Invariants
26
27	Record:
28
29	- workload, model revision, dataset/sample bucket, precision, sequence shape, batch per device, accumulation, device count, and distributed strategy;
30	- useful-model FLOPs, executed FLOPs, or profiler-attributed FLOPs;
31	- peak denominator by SKU and precision;
32	- model behaviors that must remain unchanged;
33	- target metric such as samples/s, tokens/s, MFU, HFU, memory, or time-to-train.
34
35	Do not optimize a mixed workload as though every sample has the longest shape. Benchmark each meaningful bucket and the actual weighted mixture.
36
37	### 2. Capture The Environment
38
39	Run:
40
41	```bash
42	python scripts/musa_env_report.py --output <run-dir>/environment.json
43	```
44
45	Also save the container image digest, source commit, working-tree diff, launch command, and relevant environment switches. Verify physical device visibility from inside the process rather than trusting shell variables alone.
46
47	### 3. Establish A Steady Baseline
48
49	- Warm up imports, allocator state, compilation, autotuning, dataloader workers, and collectives.
50	- Exclude compile steps, epoch boundaries, checkpoint saves, validation, and profiler steps.
51	- Capture enough steady steps to expose variance. Reverse A/B order and repeat when the expected gain is below 3%.
52	- Keep all ranks on the same shape bucket for a distributed step.
53
54	Summarize logs with:
55
56	```bash
57	python scripts/summarize_steps.py train.log --skip-first 2 --json
58	```
59
60	### 4. Quantify Utilization
61
62	For one shape:
63
64	```bash
65	python scripts/compute_mfu.py \
66	--flops-per-device-step-tflop <F> \
67	--step-seconds <T> \
68	--peak-tflops-per-device <C> \
69	--flops-kind profiler-attributed
70	```
71
72	For mixed buckets, provide a JSON config with per-bucket FLOPs, time, and either step weight or sample count plus global batch. Use the aggregate total-FLOPs / total-time result, not a naive arithmetic mean.
73
74	### 5. Descend Through Three Profiling Layers
75
76	Use the cheapest layer that answers the current question:
77
78	1. Framework layer: separate input pipeline, forward, backward/recompute, optimizer, clipping, and collectives.
79	2. System layer: use Moore Perf System to inspect CPU/GPU overlap, launches, copies, synchronization, queues, streams, and rank skew.
80	3. Kernel layer: use Moore Perf Compute only on a small reproducible segment to inspect LaunchStats, MemoryWorkloadAnalysis, SpeedOfLight, occupancy, registers, memory pipelines

People who liked this also viewed

6 picks

Type

optimize-musa-training

137stars1forks

View on GitHub

How to install optimize-musa-training?

open-infra-skills/infra-skills/optimize-musa-training

$npx -y skills add open-infra-skills/infra-skills --skill optimize-musa-training

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Optimize MUSA Training
2
3	Use a measurement-first workflow to improve MUSA training throughput without changing model semantics. Treat framework timing, system traces, and kernel counters as different layers of evidence.
4
5	## Guardrails
6
7	- Preserve model architecture, data semantics, optimizer math, precision policy, and checkpoint compatibility unless the user explicitly authorizes a change.
8	- Establish a versioned baseline before editing code. Compare forward outputs, loss, gradients, memory, and steady-state throughput after every retained change.
9	- Keep profiler overhead out of the throughput denominator. Measure FLOPs in a profiled run and steady step time in an otherwise equivalent non-profiled run.
10	- Never label profiler-attributed executed FLOPs as model MFU without stating the FLOP definition and coverage. Distinguish useful model FLOPs, executed hardware FLOPs, and profiler-attributed FLOPs.
11	- Do not infer MUSA behavior from CUDA behavior. Feature-detect the installed driver, SDK, Torch MUSA, muDNN, muBLAS, MCCL, attention backend, and profiler versions.
12	- Keep cluster transport separate from profiling logic. Do not require PowerShell, VS Code, a jump host, a specific scheduler, or a particular client operating system.
13	- Keep credentials, internal hostnames, private image registries, dataset paths, and proprietary reports out of public artifacts.
14
15	## Route The Task
16
17	1. For environment, import, device-selection, or container failures, read [environment-and-preflight.md](references/environment-and-preflight.md).
18	2. For MFU/HFU, PyTorch profiling, timeline analysis, transfers, or kernel counters, read [measurement-and-profiling.md](references/measurement-and-profiling.md).
19	3. For FA2, GEMM, compile, launch, memory, dataloader, FSDP, or MCCL optimization, read [optimization-playbook.md](references/optimization-playbook.md).
20	4. Before retaining any optimization, read [correctness-and-experiments.md](references/correctness-and-experiments.md).
21	5. For a real low-batch S5000 case and negative results worth avoiding, read [s5000-case-study.md](references/s5000-case-study.md).
22
23	## Workflow
24
25	### 1. Define The Metric And Invariants
26
27	Record:
28
29	- workload, model revision, dataset/sample bucket, precision, sequence shape, batch per device, accumulation, device count, and distributed strategy;
30	- useful-model FLOPs, executed FLOPs, or profiler-attributed FLOPs;
31	- peak denominator by SKU and precision;
32	- model behaviors that must remain unchanged;
33	- target metric such as samples/s, tokens/s, MFU, HFU, memory, or time-to-train.
34
35	Do not optimize a mixed workload as though every sample has the longest shape. Benchmark each meaningful bucket and the actual weighted mixture.
36
37	### 2. Capture The Environment
38
39	Run:
40
41	```bash
42	python scripts/musa_env_report.py --output <run-dir>/environment.json
43	```
44
45	Also save the container image digest, source commit, working-tree diff, launch command, and relevant environment switches. Verify physical device visibility from inside the process rather than trusting shell variables alone.
46
47	### 3. Establish A Steady Baseline
48
49	- Warm up imports, allocator state, compilation, autotuning, dataloader workers, and collectives.
50	- Exclude compile steps, epoch boundaries, checkpoint saves, validation, and profiler steps.
51	- Capture enough steady steps to expose variance. Reverse A/B order and repeat when the expected gain is below 3%.
52	- Keep all ranks on the same shape bucket for a distributed step.
53
54	Summarize logs with:
55
56	```bash
57	python scripts/summarize_steps.py train.log --skip-first 2 --json
58	```
59
60	### 4. Quantify Utilization
61
62	For one shape:
63
64	```bash
65	python scripts/compute_mfu.py \
66	--flops-per-device-step-tflop <F> \
67	--step-seconds <T> \
68	--peak-tflops-per-device <C> \
69	--flops-kind profiler-attributed
70	```
71
72	For mixed buckets, provide a JSON config with per-bucket FLOPs, time, and either step weight or sample count plus global batch. Use the aggregate total-FLOPs / total-time result, not a naive arithmetic mean.
73
74	### 5. Descend Through Three Profiling Layers
75
76	Use the cheapest layer that answers the current question:
77
78	1. Framework layer: separate input pipeline, forward, backward/recompute, optimizer, clipping, and collectives.
79	2. System layer: use Moore Perf System to inspect CPU/GPU overlap, launches, copies, synchronization, queues, streams, and rank skew.
80	3. Kernel layer: use Moore Perf Compute only on a small reproducible segment to inspect LaunchStats, MemoryWorkloadAnalysis, SpeedOfLight, occupancy, registers, memory pipelines

People who liked this also viewed

6 picks

Type