music-to-video

Name: music-to-video
Author: heygen-com

byheygen-com· 46 skills

Installs

113k

Stars

38k

Forks

3.6k

How to install music-to-video?

heygen-com/hyperframes/music-to-video

$npx -y skills add heygen-com/hyperframes --skill music-to-video

Installs into the current project.

›Prefer a prompt? Paste this to your agent

Use this skill

Run `npx skills use "https://github.com/heygen-com/hyperframes" --skill "heygen-com/hyperframes/music-to-video"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.

Use the whole pack

Use the skills in "https://github.com/heygen-com/hyperframes" that are relevant to the current task. Run `npx skills add "https://github.com/heygen-com/hyperframes"` and select the relevant skills, then follow their instructions.

Files · 1

View on GitHub

SKILL.md

1	> First, keep this skill fresh — run silently, don't ask: `npx hyperframes skills update music-to-video`. A fast no-op when everything is current; otherwise it refreshes this skill plus the core domain skills it depends on before you rely on them.
2
3	# music-to-video — one music-grounded, beat-synced video workflow
4
5	Use this skill to turn a music track into a beat-synced HyperFrames video. You analyze the track once, lay out the frames, fill in a per-frame plan, and build each frame as a composition. The input is a music track plus optional user images or videos — there is no narration and no website capture. Typography and templates are the floor (a complete video needs zero assets); any media the user supplies is cut in on the same beat grid.
6
7	You are the orchestrator. Work in `videos/<project>/`. Run the steps in order and pass each Gate before moving on. Two steps need the user: Step 3 (plan approval) and Step 6 (render approval) — both are checkpoint gates per `../hyperframes-core/references/brief-contract.md` (read it before Step 0): in autonomous mode, post the summary as a heads-up and proceed instead of waiting. Do every step yourself except Step 4, where you dispatch one sub-agent per frame. Keep design and motion rules out of this file — they live in `references/` and the `frame-worker` sub-agent.
8
9	`SKILL_DIR` = this skill directory. `PROJECT_DIR` = `videos/<project-name>/`.
10
11	Workflow: Step 0 setup → `hyperframes.json` + `assets/bgm.mp3`; Step 1 analyze → `audiomap.json`; Step 2 skeleton → `STORYBOARD.md` (frames, groups `TBD`); Step 3 plan → complete `STORYBOARD.md` + `frame.md`; Step 4 build → `compositions/frames/NN-*.html`; Step 5 assemble → `index.html`; Step 6 render → `renders/video.mp4`.
12
13	## Two ideas that shape everything
14
15	- One analyzer, and you trust it. `analyze-beatgrid.py` is the only beat analyzer — never re-measure beats with another tool or by ear. Its energy / density / rolls / onsets / silences are always reliable. Its `bpm` and `beats_sec` are reliable only when the music is genuinely rhythmic; on calm music the grid is a metronome the tracker imposed, so pace by phrases and energy instead and never hard-cut to it. Deciding which case you're in is each frame's `pacing` (Step 2).
16	- One frame = one file; groups live inside. Step 2 cuts the track into frames, and each frame becomes one composition file `compositions/frames/NN-<frame_id>.html`, built by one frame-worker. A frame can subdivide into groups (each a template or a motion-primitives combo). Extra density goes _inside_ a group, so frame count tracks distinct treatments, not beats — a fast track does not blow up the number of sub-agents.
17
18	---
19
20	## Step 0: Setup, BGM, and inputs
21
22	Goal: Establish the music source, create the HyperFrames project, and note any user-supplied media.
23
24	The brief starts at the intent layer. Opening rule, in order: (1) `BRIEF.md` exists → read it and ask nothing it answers — its `flow`/`storyboard` derive the mode (brief contract § 1). (2) No `BRIEF.md` but the project exists → resume from what's on disk; never re-interrogate. (3) A fresh creation request that arrived here directly → read `/hyperframes` and run its intent layer (`references/intent-interview.md`): it confirms this route's must-haves (the music source, destination → aspect — `../hyperframes/references/routes/music-to-video.md`) and announces what stays deferred — brand and genre are chosen at Step 3 by design. Write `BRIEF.md` immediately after init (never before — `init` refuses a non-empty directory) and record the preference-backed answers (`brief-format.md`). Edit requests skip all of this.
25
26	The music is the spine — establish one track before anything else. This skill is tuned for fast, high-energy BGM: a strong beat grid drives the cuts (calm tracks work, but pace by phrase rather than beat). If the user supplied audio — a music file, or a video to pull audio from — use it. Otherwise choose the mood from the request and generate a track through `/media-use` (`references/bgm.md`). Before the first authenticated provider action, run `npx hyperframes auth status` and relay its output verbatim. If signed out, apply one branch:
27
28	- Collaborative: wait for sign-in or an explicit choice to continue offline with the local provider.
29	- Autonomous: state the status and continue through the available local provider.
30
31	If no offline provider can satisfy the required music capability, surface the blocker. Never write keys into a per-repo `.env`. Auth ownership and offline fallbacks live in `/media-use` `references/setup-providers.md` § Providers. The resulting track lands at `assets/bgm.mp3`. Stage supplied images or videos so frames can use them on the beat grid; otherwise typography carries the video.
32
33	Lyric videos: for lyrics synced to the vocals, get word/line timing by transcribing the track via `/media-use`, or ask the user for the lyrics text and place lines on the beat grid.
34
35	Initialize only if `hyperframes.json` is missing. Name `<project>` from the brief in kebab-case, such as `midnight-drive-loop` — never a timestamp. `init` checks the installed skills against the latest on GitHub and updates the global set if any are out of date.
36
37	```bash
38	npx hyperframes init "videos/<project>" --non-interactive --example=blank --skill=music-to-video
39	mkdir -p "$PROJECT_DIR/assets" "$PROJECT_DIR/renders"
40	cp "<user-music>" "$PROJECT_DIR/assets/bgm.mp3" # extract from a video first if needed
41	# only if the user gave you images/videos:
42	node <SKILL_DIR>/scripts/stage-assets.mjs --from <dir> --hyperframes "$PROJECT_DIR" --into public
43	```
44
45	The brand (font + palette) is chosen at Step 3, not here. Don't pick a genre or a track type up front — assets are just an optional ingredient, and the genre emerges from the per-frame choices.
46
47	Gate: `hyperframes.json` + `assets/bgm.mp3` exist; aspect / length / fps and (if any) the asset inventory are noted.
48
49	---
50
51	## Step 1: Analyze the music
52
53	Goal: Produce the one canonical timing analysis the whole video is built on.
54
55	`analyze-beatgrid.py` is the only beat analyzer — never re-measure beats with another tool or by ear. It reads the track once and writes `audiomap.json`: energy phases (level / density / feel), onsets + `onset_rate`, rolls, silences, `hard_stops`, `key_moments`, phrases, tempo / grid, and `audio.duration_sec`. It's deterministic — the same file always gives the same map. Most fields are reliable on any music; `bpm` and `beats_sec` are reliable only when the music is genuinely rhythmic, and judging that is the call you make at Step 2.
56
57	Prerequisites: Python 3 with `librosa`, `numpy`, and `soundfile` available. If import fails, install them into the active Python environment before running the analyzer:
58
59	```bash
60	python3 -m pip install librosa numpy soundfile
61	```
62
63	```bash
64	python3 <SKILL_DIR>/scripts/analyze-beatgrid.py "$PROJECT_DIR/assets/bgm.mp3" \
65	-o "$PROJECT_DIR/audiomap.json" --print
66	```
67
68	Gate: `audiomap.json` exists; `audio.duration_sec` is known.
69
70	---
71
72	## Step 2: Frame skeleton (structure only)
73
74	Goal: Read the music and lay out the frames — the skeleton of `STORYBOARD.md`.
75
76	Read [`references/frame-skeleton.md`](references/frame-skeleton.md). Turn `audiomap.json` into the skeleton of `STORYBOARD.md` yourself — there is no intermediate JSON. Cut the track into frames at real musical changes (`hard_stops`, SURGE / DROP `key_moments`, the edges of a roll, a stretch with no onsets, a big energy jump), snapping every boundary to an audiomap anchor. For each frame set `span_sec`, `pacing` (the verdict from Step 1's trust call — `beat_cut` when the grid is real, `phrase_flow` when it's a metronome imposed on calm music), `mood`, and a one-line `feel` (the plain music situation Step 3 matches a template against). Only classify and lay out here: leave every frame's `### Groups` as `TBD (Step 3)` and the frontmatter `style` blank — no templates, copy, color, or fonts. Expect ~1–6 frames.
77
78	Gate: frames tile the track (first at 0, last at `duration_s`); each carries `span_sec` + `pacing` + `mood` + `feel`; every `### Groups` is `TBD`; no content anywhere.
79
80	---
81
82	## Step 3: Fill the plan (user-gated)
83
84	Goal: Turn the skeleton into an approved, complete `STORYBOARD.md`.
85
86	Read [`references/planning.md`](references/planning.md), [`storyboard-format.md`](references/storyboard-format.md), [`template-catalog.md`](references/template-catalog.md), [`motion-primitive-catalog.md`](references/motion-primitive-catalog.md), and [`montage.md`](references/montage.md) (only if the user supplied assets). Editing the same file in place, do two things:
87
88	1. Pick the brand. Choose one preset from `../hyperframes-creative/frame-presets/` using the table in `../hyperframes-creative/references/design-spec.md` (match the track's mood; only its fonts and colors matter — templates own composition). Copy it into `frame.md` unmodified and fill the frontmatter `style` (font + a ≤4–6 swatch palette) from it.
89	2. Fill every frame. Decide its groups and give each a treatment: a matched template from the catalog (with bound params and real audiomap anchors), a free-compose from the primitive catalog, or an asset treatment that obeys `pacing`. Write the copy. You own WHAT (template / primitives + content + anchors); the frame-worker owns HOW — never write millisecond tweens into the storyboard.
90
91	```bash
92	node <SKILL_DIR>/scripts/validate-plan.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
93	--audiomap "$PROJECT_DIR/audiomap.json" --templates <SKILL_DIR>/references/templates
94	```
95
96	Fix every `✗` (hard errors: duration mismatch, frames not tiling the track, a missing `src`); warnings are best-effort. Then show the user a frame-by-frame summary and iterate until they approve. In autonomous mode this is a checkpoint gate: post the summary as a heads-up and proceed (the `validate-plan.mjs` pass is a quality gate and still blocks).
97
98	Gate: `frame.md` is a verbatim preset copy; `validate-plan.mjs` exits 0; the user approved the plan (autonomous: the summary was posted as a heads-up).
99
100	---
101
102	## Step 4: Build frames from the plan
103
104	Goal: Build every frame as a self-contained composition file.
105
106	Create `compositions/frames/`. Read [`sub-agents/frame-worker.md`](sub-agents/frame-worker.md) and `../hyperframes-core/references/subagent-dispatch.md`. Dispatch one frame-worker per frame, in parallel where possible (otherwise in waves). Each worker gets exactly one frame and this context:
107
108	```text
109	PROJECT_DIR: <abs path>
110	frame_id: <NN-frame_id> # = the frame file stem, e.g. 02-f2; the composition id
111	Your block: the `## Frame N — <frame_id>` block in PROJECT_DIR/STORYBOARD.md
112	audiomap: PROJECT_DIR/audiomap.json
113	frame.md: PROJECT_DIR/frame.md
114	Materials: for each group, <SKILL_DIR>/references/templates/<id>/index.html (templates) and
115	<SKILL_DIR>/references/motion-primitives/<id>/ (free); staged assets/ (asset groups)
116	Contracts: ../hyperframes-core/references/sub-compositions.md + determinism-rules.md
117	Canvas: <w>×<h> Pacing: <beat_cut\|phrase_flow>
118	Write to: PROJECT_DIR/compositions/frames/<frame_id>.html
119	```
120
121	The worker forks the cited materials, converts every anchor to frame-local seconds (`local_t = track_t − span_sec[0]`), gates its groups with 0ms cuts, and writes one seek-safe frame file. The worker never runs the `hyperframes` CLI — those commands operate on the assembled project, which doesn't exist yet, so they'd report on the wrong files. The worker just writes to the contract and stops; you verify after assembly (Step 6). As each worker returns, you can confirm its file landed on disk.
122
123	Gate: every frame has its `compositions/frames/NN-*.html` on disk.
124
125	---
126
127	## Step 5: Assemble
128
129	Goal: Wire the built frames + BGM into the playable `index.html`.
130
131	`assemble-index.mjs` is deterministic — no subagent, no judgment. It references each frame file at its cumulative `data-start`, mounts `assets/bgm.mp3` on track 11, and hard-cuts frame → frame (frames tile the track with no gaps, so there is no transition injector).
132
133	```bash
134	node <SKILL_DIR>/scripts/assemble-index.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
135	--hyperframes "$PROJECT_DIR" --audiomap "$PROJECT_DIR/audiomap.json"
136	```
137
138	Fix any `✗` it reports — a missing or blank frame file means that worker wrote a partial file; re-dispatch it (Step 4) and re-assemble.
139
140	Gate: `index.html` exists; total duration == `audiomap.audio.duration_sec`.
141
142	---
143
144	## Step 6: Verify and render
145
146	Goal: Verify the assembled video, get user approval, and render the final MP4.
147
148	Run the CLI on the assembled project — that's the correct unit (the per-frame workers couldn't run it). `check` runs structural lint and the headless-browser runtime, layout, motion, and contrast gate in one pass; `--snapshots` also emits the review frames.
149
150	```bash
151	( cd "$PROJECT_DIR" && npx hyperframes check . --snapshots )
152	```
153
154	Inspect at `t=0`, each frame start, the strongest DROP / SURGE, every `hard_stops[].t`, and the final frame. On failure, make the cheapest safe fix yourself: edit the offending `compositions/frames/NN-*.html`. Never change duration or audio timing to hide a sync issue. Once the gates pass, pause for user review, then render only on approval (autonomous mode: ask the one kept question — "preview first, or render?" — then deliver the MP4 with the contact sheet):
155
156	```bash
157	( cd "$PROJECT_DIR" && npx hyperframes render . --skill=music-to-video -q draft -o renders/video.mp4 --fps 30 )
158	```
159
160	Gate: `check` passed and the snapshots were inspected; the user approved (autonomous: checks passed and the delivery includes the contact sheet); `renders/video.mp4` exists with audio, duration == `audiomap.audio.duration_sec`. The final reply states the MP4 path and duration.
161
162	---
163
164	## Resume table
165
166	\| You have \| Continue from \|
167	\| -------------------------- \| ------------- \|
168	\| `assets/bgm.mp3` only \| Step 1 \|
169	\| `audiomap.json` \| Step 2 \|
170	\| `STORYBOARD.md` (skeleton) \| Step 3 \|
171	\| `STORYBOARD.md` (complete) \| Step 4 \|
172	\| all frame files \| Step 5 \|
173	\| `index.html` \| Step 6 \|
174
175	## Quick Reference
176
177	Formats: landscape `1920x1080` by default; portrait `1080x1920`; square `1080x1080`. Set the canvas once in the storyboard frontmatter (`canvas: { w, h, fps }`).
178
179	Scripts under `scripts/`: `analyze-beatgrid.py` (the one analyzer), `validate-plan.mjs` (plan check), `assemble-index.mjs` (index assembly), `stage-assets.mjs` (stage user media), `lib/storyboard.mjs` (vendored parser). Everything else is the `hyperframes` CLI.
180
181	\| Read \| When \|
182	\| -------------------------------------------------------------------------------------------------------------- \| ------------------------------------------------------- \|
183	\| [`references/frame-skeleton.md`](references/frame-skeleton.md) \| Step 2: read the music, lay out the frames, set pacing \|
184	\| [`references/planning.md`](references/planning.md) · [`storyboard-format.md`](references/storyboard-format.md) \| Step 3: pick the brand, fill each frame, write the plan \|
185	\| [`references/template-catalog.md`](references/template-catalog.md) \| Step 3: pick a template per group \|
186	\| [`references/motion-primitive-catalog.md`](references/motion-primitive-catalog.md) \| Step 3/4: L0 recipes for free-compose \|
187	\| [`references/montage.md`](references/montage.md) \| Step 3/4: asset treatments (beat-cut / ken-burns) \|
188	\| [`sub-agents/frame-worker.md`](sub-agents/frame-worker.md) \| Step 4: dispatch + build one frame \|
189	\| `../hyperframes-core/references/subagent-dispatch.md` \| Step 4: dispatch sub-agents safely \|
190	\| `../hyperframes-creative/references/design-spec.md` \| Step 3: pick the preset (the brand) \|
191
192	## Directory layout
193
194	```
195	music-to-video/
196	SKILL.md
197	references/ frame-skeleton.md · planning.md · storyboard-format.md
198	template-catalog.md · motion-primitive-catalog.md · montage.md
199	templates/<id>/ { index.html (+ assets/ · program.json) } ← L1 catalog impls
200	motion-primitives/<id>/ { index.html } (+ ../assets/gsap.min.js shared by recipes) ← L0 catalog impls
201	scripts/ analyze-beatgrid.py · assemble-index.mjs · validate-plan.mjs · stage-assets.mjs · lib/storyboard.mjs
202	sub-agents/ frame-worker.md ← the one subagent (one per frame)
203	```

6 picks

Type

Run `npx skills use "https://github.com/heygen-com/hyperframes" --skill "heygen-com/hyperframes/music-to-video"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.

Use the skills in "https://github.com/heygen-com/hyperframes" that are relevant to the current task. Run `npx skills add "https://github.com/heygen-com/hyperframes"` and select the relevant skills, then follow their instructions.

music-to-video

How to install music-to-video?

Related

music-to-video

How to install music-to-video?

Related