$npx -y skills add calesthio/OpenMontage --skill elevenlabsGenerate AI voiceovers, sound effects, and music using ElevenLabs APIs. Use when creating audio content for videos, podcasts, or games. Triggers include generating voiceovers, narration, dialogue, sound effects from descriptions, background music, soundtrack generation, voice clo
| 1 | # ElevenLabs Audio Generation |
| 2 | |
| 3 | Requires `ELEVENLABS_API_KEY` in `.env`. |
| 4 | |
| 5 | ## Text-to-Speech |
| 6 | |
| 7 | ```python |
| 8 | from elevenlabs.client import ElevenLabs |
| 9 | from elevenlabs import save, VoiceSettings |
| 10 | import os |
| 11 | |
| 12 | client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY")) |
| 13 | |
| 14 | audio = client.text_to_speech.convert( |
| 15 | text="Welcome to my video!", |
| 16 | voice_id="JBFqnCBsd6RMkjVDRZzb", |
| 17 | model_id="eleven_multilingual_v2", |
| 18 | voice_settings=VoiceSettings( |
| 19 | stability=0.5, |
| 20 | similarity_boost=0.75, |
| 21 | style=0.5, |
| 22 | speed=1.0 |
| 23 | ) |
| 24 | ) |
| 25 | save(audio, "voiceover.mp3") |
| 26 | ``` |
| 27 | |
| 28 | ### Models |
| 29 | |
| 30 | | Model | Quality | SSML Support | Notes | |
| 31 | |-------|---------|--------------|-------| |
| 32 | | `eleven_multilingual_v2` | Highest consistency | None | Stable, production-ready, 29 languages | |
| 33 | | `eleven_flash_v2_5` | Good | `<break>`, `<phoneme>` | Fast, supports pause/pronunciation tags | |
| 34 | | `eleven_turbo_v2_5` | Good | `<break>`, `<phoneme>` | Fastest latency | |
| 35 | | `eleven_v3` | Most expressive | None | Alpha — unreliable, needs prompt engineering | |
| 36 | |
| 37 | **Choose:** multilingual_v2 for reliability, flash/turbo for SSML control, v3 for maximum expressiveness (expect retakes). |
| 38 | |
| 39 | ### Voice Settings by Style |
| 40 | |
| 41 | | Style | stability | similarity | style | speed | |
| 42 | |-------|-----------|------------|-------|-------| |
| 43 | | Natural/professional | 0.75-0.85 | 0.9 | 0.0-0.1 | 1.0 | |
| 44 | | Conversational | 0.5-0.6 | 0.85 | 0.3-0.4 | 0.9-1.0 | |
| 45 | | Energetic/YouTuber | 0.3-0.5 | 0.75 | 0.5-0.7 | 1.0-1.1 | |
| 46 | |
| 47 | ### Pauses Between Sections |
| 48 | |
| 49 | **With flash/turbo models:** Use SSML break tags inline: |
| 50 | ``` |
| 51 | ...end of section. <break time="1.5s" /> Start of next... |
| 52 | ``` |
| 53 | Max 3 seconds per break. Excessive breaks can cause speed artifacts. |
| 54 | |
| 55 | **With multilingual_v2 / v3:** No SSML support. Options: |
| 56 | - Paragraph breaks (blank lines) — creates ~0.3-0.5s natural pause |
| 57 | - Post-process with ffmpeg: split audio and insert silence |
| 58 | |
| 59 | **WARNING:** `...` (ellipsis) is NOT a reliable pause — it can be vocalized as a word/sound. Do not use ellipsis as a pause mechanism. |
| 60 | |
| 61 | ### Pronunciation Control |
| 62 | |
| 63 | **Phonetic spelling (any model):** Write words as you want them pronounced: |
| 64 | - `Janus` → `Jan-us` |
| 65 | - `nginx` → `engine-x` |
| 66 | - Use dashes, capitals, apostrophes to guide pronunciation |
| 67 | |
| 68 | **SSML phoneme tags (flash/turbo only):** |
| 69 | ``` |
| 70 | <phoneme alphabet="ipa" ph="ˈdʒeɪnəs">Janus</phoneme> |
| 71 | ``` |
| 72 | |
| 73 | ### Iterative Workflow |
| 74 | |
| 75 | 1. Generate → listen → identify pronunciation/pacing issues |
| 76 | 2. Adjust: phonetic spellings, break tags, voice settings |
| 77 | 3. Regenerate. If pauses aren't precise enough, add silence in post with ffmpeg rather than fighting the TTS engine. |
| 78 | |
| 79 | ## Voice Cloning |
| 80 | |
| 81 | ### Instant Voice Clone |
| 82 | |
| 83 | ```python |
| 84 | with open("sample.mp3", "rb") as f: |
| 85 | voice = client.voices.ivc.create( |
| 86 | name="My Voice", |
| 87 | files=[f], |
| 88 | remove_background_noise=True |
| 89 | ) |
| 90 | print(f"Voice ID: {voice.voice_id}") |
| 91 | ``` |
| 92 | |
| 93 | - Use `client.voices.ivc.create()` (not `client.voices.clone()`) |
| 94 | - Pass file handles in binary mode (`"rb"`), not paths |
| 95 | - Convert m4a first: `ffmpeg -i input.m4a -codec:a libmp3lame -qscale:a 2 output.mp3` |
| 96 | - Multiple samples (2-3 clips) improve accuracy |
| 97 | - Save voice ID for reuse |
| 98 | |
| 99 | **Professional Voice Clone:** Requires Creator plan+, 30+ min audio. See [reference.md](reference.md). |
| 100 | |
| 101 | ## Sound Effects |
| 102 | |
| 103 | Max 22 seconds per generation. |
| 104 | |
| 105 | ```python |
| 106 | result = client.text_to_sound_effects.convert( |
| 107 | text="Thunder rumbling followed by heavy rain", |
| 108 | duration_seconds=10, |
| 109 | prompt_influence=0.3 |
| 110 | ) |
| 111 | with open("thunder.mp3", "wb") as f: |
| 112 | for chunk in result: |
| 113 | f.write(chunk) |
| 114 | ``` |
| 115 | |
| 116 | **Prompt tips:** Be specific — "Heavy footsteps on wooden floorboards, slow and deliberate, with creaking" |
| 117 | |
| 118 | ## Music Generation |
| 119 | |
| 120 | 10 seconds to 5 minutes. Use `client.music.compose()` (not `.generate()`). |
| 121 | |
| 122 | ```python |
| 123 | result = client.music.compose( |
| 124 | prompt="Upbeat indie rock, catchy guitar riff, energetic drums, travel vlog", |
| 125 | music_length_ms=60000, |
| 126 | force_instrumental=True |
| 127 | ) |
| 128 | with open("music.mp3", "wb") as f: |
| 129 | for chunk in result: |
| 130 | f.write(chunk) |
| 131 | ``` |
| 132 | |
| 133 | **Prompt structure:** Genre, mood, instruments, tempo, use case. Add "no vocals" or use `force_instrumental=True` for background music. |
| 134 | |
| 135 | ## Remotion Integration |
| 136 | |
| 137 | ### Complete Workflow: Script to Synchronized Scene |
| 138 | |
| 139 | ``` |
| 140 | VOICEOVER-SCRIPT.md → voiceover.py → public/audio/ → Remotion composition |
| 141 | ↓ ↓ ↓ ↓ |
| 142 | Scene narration Generate MP3 Audio files <Audio> component |
| 143 | with durations per scene with timing synced to scenes |
| 144 | ``` |
| 145 | |
| 146 | ### Step 1: Generate Per-Scene Audio |
| 147 | |
| 148 | Use the toolkit's voiceover tool to generate audio for each scene: |
| 149 | |
| 150 | ```bash |
| 151 | # Generate voiceover files for each sc |