Extracting lyrics from the song

Intro

Lyric Video Studio has many ways of importing the lyrics in, but there’s always the drag of syncing them. Lyric Video studio has nice “sync as you play” feature, but you can skip it too with modern tools. Lyric Video Studio has three modes for extracting the lyrics from audio track: Locally runnable Whisper (even with cpu!), OpenAi APi (in practice still Whiper) and Elevenlabs API. We’ll concentrate on extracting lyrics with Whisper, the local version.

The Works

To start, right click on the audio file in your project and select “Extract text”. You’ll get a popup with few options:

Set the folder for Whisper models, implementatation will auto-download those. For model type, usually MediumEn is enough, but for certain vocals, even tiny can bring better results or LargeV3. You basically just have to try it out. Next toggle is the choice between music mode and speech mode. These are only available in local mode and adjust the available options. Let’s stick with music for now.

The tiny details

“Save separated audio” saves the separated vocals and instrumentals and adds those to timeline when operatin is finishen. MOre of those at the end of this tutorial 😉 “Skip stem separation” skips the vocal / instumental separation completely. For some songs, Whisper performs better when it is fed the complete audio, not the vocals only. But for now, leave it “off”. “Skip voice detection” is on by default for music mode: It tries to remove the silent parts from the separated audio. It works good for speech, but not to music. Next, you just click “Process” and wait for Whisper to do it’s thing

Fine tuning the output

Even though the Whisper is quite old, it still is the best one for lyrics extraction, on local machines. It is bit surpising that there is no better one 🙂 It has one big weakness: it tends start the words way too early and you’ll be the one to clear the mess XD Luckily the item trim feature in Lyric Video Studio is so great, that it’s easy to fix with that.

However, if you don’t want to do that manually, there’s one more trick in the sleeve we can do: remember the “Save separated audio”? Yep, that’s the key. What you can do, is to right click the extracted vocals and select “Remove silent parts”. There’s few knobs there as well, the threshold slider being to most important. Click Analyze and then find the sweetspot to remove only the parts where there is no vocals. Make sure you have un-checked the “Close gaps” toggle from the view.

Next, select all the trimmed vocal parts and click “Extract text” again. This time, check the “Skip stem separation” and click process: now you should have perfectly aligned lyrocs from your vocals. Just remember to mute the vocal & instrument parts or you’ll duplicate your sound 🙂 You can find more help about automatic trimming from help page

This extra step is intentionally left to “do it manually”, because it might now always be required and it requires some manual intervention to clip the audio by its silence. I’d be very surprised if no better “ASR for music + timestamps” would come out soon, since Whisper ws not really meant for vocals and it’s very old. But until then, this is the best tool for automatically syncing the lyrics