Auto captions on a talking-head video are usually good enough to fix by hand in a few minutes. On a song the same tool can produce lines that look like real words and have nothing to do with what was sung. That is not bad luck. Speech recognition models are trained mostly on speech, and a song breaks several of their assumptions at the same time.
This guide covers what goes wrong, what the captions in the video editor and the Audio to Text tool do about each problem, and the approach that works best: giving the tool the words and letting it work out only when each one is sung.
Problem 1: the music is in the same place as the voice
A singing voice and the instruments around it share most of the same frequencies. The model is trying to hear words, and when it is unsure it does not say "I don't know". It produces the most likely-looking text for the sound it heard. Over a drum loop or a held synth chord, that can be a full, fluent, entirely invented sentence. Whisper, the open model these tools use, is well known for doing this over music and silence.
The tools here take four steps against it:
- Music-only stretches are skipped. The recording is heard in windows of up to 30 seconds. A window that is near-silent is not sent to the model at all, and a window the model itself says contains no speech is left empty instead of being filled with words.
- The voice is brought forward. When the video editor hears a song, it filters the sound to the band where voices sit, roughly 100 Hz to 6 kHz, before recognition. That removes bass and cymbals, though not the guitars and keyboards in the middle.
- Unsure phrases are dropped. Whisper gives a probability for each word. A phrase whose words average below one half is left out. On a Hindi song tested against its lyrics, this cut the caption lines that did not match the singing from 24 to 9 without losing any correctly heard words. A missing caption is better than a wrong one.
- Impossible words are removed. Over music the model sometimes strings letters together into things no language spells, such as a run of consonants with no vowel, or Devanagari with more half-letter joins than any Hindi word has. Those are taken out; a real word is never touched by this rule.
None of this makes the model hear words it cannot hear. It only stops the worst of what it writes when it cannot.
Problem 2: rap is fast, and loops are easy
Fast delivery packs many syllables into each second, and the model has to place every one. When it loses its place it can fall into a loop, repeating a syllable or a word until the window ends, or produce times that run backwards, or a single "word" that lasts many seconds.
The tools hear each window on its own, so a mistake in one cannot carry into the next and run through the rest of the song. A window that comes out broken is decoded again with stricter settings; if that also fails, it is split at its quietest point and each half is heard separately. On one real song this recovered 28 seconds of rap that the largest model had silently dropped. Anything that still cannot be written is listed with its start and end time, so you know exactly which stretch is missing rather than finding out later.
One side effect to know about: a word repeated three or more times in a row is kept once, because that pattern is almost always a loop. If your lyric really does repeat a word three times, add the repeats back by hand.
Problem 3: Hindi, Urdu and Hinglish
Spoken Hindi and Urdu are very close, and Whisper sometimes writes Hindi speech in Urdu script. For Hindi the tools keep Devanagari and drop words written in the wrong script.
Many people want something else entirely: Hinglish, Hindi words in English letters, the way most people type on a phone. The video editor has a "Hindi captions in" setting with both options. Hinglish is produced by converting the Hindi letters, so spellings follow a consistent rule rather than any one person's habit; expect to adjust a few words to the spelling your audience uses.
Choose the language rather than leaving it on detect. The tools say so themselves: it is faster and more accurate, and on a song, where detection is guessing from a mix of voice and music, it avoids a wrong guess at the start.
Problem 4: the model size matters more on songs
The tools offer several sizes of the same model. On clear speech the differences are modest. On music they are large. The site's own test on a real Hindi rap track with loud music, no lyrics on screen, went like this:
- Base: almost nothing usable. Most lines were invented.
- Small (on a laptop processor): about half the words recognisable, with poor spelling, and it took several minutes for a song of under five minutes.
- Large (on a built-in Intel graphics chip): most lines recognisably right, with a few wrong consonants and words split in the wrong place.
That is why, on a computer, the tools move from Tiny or Base to Small when they hear a song or when you choose Hindi, and the video editor moves to Large when your graphics chip can run it. A model you pick yourself is never changed for you, and nothing larger is downloaded without you pressing the button. The trade-offs are covered in choosing a speech-to-text model.
Even Large did not get every line right on that song. For anything you are going to publish, the next approach is the one to use.
The reliable fix: paste the words you already have
If you wrote the song, or have the lyrics from the artist or a licence, you already know the words. What you do not have is the timing, and timing is a much easier problem for a speech model than recognition. Given the exact words, the model only has to decide where in the sound each one fits best. This is called forced alignment.
Both tools take your text: "Your lyrics or script" in the video editor, "Your text" in Audio to Text. Paste one caption line per line. What happens next:
- Your words are kept exactly as you wrote them. The tool never respells them.
- Blank lines and section labels such as [Chorus] or "Verse 2:" are left out. A line ending in (x2) is sung twice; a line in brackets, such as backing vocals, is kept without the brackets.
- A speech pass first finds roughly where each line falls, matching words in order by how they sound, so a chorus sung twice is placed at both points.
- Each word is then timed to the voice. A word that follows a pause starts where it is heard most clearly, not in the silence before it.
- A line the voice does not clearly confirm is marked "Check this line", with its time, and its words are placed on the points where the voice actually starts sounds rather than spread evenly.
Look at the marked lines first; the rest usually need little or no change. Long lines are split at phrase boundaries to fit the caption length, never before a small word that belongs to the one before it, such as है or में.
A note on rights: lyrics are usually copyrighted separately from the recording. Pasting them to make captions for your own video is one thing; publishing them is another. Use words you have the right to use.
When the words are already in the video
Lyric videos and many music videos already show the words on screen. The video editor looks for text burned into the picture as soon as you add a video. If it finds words along the bottom, your captions go to the top so they do not cover them, and one tap moves them back.
If those words can be read, the editor offers "Words in the video" as the source for the captions. It reads each line as it appears, times it by when it is on screen, and then "Time the words to the voice" times every word to the singing. Reading text off video frames mixes up certain letters, especially leaning ones like t, l, f, i, y and v; where a misread word fits the singing badly and a close spelling fits it clearly better, the closer spelling replaces it. A word that is already a real word is never swapped.
A routine that works
- Choose the language instead of detect.
- If you have the lyrics, paste them. This beats any model size.
- If the video already shows the lyrics, use the words in the video, then time them to the voice.
- Otherwise use Small or Large on a computer, not Tiny or Base.
- Read every line marked for checking, and every gap listed as missing.
- Export as captions burned into the video, or as a subtitle file; which suits where is covered in SRT, VTT or burned-in captions.
Everything above runs in your browser. The model is downloaded once to your device and the song never leaves it.