By Tat Banerjee
Today we are going to look at AI dubbing and its various use cases. If you are wondering, “is my content suitable for AI dubbing or AI localisation?”
In simple terms:
- What is dubbing or dub localisation?
- Should I use a human or a machine for my dubbing?
- How do I use a videotranslator.ai to dub my video?
- What are the rules of thumb, or heuristics to achieve a superior outcome with AI dubbing?
On reviewing this visual guide, you will be able to:
- Upload your own video content
- Transcribe your content
- Dub your content into a different dialect
Note: We are going to take a video in Australian English and switch it to American English.
Obviously, the usual use case is to translate into another language first, and then do the dubbing. We forego the translation, for the purpose of making this guide easier to understand.
From Wikipedia, ‘dubbing, mixing, or re-recording is a post-production process used in film making and video production in which additional or supplementary recordings are “mixed” with original production sound to create the finished soundtrack.’
In everyday usage however, the term ‘dubbing’ commonly refers to the replacement of actor’s voices with those of different performers, speaking another language. This is called re-voicing, or a voice-over in the film industry.
Outcomes — Before and After
To make it obvious what we are talking about, the after is provided.
Steps: Transcribe your Content Using the Provided AI
- We are working with and English video today, so select the English template and create a new item. Note that the account we are using to show this functionality has Chinese, French and Vietnamese as translation options. We will not be using this functionality today.
- This is the original content:
- We know the speaker is Australian, so we select English (Australia) as the preferred dialect to transcribe with. Picking the correct dialect makes AI accuracy much better.
- Once transcription is complete, use the Picture-In-Picture (PIP) to clean up the transcript. Note that the text shown in the PIP is real time, so you can edit both timestamps/text of the transcript and see the changes directly.
- You can see we removed text fragment 9 and merged it into text fragment 8 as an example. This makes no difference to the rendering of the captions. Use the Auto-Index button to fix this up if you require your *.srt file to work properly.
- Save your changes and your transcription is complete. If you need the captions burnt into the asset, use the Auto-Overlay. Nicely done!
Steps: Dubbing/Adding a Voice Over with AI
- To dub the content using a different voice, our first step is to copy the captions from our video component into our audio component. Click into your editor and hit Ctrl-A and Ctrl-C.
- Go to the Audio component, click add captions and paste your content. Use the highlighted button to add captions without adding an audio file.
- We will transcribe this time again using the Text-To-Speech transcription AI. In the below image, we use the Text-To-Speech tab. It is also possible to break up your captions into fragments, as shown below. We will only use the first fragment, but you can break your content to produce different *.mp3 files.
- A few quick notes. The options shown are the different voices, with (M) for Male and (F) for Female. The accent can also be seen and we are using English (US) (M) which is American English Male. The S/P indicates Standard or Professional. We recommend Professional as Standard is simply an older technology (sounds more robotic). Once done, it looks like below.
- The audio content is below. Have a listen! Sweet – dubbing is complete.
Steps: Overlaying our Audio on our Video
- We are now going to add our generated audio to our original video. Go back to your video component, and click on Audio Overlays. Click the + and your audio fragment will be shown.
- Notice, we have not set any times. Add (in seconds) the time. We added 0.5 and 21 as start and end times, and then clicked the + button to bake in the audio overlay. That is the entire process. The final output is shown below.
In this visual guide, we used our video translator to dub, or overlay a different voice on our video. It should be noted, that most clients prefer to download assets and do the last audio overlay step themselves. But you can use our app if you think it makes sense! Send us some feedback if you do.
If you are interested in trying out our technology, please try our platform or drop us an email at firstname.lastname@example.org.