Dubformer announced the launch of Emotion Transfer, its proprietary AI dubbing technology that delivers highly emotional and expressive voiceover, regardless of voice or timbre. Unlike existing AI dubbing solutions, it reproduces the emotional dynamics of speech – such as pitch, volume, and speed – breaking down language barriers with natural-sounding dubbing.
Problem and solution
Despite advancements in AI dubbing technology, achieving high-quality emotional expression has remained a key challenge. Existing approaches, which rely on voice cloning or pre-recorded voices, often produce flat, unnatural dubbing that fails to convey the emotional depth of original performances.
Dubformer’s Emotion Transfer technology addresses this problem by transferring emotions and intonations instead of replicating voice and timbre, making voice overs as expressive as the original performance. This innovation is essential for high-quality productions like TV shows, series, and animation, where emotional depth plays a crucial role in viewer engagement.
The future of Emotion Transfer technology holds great promise with its future potential to transfer emotions through different voices, capturing only the emotional essence without copying the actor’s voice. This development could transform the industry, especially as the use of voice cloning technology has sparked legal and ethical concerns, with voice actors finding their voices cloned without consent.
How it works
AI dubbing technology has come a long way. Earlier technologies like text-to-speech (TTS) were useful for simple tasks but lacked emotional depth. Voice cloning took it a step further, creating digital replicas of voices, though with risks surrounding the misuse of actor data. Now, with Emotion Transfer technology, AI dubbing reaches a new frontier, capturing emotions instead of replicating voices.
Emotion Transfer separates the elements of speech that convey emotion – such as pitch, volume, and speed – from those related to the timbre, which is unique to each individual voice, and transfers them. It processes each phrase individually, using the original voice as a reference to capture the emotional essence of the performance, similar to the process professional dubbing actors follow before delivering their lines. The technology cannot operate beyond the specific content it is used to process, nor does it use actor data for training. Emotion Transfer is capable of translating 20 languages into English, German, Spanish, and French.
“Until now, AI dubbing has struggled to capture the emotional nuances that human voice actors bring to their performances. With Emotion Transfer, we elevate the entire AI dubbing process by focusing on the emotions behind each phrase, enhancing the expressiveness of the output. By transferring emotional dynamics instead of cloning voices, we’re able to create a result that’s both more engaging and true to the original performance. This approach represents a significant leap in AI dubbing technology,” said Anton Dvorkovich, Dubformer’s CEO and Founder.
Emotion Transfer is also available for users and creators on Dubformer’s open platform.
