Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Alexander Waibel*, Moritz Behr, Dogucan Yaman, Fevziye Irem Eyiokur, Tuan Nam Nguyen, Carlos Mullov, Mehmet Arif Demirtas, Alperen Kantarci, Stefan Constantin, Hazim Kemal Ekenel

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In this paper, we propose a neural end-to-end system for voice preserving and lip-synchronous video translation. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, and face video of the original speaker. The result is a video of a speaker speaking in another language without actually knowing it. For the evaluation, we present a user study of the complete system and separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set to evaluate our system. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics.

Original languageEnglish
Title of host publicationICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9798350302615
DOIs
Publication statusPublished - 2023
Event2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023 - Rhodes Island, Greece
Duration: 4 Jun 202310 Jun 2023

Publication series

NameICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings

Conference

Conference2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023
Country/TerritoryGreece
CityRhodes Island
Period4/06/2310/06/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

  • end-to-end video translation
  • lip generation
  • speech translation
  • text-to-speech
  • voice conversion

Fingerprint

Dive into the research topics of 'Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos'. Together they form a unique fingerprint.

Cite this