Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Alexander Waibel; Moritz Behr; Dogucan Yaman; Fevziye Irem Eyiokur; Tuan Nam Nguyen; Carlos Mullov; Mehmet Arif Demirtas; Alperen Kantarci; Stefan Constantin; Hazim Kemal Ekenel

doi:10.1109/ICASSPW59220.2023.10193719

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Alexander Waibel^*, Moritz Behr, Dogucan Yaman, Fevziye Irem Eyiokur, Tuan Nam Nguyen, Carlos Mullov, Mehmet Arif Demirtas, Alperen Kantarci, Stefan Constantin, Hazim Kemal Ekenel

^*Corresponding author for this work

Department of Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Citations (Scopus)

Abstract

In this paper, we propose a neural end-to-end system for voice preserving and lip-synchronous video translation. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, and face video of the original speaker. The result is a video of a speaker speaking in another language without actually knowing it. For the evaluation, we present a user study of the complete system and separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set to evaluate our system. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics.

Original language	English
Title of host publication	ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350302615
DOIs	https://doi.org/10.1109/ICASSPW59220.2023.10193719
Publication status	Published - 2023
Event	2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023 - Rhodes Island, Greece Duration: 4 Jun 2023 → 10 Jun 2023

Publication series

Name	ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings

Conference

Conference	2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023
Country/Territory	Greece
City	Rhodes Island
Period	4/06/23 → 10/06/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

end-to-end video translation
lip generation
speech translation
text-to-speech
voice conversion

Access to Document

10.1109/ICASSPW59220.2023.10193719

Cite this

Waibel, A., Behr, M., Yaman, D., Eyiokur, F. I., Nguyen, T. N., Mullov, C., Demirtas, M. A., Kantarci, A., Constantin, S., & Ekenel, H. K. (2023). Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos. In ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings (ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSPW59220.2023.10193719

Waibel, Alexander ; Behr, Moritz ; Yaman, Dogucan et al. / Face-Dubbing++ : LIP-Synchronous, Voice Preserving Translation Of Videos. ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. (ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings).

@inproceedings{2639a1173cf84dd28a4bb618724af226,

title = "Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos",

abstract = "In this paper, we propose a neural end-to-end system for voice preserving and lip-synchronous video translation. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, and face video of the original speaker. The result is a video of a speaker speaking in another language without actually knowing it. For the evaluation, we present a user study of the complete system and separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set to evaluate our system. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics.",

keywords = "end-to-end video translation, lip generation, speech translation, text-to-speech, voice conversion",

author = "Alexander Waibel and Moritz Behr and Dogucan Yaman and Eyiokur, {Fevziye Irem} and Nguyen, {Tuan Nam} and Carlos Mullov and Demirtas, {Mehmet Arif} and Alperen Kantarci and Stefan Constantin and Ekenel, {Hazim Kemal}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023 ; Conference date: 04-06-2023 Through 10-06-2023",

year = "2023",

doi = "10.1109/ICASSPW59220.2023.10193719",

language = "English",

series = "ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings",

address = "United States",

}

Waibel, A, Behr, M, Yaman, D, Eyiokur, FI, Nguyen, TN, Mullov, C, Demirtas, MA, Kantarci, A, Constantin, S & Ekenel, HK 2023, Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos. in ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings. ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings, Institute of Electrical and Electronics Engineers Inc., 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023, Rhodes Island, Greece, 4/06/23. https://doi.org/10.1109/ICASSPW59220.2023.10193719

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos. / Waibel, Alexander; Behr, Moritz; Yaman, Dogucan et al.
ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. (ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Face-Dubbing++

T2 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, ICASSPW 2023

AU - Waibel, Alexander

AU - Behr, Moritz

AU - Yaman, Dogucan

AU - Eyiokur, Fevziye Irem

AU - Nguyen, Tuan Nam

AU - Mullov, Carlos

AU - Demirtas, Mehmet Arif

AU - Kantarci, Alperen

AU - Constantin, Stefan

AU - Ekenel, Hazim Kemal

PY - 2023

Y1 - 2023

N2 - In this paper, we propose a neural end-to-end system for voice preserving and lip-synchronous video translation. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, and face video of the original speaker. The result is a video of a speaker speaking in another language without actually knowing it. For the evaluation, we present a user study of the complete system and separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set to evaluate our system. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics.

AB - In this paper, we propose a neural end-to-end system for voice preserving and lip-synchronous video translation. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, and face video of the original speaker. The result is a video of a speaker speaking in another language without actually knowing it. For the evaluation, we present a user study of the complete system and separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set to evaluate our system. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics.

KW - end-to-end video translation

KW - lip generation

KW - speech translation

KW - text-to-speech

KW - voice conversion

UR - http://www.scopus.com/inward/record.url?scp=85168248816&partnerID=8YFLogxK

U2 - 10.1109/ICASSPW59220.2023.10193719

DO - 10.1109/ICASSPW59220.2023.10193719

M3 - Conference contribution

AN - SCOPUS:85168248816

T3 - ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings

BT - ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 4 June 2023 through 10 June 2023

ER -

Waibel A, Behr M, Yaman D, Eyiokur FI, Nguyen TN, Mullov C et al. Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos. In ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings. Institute of Electrical and Electronics Engineers Inc. 2023. (ICASSPW 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops, Proceedings). doi: 10.1109/ICASSPW59220.2023.10193719

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this