Audio-Driven Talking Face Generation with Stabilized Synchronization Loss

Dogucan Yaman; Fevziye Irem Eyiokur; Leonard Bärmann; Hazım Kemal Ekenel; Alexander Waibel

doi:10.1007/978-3-031-72655-2_24

Audio-Driven Talking Face Generation with Stabilized Synchronization Loss

Dogucan Yaman^*, Fevziye Irem Eyiokur, Leonard Bärmann, Hazım Kemal Ekenel, Alexander Waibel

^*Bu çalışma için yazışmadan sorumlu yazar

Bilgisayar Mühendisliği Bölümü

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

Özet

Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.

Orijinal dil	İngilizce
Ana bilgisayar yayını başlığı	Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
Editörler	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Yayınlayan	Springer Science and Business Media Deutschland GmbH
Sayfalar	417-435
Sayfa sayısı	19
ISBN (Basılı)	9783031726545
DOI'lar	https://doi.org/10.1007/978-3-031-72655-2_24
Yayın durumu	Yayınlandı - 2025
Etkinlik	18th European Conference on Computer Vision, ECCV 2024 - Milan, Italy Süre: 29 Eyl 2024 → 4 Eki 2024

Yayın serisi

Adı	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Hacim	15077 LNCS
ISSN (Basılı)	0302-9743
ISSN (Elektronik)	1611-3349

???event.eventtypes.event.conference???

???event.eventtypes.event.conference???	18th European Conference on Computer Vision, ECCV 2024
Ülke/Bölge	Italy
Şehir	Milan
Periyot	29/09/24 → 4/10/24

Bibliyografik not

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Belgeye Erişim

10.1007/978-3-031-72655-2_24

Diğer Dosyalar ve Linkler

Link to publication in Scopus

Alıntı Yap

Yaman, D., Eyiokur, F. I., Bärmann, L., Ekenel, H. K., & Waibel, A. (2025). Audio-Driven Talking Face Generation with Stabilized Synchronization Loss. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (pp. 417-435). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Hac. 15077 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-72655-2_24

Yaman, Dogucan ; Eyiokur, Fevziye Irem ; Bärmann, Leonard et al. / Audio-Driven Talking Face Generation with Stabilized Synchronization Loss. Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. editör / Aleš Leonardis ; Elisa Ricci ; Stefan Roth ; Olga Russakovsky ; Torsten Sattler ; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. pp. 417-435 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{105af4c9195346e2b37a4e2d2039ea0e,

title = "Audio-Driven Talking Face Generation with Stabilized Synchronization Loss",

abstract = "Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.",

keywords = "Lip leaking, Lip synchronization, Talking face generation",

author = "Dogucan Yaman and Eyiokur, {Fevziye Irem} and Leonard B{\"a}rmann and Ekenel, {Hazım Kemal} and Alexander Waibel",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 18th European Conference on Computer Vision, ECCV 2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2025",

doi = "10.1007/978-3-031-72655-2_24",

language = "English",

isbn = "9783031726545",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "417--435",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Olga Russakovsky and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision – ECCV 2024 - 18th European Conference, Proceedings",

address = "Germany",

}

Yaman, D, Eyiokur, FI, Bärmann, L, Ekenel, HK & Waibel, A 2025, Audio-Driven Talking Face Generation with Stabilized Synchronization Loss. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), hac. 15077 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 417-435, 18th European Conference on Computer Vision, ECCV 2024, Milan, Italy, 29/09/24. https://doi.org/10.1007/978-3-031-72655-2_24

Audio-Driven Talking Face Generation with Stabilized Synchronization Loss. / Yaman, Dogucan; Eyiokur, Fevziye Irem; Bärmann, Leonard et al.
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer Science and Business Media Deutschland GmbH, 2025. p. 417-435 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Hac. 15077 LNCS).

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

TY - GEN

T1 - Audio-Driven Talking Face Generation with Stabilized Synchronization Loss

AU - Yaman, Dogucan

AU - Eyiokur, Fevziye Irem

AU - Bärmann, Leonard

AU - Ekenel, Hazım Kemal

AU - Waibel, Alexander

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.

AB - Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.

KW - Lip leaking

KW - Lip synchronization

KW - Talking face generation

UR - http://www.scopus.com/inward/record.url?scp=85213044117&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-72655-2_24

DO - 10.1007/978-3-031-72655-2_24

M3 - Conference contribution

AN - SCOPUS:85213044117

SN - 9783031726545

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 417

EP - 435

BT - Computer Vision – ECCV 2024 - 18th European Conference, Proceedings

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer Science and Business Media Deutschland GmbH

T2 - 18th European Conference on Computer Vision, ECCV 2024

Y2 - 29 September 2024 through 4 October 2024

ER -

Yaman D, Eyiokur FI, Bärmann L, Ekenel HK, Waibel A. Audio-Driven Talking Face Generation with Stabilized Synchronization Loss. In Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editörler, Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2025. p. 417-435. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72655-2_24