Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Marina Zimmermann; Mostafa Mehdipour Ghazi; Hazım Kemal Ekenel; Jean Philippe Thiran

doi:10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Marina Zimmermann^*
, Mostafa Mehdipour Ghazi
, Hazım Kemal Ekenel
, Jean Philippe Thiran

^*Bu çalışma için yazışmadan sorumlu yazar

Bilgisayar Mühendisliği Bölümü

Swiss Federal Institute of Technology Lausanne
Sabanci University

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

10 Atıf (Scopus)

Özet

Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

Orijinal dil	İngilizce
Ana bilgisayar yayını başlığı	Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers
Editörler	Kai-Kuang Ma, Jiwen Lu, Chu-Song Chen
Yayınlayan	Springer Verlag
Sayfalar	264-276
Sayfa sayısı	13
ISBN (Basılı)	9783319544267
DOI'lar	https://doi.org/10.1007/978-3-319-54427-4_20
Yayın durumu	Yayınlandı - 2017
Etkinlik	13th Asian Conference on Computer Vision, ACCV 2016 - Taipei, Taiwan, Province of China Süre: 20 Kas 2016 → 24 Kas 2016

Yayın serisi

Adı	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Hacim	10117 LNCS
ISSN (Basılı)	0302-9743
ISSN (Elektronik)	1611-3349

???event.eventtypes.event.conference???

???event.eventtypes.event.conference???	13th Asian Conference on Computer Vision, ACCV 2016
Ülke/Bölge	Taiwan, Province of China
Şehir	Taipei
Periyot	20/11/16 → 24/11/16

Bibliyografik not

Publisher Copyright:
© Springer International Publishing AG 2017.

Finansman

This work was supported by TUBITAK project number 113E067 and by a Marie Curie FP7 Integration Grant within the 7th EU Framework Programme.

Finansörler	Finansör numarası
TUBITAK	113E067
Seventh Framework Programme
FP7 People: Marie-Curie Actions

Belgeye Erişim

10.1007/978-3-319-54427-4_20

Diğer Dosyalar ve Linkler

Link to publication in Scopus

Alıntı Yap

Zimmermann, M., Mehdipour Ghazi, M., Ekenel, H. K., & Thiran, J. P. (2017). Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. In K.-K. Ma, J. Lu, & C.-S. Chen (Eds.), Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers (pp. 264-276). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Hac. 10117 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-54427-4_20

Zimmermann, Marina ; Mehdipour Ghazi, Mostafa ; Ekenel, Hazım Kemal et al. / Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. editör / Kai-Kuang Ma ; Jiwen Lu ; Chu-Song Chen. Springer Verlag, 2017. pp. 264-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a89176bc561e480496b4264a9dde8c1f,

title = "Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system",

abstract = "Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79\% and 73\%, respectively, as compared to the baseline of 74\% on cross-validation.",

author = "Marina Zimmermann and \{Mehdipour Ghazi\}, Mostafa and Ekenel, \{Hazım Kemal\} and Thiran, \{Jean Philippe\}",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017.; 13th Asian Conference on Computer Vision, ACCV 2016 ; Conference date: 20-11-2016 Through 24-11-2016",

year = "2017",

doi = "10.1007/978-3-319-54427-4\_20",

language = "English",

isbn = "9783319544267",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "264--276",

editor = "Kai-Kuang Ma and Jiwen Lu and Chu-Song Chen",

booktitle = "Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers",

address = "Germany",

}

Zimmermann, M, Mehdipour Ghazi, M, Ekenel, HK & Thiran, JP 2017, Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. in K-K Ma, J Lu & C-S Chen (eds), Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), hac. 10117 LNCS, Springer Verlag, pp. 264-276, 13th Asian Conference on Computer Vision, ACCV 2016, Taipei, Taiwan, Province of China, 20/11/16. https://doi.org/10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. / Zimmermann, Marina; Mehdipour Ghazi, Mostafa; Ekenel, Hazım Kemal et al.
Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. ed. / Kai-Kuang Ma; Jiwen Lu; Chu-Song Chen. Springer Verlag, 2017. p. 264-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Hac. 10117 LNCS).

Araştırma sonucu: Kitap/Rapor/Konferans Bildirisinde Bölüm › Konferans katkısı › bilirkişi

TY - GEN

T1 - Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

AU - Zimmermann, Marina

AU - Mehdipour Ghazi, Mostafa

AU - Ekenel, Hazım Kemal

AU - Thiran, Jean Philippe

PY - 2017

Y1 - 2017

N2 - Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

AB - Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

UR - https://www.scopus.com/pages/publications/85016096334

U2 - 10.1007/978-3-319-54427-4_20

DO - 10.1007/978-3-319-54427-4_20

M3 - Conference contribution

AN - SCOPUS:85016096334

SN - 9783319544267

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 264

EP - 276

BT - Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers

A2 - Ma, Kai-Kuang

A2 - Lu, Jiwen

A2 - Chen, Chu-Song

PB - Springer Verlag

T2 - 13th Asian Conference on Computer Vision, ACCV 2016

Y2 - 20 November 2016 through 24 November 2016

ER -

Zimmermann M, Mehdipour Ghazi M, Ekenel HK, Thiran JP. Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. In Ma KK, Lu J, Chen CS, editörler, Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. Springer Verlag. 2017. p. 264-276. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Özet

Yayın serisi

???event.eventtypes.event.conference???

Bibliyografik not

Finansman

Belgeye Erişim

Diğer Dosyalar ve Linkler

Parmak izi

Alıntı Yap