Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Marina Zimmermann; Mostafa Mehdipour Ghazi; Hazım Kemal Ekenel; Jean Philippe Thiran

doi:10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Marina Zimmermann^*, Mostafa Mehdipour Ghazi, Hazım Kemal Ekenel, Jean Philippe Thiran

^*Corresponding author for this work

Department of Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Citations (Scopus)

Abstract

Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

Original language	English
Title of host publication	Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers
Editors	Kai-Kuang Ma, Jiwen Lu, Chu-Song Chen
Publisher	Springer Verlag
Pages	264-276
Number of pages	13
ISBN (Print)	9783319544267
DOIs	https://doi.org/10.1007/978-3-319-54427-4_20
Publication status	Published - 2017
Event	13th Asian Conference on Computer Vision, ACCV 2016 - Taipei, Taiwan, Province of China Duration: 20 Nov 2016 → 24 Nov 2016

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10117 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	13th Asian Conference on Computer Vision, ACCV 2016
Country/Territory	Taiwan, Province of China
City	Taipei
Period	20/11/16 → 24/11/16

Bibliographical note

Publisher Copyright:
© Springer International Publishing AG 2017.

Access to Document

10.1007/978-3-319-54427-4_20

Cite this

Zimmermann, M., Mehdipour Ghazi, M., Ekenel, H. K., & Thiran, J. P. (2017). Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. In K.-K. Ma, J. Lu, & C.-S. Chen (Eds.), Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers (pp. 264-276). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10117 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-54427-4_20

Zimmermann, Marina ; Mehdipour Ghazi, Mostafa ; Ekenel, Hazım Kemal et al. / Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. editor / Kai-Kuang Ma ; Jiwen Lu ; Chu-Song Chen. Springer Verlag, 2017. pp. 264-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a89176bc561e480496b4264a9dde8c1f,

title = "Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system",

abstract = "Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.",

author = "Marina Zimmermann and {Mehdipour Ghazi}, Mostafa and Ekenel, {Hazım Kemal} and Thiran, {Jean Philippe}",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017.; 13th Asian Conference on Computer Vision, ACCV 2016 ; Conference date: 20-11-2016 Through 24-11-2016",

year = "2017",

doi = "10.1007/978-3-319-54427-4_20",

language = "English",

isbn = "9783319544267",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "264--276",

editor = "Kai-Kuang Ma and Jiwen Lu and Chu-Song Chen",

booktitle = "Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers",

address = "Germany",

}

Zimmermann, M, Mehdipour Ghazi, M, Ekenel, HK & Thiran, JP 2017, Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. in K-K Ma, J Lu & C-S Chen (eds), Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10117 LNCS, Springer Verlag, pp. 264-276, 13th Asian Conference on Computer Vision, ACCV 2016, Taipei, Taiwan, Province of China, 20/11/16. https://doi.org/10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. / Zimmermann, Marina; Mehdipour Ghazi, Mostafa; Ekenel, Hazım Kemal et al.
Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. ed. / Kai-Kuang Ma; Jiwen Lu; Chu-Song Chen. Springer Verlag, 2017. p. 264-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10117 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

AU - Zimmermann, Marina

AU - Mehdipour Ghazi, Mostafa

AU - Ekenel, Hazım Kemal

AU - Thiran, Jean Philippe

N1 - Publisher Copyright: © Springer International Publishing AG 2017.

PY - 2017

Y1 - 2017

N2 - Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

AB - Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

UR - http://www.scopus.com/inward/record.url?scp=85016096334&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-54427-4_20

DO - 10.1007/978-3-319-54427-4_20

M3 - Conference contribution

AN - SCOPUS:85016096334

SN - 9783319544267

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 264

EP - 276

BT - Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers

A2 - Ma, Kai-Kuang

A2 - Lu, Jiwen

A2 - Chen, Chu-Song

PB - Springer Verlag

T2 - 13th Asian Conference on Computer Vision, ACCV 2016

Y2 - 20 November 2016 through 24 November 2016

ER -

Zimmermann M, Mehdipour Ghazi M, Ekenel HK, Thiran JP. Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system. In Ma KK, Lu J, Chen CS, editors, Computer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers. Springer Verlag. 2017. p. 264-276. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-54427-4_20

Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this