A new facial expression processing system for an affectively aware robot

Engin Baglayici; Cemal Gurpinar; Pinar Uluer; Hatice Kose

doi:10.1007/978-3-030-68790-8_4

A new facial expression processing system for an affectively aware robot

Engin Baglayici^*, Cemal Gurpinar, Pinar Uluer, Hatice Kose

^*Corresponding author for this work

Department of Artificial Intelligence and Data Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

Abstract

This paper introduces an emotion recognition system for an affectively aware hospital robot for children, and a data labeling and processing tool called LabelFace for facial expression recognition (FER) to be employed within the presented system. The tool provides an interface for automatic/manual labeling and visual information processing for emotion and facial action unit (AU) recognition with the assistant models based on deep learning. The tool is developed primarily to support the affective intelligence of a socially assistive robot for supporting the healthcare of children with hearing impairments. In the proposed approach, multi-label AU detection models are used for this purpose. To the best of our knowledge, the proposed children AU detector model is the first model which targets 5-to 9-year old children. The model is trained with well-known posed-datasets and tested with a real-world non-posed dataset collected from hearing-impaired children. Our tool LabelFace is compared to a widely-used facial expression tool in terms of data processing and data labeling capabilities for benchmarking, and performs better with its AU detector models for children on both posed-data and non-posed data testing.

Original language	English
Title of host publication	Pattern Recognition - ICPR International Workshops and Challenges, Proceedings
Editors	Alberto Del Bimbo, Marco Bertini, Stan Sclaroff, Tao Mei, Hugo Jair Escalante, Rita Cucchiara, Roberto Vezzani, Giovanni Maria Farinella
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	36-51
Number of pages	16
ISBN (Print)	9783030687892
DOIs	https://doi.org/10.1007/978-3-030-68790-8_4
Publication status	Published - 2021
Event	25th International Conference on Pattern Recognition Workshops, ICPR 2020 - Virtual, Online Duration: 10 Jan 2021 → 15 Jan 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12662 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	25th International Conference on Pattern Recognition Workshops, ICPR 2020
City	Virtual, Online
Period	10/01/21 → 15/01/21

Bibliographical note

Publisher Copyright:
© Springer Nature Switzerland AG 2021.

Funding

Acknowledgment. This study is supported by the Scientific and Technological Research Council of Turkey (TUBITAK), RoboRehab project, under contract no 118E214.

Funders	Funder number
TUBITAK	118E214
Türkiye Bilimsel ve Teknolojik Araştirma Kurumu

Keywords

Action unit recognition
Affective computing
Child-robot interaction
Facial expression recognition

Access to Document

10.1007/978-3-030-68790-8_4

Cite this

Baglayici, E., Gurpinar, C., Uluer, P., & Kose, H. (2021). A new facial expression processing system for an affectively aware robot. In A. Del Bimbo, M. Bertini, S. Sclaroff, T. Mei, H. J. Escalante, R. Cucchiara, R. Vezzani, & G. M. Farinella (Eds.), Pattern Recognition - ICPR International Workshops and Challenges, Proceedings (pp. 36-51). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12662 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-68790-8_4

Baglayici, Engin ; Gurpinar, Cemal ; Uluer, Pinar et al. / A new facial expression processing system for an affectively aware robot. Pattern Recognition - ICPR International Workshops and Challenges, Proceedings. editor / Alberto Del Bimbo ; Marco Bertini ; Stan Sclaroff ; Tao Mei ; Hugo Jair Escalante ; Rita Cucchiara ; Roberto Vezzani ; Giovanni Maria Farinella. Springer Science and Business Media Deutschland GmbH, 2021. pp. 36-51 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{8895c261ed6b4d76b6844367037fd500,

title = "A new facial expression processing system for an affectively aware robot",

abstract = "This paper introduces an emotion recognition system for an affectively aware hospital robot for children, and a data labeling and processing tool called LabelFace for facial expression recognition (FER) to be employed within the presented system. The tool provides an interface for automatic/manual labeling and visual information processing for emotion and facial action unit (AU) recognition with the assistant models based on deep learning. The tool is developed primarily to support the affective intelligence of a socially assistive robot for supporting the healthcare of children with hearing impairments. In the proposed approach, multi-label AU detection models are used for this purpose. To the best of our knowledge, the proposed children AU detector model is the first model which targets 5-to 9-year old children. The model is trained with well-known posed-datasets and tested with a real-world non-posed dataset collected from hearing-impaired children. Our tool LabelFace is compared to a widely-used facial expression tool in terms of data processing and data labeling capabilities for benchmarking, and performs better with its AU detector models for children on both posed-data and non-posed data testing.",

keywords = "Action unit recognition, Affective computing, Child-robot interaction, Facial expression recognition",

author = "Engin Baglayici and Cemal Gurpinar and Pinar Uluer and Hatice Kose",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2021.; 25th International Conference on Pattern Recognition Workshops, ICPR 2020 ; Conference date: 10-01-2021 Through 15-01-2021",

year = "2021",

doi = "10.1007/978-3-030-68790-8_4",

language = "English",

isbn = "9783030687892",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "36--51",

editor = "{Del Bimbo}, Alberto and Marco Bertini and Stan Sclaroff and Tao Mei and Escalante, {Hugo Jair} and Rita Cucchiara and Roberto Vezzani and Farinella, {Giovanni Maria}",

booktitle = "Pattern Recognition - ICPR International Workshops and Challenges, Proceedings",

address = "Germany",

}

Baglayici, E, Gurpinar, C, Uluer, P & Kose, H 2021, A new facial expression processing system for an affectively aware robot. in A Del Bimbo, M Bertini, S Sclaroff, T Mei, HJ Escalante, R Cucchiara, R Vezzani & GM Farinella (eds), Pattern Recognition - ICPR International Workshops and Challenges, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12662 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 36-51, 25th International Conference on Pattern Recognition Workshops, ICPR 2020, Virtual, Online, 10/01/21. https://doi.org/10.1007/978-3-030-68790-8_4

A new facial expression processing system for an affectively aware robot. / Baglayici, Engin; Gurpinar, Cemal; Uluer, Pinar et al.
Pattern Recognition - ICPR International Workshops and Challenges, Proceedings. ed. / Alberto Del Bimbo; Marco Bertini; Stan Sclaroff; Tao Mei; Hugo Jair Escalante; Rita Cucchiara; Roberto Vezzani; Giovanni Maria Farinella. Springer Science and Business Media Deutschland GmbH, 2021. p. 36-51 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12662 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A new facial expression processing system for an affectively aware robot

AU - Baglayici, Engin

AU - Gurpinar, Cemal

AU - Uluer, Pinar

AU - Kose, Hatice

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2021.

PY - 2021

Y1 - 2021

N2 - This paper introduces an emotion recognition system for an affectively aware hospital robot for children, and a data labeling and processing tool called LabelFace for facial expression recognition (FER) to be employed within the presented system. The tool provides an interface for automatic/manual labeling and visual information processing for emotion and facial action unit (AU) recognition with the assistant models based on deep learning. The tool is developed primarily to support the affective intelligence of a socially assistive robot for supporting the healthcare of children with hearing impairments. In the proposed approach, multi-label AU detection models are used for this purpose. To the best of our knowledge, the proposed children AU detector model is the first model which targets 5-to 9-year old children. The model is trained with well-known posed-datasets and tested with a real-world non-posed dataset collected from hearing-impaired children. Our tool LabelFace is compared to a widely-used facial expression tool in terms of data processing and data labeling capabilities for benchmarking, and performs better with its AU detector models for children on both posed-data and non-posed data testing.

AB - This paper introduces an emotion recognition system for an affectively aware hospital robot for children, and a data labeling and processing tool called LabelFace for facial expression recognition (FER) to be employed within the presented system. The tool provides an interface for automatic/manual labeling and visual information processing for emotion and facial action unit (AU) recognition with the assistant models based on deep learning. The tool is developed primarily to support the affective intelligence of a socially assistive robot for supporting the healthcare of children with hearing impairments. In the proposed approach, multi-label AU detection models are used for this purpose. To the best of our knowledge, the proposed children AU detector model is the first model which targets 5-to 9-year old children. The model is trained with well-known posed-datasets and tested with a real-world non-posed dataset collected from hearing-impaired children. Our tool LabelFace is compared to a widely-used facial expression tool in terms of data processing and data labeling capabilities for benchmarking, and performs better with its AU detector models for children on both posed-data and non-posed data testing.

KW - Action unit recognition

KW - Affective computing

KW - Child-robot interaction

KW - Facial expression recognition

UR - http://www.scopus.com/inward/record.url?scp=85110646692&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-68790-8_4

DO - 10.1007/978-3-030-68790-8_4

M3 - Conference contribution

AN - SCOPUS:85110646692

SN - 9783030687892

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 36

EP - 51

BT - Pattern Recognition - ICPR International Workshops and Challenges, Proceedings

A2 - Del Bimbo, Alberto

A2 - Bertini, Marco

A2 - Sclaroff, Stan

A2 - Mei, Tao

A2 - Escalante, Hugo Jair

A2 - Cucchiara, Rita

A2 - Vezzani, Roberto

A2 - Farinella, Giovanni Maria

PB - Springer Science and Business Media Deutschland GmbH

T2 - 25th International Conference on Pattern Recognition Workshops, ICPR 2020

Y2 - 10 January 2021 through 15 January 2021

ER -

Baglayici E, Gurpinar C, Uluer P, Kose H. A new facial expression processing system for an affectively aware robot. In Del Bimbo A, Bertini M, Sclaroff S, Mei T, Escalante HJ, Cucchiara R, Vezzani R, Farinella GM, editors, Pattern Recognition - ICPR International Workshops and Challenges, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 36-51. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-68790-8_4

A new facial expression processing system for an affectively aware robot

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this