Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

Rainer Stiefelhagen; Hazim Kemal Ekenel; Christian Fügen; Petra Gieselmann; Hartwig Holzapfel; Florian Kraft; Kai Nickel; Micheal Voit; Alex Waibel

doi:10.1109/TRO.2007.907484

Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

Rainer Stiefelhagen^*
, Hazim Kemal Ekenel
, Christian Fügen
, Petra Gieselmann
, Hartwig Holzapfel
, Florian Kraft
, Kai Nickel
, Micheal Voit
, Alex Waibel

^*Corresponding author for this work

Karlsruhe Institute of Technology
Lucy Software and Services GmbH
Carnegie Mellon University

Research output: Contribution to journal › Article › peer-review

123 Citations (Scopus)

Abstract

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.

Original language	English
Pages (from-to)	840-851
Number of pages	12
Journal	IEEE Transactions on Robotics
Volume	23
Issue number	5
DOIs	https://doi.org/10.1109/TRO.2007.907484
Publication status	Published - 1 Oct 2007
Externally published	Yes

Funding

Manuscript received October 14, 2006; revised May 23, 2007. This paper was recommended for publication by Associate Editor C. Laschi and Editor H. Arai upon evaluation of the reviewers’ comments. This work was supported in part by the German Research Foundation (DFG) under Sonderforschungsbereich SFB 588—Humanoid Robots. This paper was presented in part at the 13th European Signal Processing Conference, Antalya, Turkey, 2005, in part at the ICSLP, Jeju-Islands, Korea, 2004, in part at the International Conference on Multimodal Interfaces (ICMI), State College, 2004, in part at the KI 2006, Bremen, Germany, in part at the INTERSPEECH, Pittsburgh, PA, 2006, in part at the Proceedings of the 7th International Conference on Multimodal Interfaces, Trento, Italy, October 4–6, 2005, in part at the Sixth International Conference on Face and Gesture Recognition—FG 2004, May, Seoul, Korea, and in part at the First International CLEAR Evaluation Workshop, Southampton, U.K., April 2006.

Funders	Funder number
Deutsche Forschungsgemeinschaft	SFB 588

Keywords

Audiovisual perception
Human-centered robotics
Human-robot interaction
Multimodal interaction

Access to Document

10.1109/TRO.2007.907484

Cite this

@article{6762891cde3143229f651c447aa26cf3,

title = "Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot",

abstract = "In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.",

keywords = "Audiovisual perception, Human-centered robotics, Human-robot interaction, Multimodal interaction",

author = "Rainer Stiefelhagen and Ekenel, \{Hazim Kemal\} and Christian F{\"u}gen and Petra Gieselmann and Hartwig Holzapfel and Florian Kraft and Kai Nickel and Micheal Voit and Alex Waibel",

year = "2007",

month = oct,

day = "1",

doi = "10.1109/TRO.2007.907484",

language = "English",

volume = "23",

pages = "840--851",

journal = "IEEE Transactions on Robotics",

issn = "1552-3098",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "5",

}

TY - JOUR

T1 - Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

AU - Stiefelhagen, Rainer

AU - Ekenel, Hazim Kemal

AU - Fügen, Christian

AU - Gieselmann, Petra

AU - Holzapfel, Hartwig

AU - Kraft, Florian

AU - Nickel, Kai

AU - Voit, Micheal

AU - Waibel, Alex

PY - 2007/10/1

Y1 - 2007/10/1

N2 - In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.

AB - In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.

KW - Audiovisual perception

KW - Human-centered robotics

KW - Human-robot interaction

KW - Multimodal interaction

UR - https://www.scopus.com/pages/publications/35348860213

U2 - 10.1109/TRO.2007.907484

DO - 10.1109/TRO.2007.907484

M3 - Article

AN - SCOPUS:35348860213

SN - 1552-3098

VL - 23

SP - 840

EP - 851

JO - IEEE Transactions on Robotics

JF - IEEE Transactions on Robotics

IS - 5

ER -

Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this