An audiovisual interface-based drumming system for multimodal human–robot interaction

Gökhan Ince*, Rabia Yorganci, Ahmet Ozkul, Taha Berkay Duman, Hatice Köse

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)

Abstract

This paper presents a study of an audiovisual interface-based drumming system for multimodal human–robot interaction. The interactive multimodal drumming game is used in conjunction with humanoid robots to establish an audiovisual interactive interface. This study is part of a project to design robot and avatar assistants for education and therapy, especially for children with special needs. It specifically focuses on evaluating robot/virtual avatar tutors, tangible interaction devices, and mobile multimedia devices within a simple drumming-based interactive music tutoring game scenario. Several parameters, including the effect of the embodiment of the tutor/interface and the presence of feedback and training mechanisms, were the focus of interest. For that purpose, we created an interactive drumming game relying on turn-taking and imitation principles, in which a human user is able to play drums with a humanoid robot (Robotic Drum Mate). Three interactive scenarios with different experimental setups for humanoid robots and mobile devices were developed and tested. As a part of those scenarios, a system that enables drum strokes to be automatically detected and recognized using auditory cues was implemented and incorporated into the experimental framework. We verified the applicability and effectiveness of the proposed system in a drum-playing game with adult human test subjects by evaluating it both objectively and subjectively. The results showed that the physical robot tutor, the feedback and training mechanisms had a positive effect on the subjects’ performance and, as expected, although the physical medium is preferred, the virtual medium for drumming caused less failure.

Original languageEnglish
Pages (from-to)413-428
Number of pages16
JournalJournal on Multimodal User Interfaces
Volume15
Issue number4
DOIs
Publication statusPublished - Dec 2021

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Funding

This work is supported by the Scientific Research Project Unit (BAP) of Istanbul Technical University, Project Number: MOA-2019-42321.

FundersFunder number
Istanbul Teknik ÜniversitesiMOA-2019-42321

    Keywords

    • Audio processing
    • Audiovisual interaction
    • Drum game
    • Human–computer-interaction (HCI)
    • Human–robot-interaction (HRI)
    • Multimodality
    • Non-verbal communication

    Fingerprint

    Dive into the research topics of 'An audiovisual interface-based drumming system for multimodal human–robot interaction'. Together they form a unique fingerprint.

    Cite this