Audio-visual signal processing in a multimodal assisted living environment

Alexey Karpov*, Lale Akarun, Hülya Yalcin, Alexander Ronzhin, Bariş Evrim Demiröz, Aysun Çoban, Miloš Železný

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

12 Citations (Scopus)


In this paper, we present some novel methods and applications for audio and video signal processing for a multimodal environment of an assisted living smart space. This intelligent environment was developed during the 7th Summer Workshop on Multimodal Interfaces eNTERFACE. It integrates automatic systems for audio and video-based monitoring and user tracking in the smart space. In the assisted living environment, users are tracked by some omnidirectional video cameras, as well as speech and non-speech audio events are recognized by an array of microphones. The multiple objects tracking precision (MOTP) of the developed video monitoring system was 0.78 and 0.73 and the multiple objects tracking accuracy (MOTA) was 62.81% and 72.31% for single person and three people scenarios, respectively. The recognition accuracy of the proposed multilingual speech and audio events recognition system was 96.5% and 93.8% for user's speech commands and non-speech acoustic events, correspondingly. The design of the assisted living environment, the certain test scenarios and the process of audio-visual database collection are described in the paper.

Original languageEnglish
Pages (from-to)1023-1027
Number of pages5
JournalProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Publication statusPublished - 2014
Event15th Annual Conference of the International Speech Communication Association: Celebrating the Diversity of Spoken Languages, INTERSPEECH 2014 - Singapore, Singapore
Duration: 14 Sept 201418 Sept 2014

Bibliographical note

Publisher Copyright:
Copyright © 2014 ISCA.


  • Acoustic event detection
  • Assistive technology
  • Elderly healthcare
  • Speech recognition
  • Video surveillance


Dive into the research topics of 'Audio-visual signal processing in a multimodal assisted living environment'. Together they form a unique fingerprint.

Cite this