FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection

Arda Inceoglu, Eren Erdal Aksoy, Abdullah Cihan Ak, Sanem Sariel

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

9 Citations (Scopus)

Abstract

We need robots more aware of the unintended outcomes of their actions for ensuring safety. This can be achieved by an onboard failure detection system to monitor and detect such cases. Onboard failure detection is challenging with a limited set of onboard sensor setup due to the limitations of sensing capabilities of each sensor. To alleviate these challenges, we propose FINO-Net, a novel multimodal sensor fusion based deep neural network to detect and identify manipulation failures. We also introduce FAILURE, a multimodal dataset, containing 229 real-world manipulation data recorded with a Baxter robot. Our network combines RGB, depth and audio readings to effectively detect failures. Results indicate that fusing RGB with depth and audio modalities significantly improves the performance. FINO-Net achieves %98.60 detection accuracy on our novel dataset. Code and data are publicly available at https://github.com/ardai/fino-net.

Original languageEnglish
Title of host publicationIEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages6841-6847
Number of pages7
ISBN (Electronic)9781665417143
DOIs
Publication statusPublished - 2021
Event2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021 - Prague, Czech Republic
Duration: 27 Sept 20211 Oct 2021

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021
Country/TerritoryCzech Republic
CityPrague
Period27/09/211/10/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Fingerprint

Dive into the research topics of 'FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection'. Together they form a unique fingerprint.

Cite this