CLUE-AI: A Convolutional Three-Stream Anomaly Identification Framework for Robot Manipulation

Dogan Altan*, Sanem Sariel

*Bu çalışma için yazışmadan sorumlu yazar

Araştırma sonucu: ???type-name???Makalebilirkişi

1 Atıf (Scopus)


Despite the great promise of service robots in everyday tasks, many roboethics issues remain to be addressed before these robots can physically work in human environments. Robot safety is one of the essential concerns for roboethics which is not just a design-time issue. It is also crucial to devise the required onboard monitoring and control strategies to enable robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during their operations in the real world. The detection and identification of these anomalies is an essential first step toward fulfilling these requirements. Although several architectures have been proposed for anomaly detection; identification has not yet been thoroughly investigated. This task is challenging since indicators may appear long before anomalies are detected. In this paper, we propose a ConvoLUtional threE-stream Anomaly Identification (CLUE-AI) framework to address this problem. The framework fuses visual, auditory and proprioceptive data streams to identify everyday object manipulation anomalies. A stream of 2D images gathered through an RGB-D camera placed on the head of the robot is processed within a self-attention-enabled visual stage to capture visual anomaly indicators. The auditory modality provided by the microphone placed on the robot's lower torso is processed within a designed convolutional neural network (CNN) in the auditory stage. Last, the force applied by the gripper and the gripper state is processed within a CNN to obtain proprioceptive features. These outputs are then combined with a late fusion scheme. Our novel three-stream framework design is analyzed on everyday object manipulation tasks with a Baxter humanoid robot in a semi-structured setting. The results indicate that CLUE-AI achieves an f-score of 94%, outperforming the other baselines in classifying anomalies.

Orijinal dilİngilizce
Sayfa (başlangıç-bitiş)48347-48357
Sayfa sayısı11
DergiIEEE Access
Yayın durumuYayınlandı - 2023

Bibliyografik not

Publisher Copyright:
© 2013 IEEE.


This work was supported in part by the Istanbul Technical University Scientific Research Projects (ITU-BAP) under Grant 40240, and in part by the Scientific and Technological Research Council of Turkey (TUBITAK) under Grant 119E-436.

FinansörlerFinansör numarası
Türkiye Bilimsel ve Teknolojik Araştırma Kurumu119E-436
Istanbul Teknik Üniversitesi
Bilimsel Araştırma Projeleri Birimi, İstanbul Teknik Üniversitesi40240

    Parmak izi

    CLUE-AI: A Convolutional Three-Stream Anomaly Identification Framework for Robot Manipulation' araştırma başlıklarına git. Birlikte benzersiz bir parmak izi oluştururlar.

    Alıntı Yap