Bilgi Damıtma ile Robot-Nesne Etkileşim Hatalarını Tahminleme

An autonomous service robot should be able to safely interact with its environment. However, failures can occur during manipulation execution due to various uncertainties such as perception errors, manipulation inaccuracies, or unforeseen external events. While existing research has primarily focused on the detection and classification of robot failures, this work focuses on anticipation of such failures. The premise is that if a failure can be anticipated early enough, prevention actions can be taken. To this end, we introduce a novel knowledge distillation-based anticipation framework. Our framework leverages the power of video transformers and incorporates a multimodal sensor fusion network capable of processing RGB, depth, and optical flow data. We evaluate the success of our approach using a real-world robot manipulation dataset named FAILURE. Experimental results demonstrate that our proposed framework achieves an 82.12% F1 score, showcasing its efficacy in anticipating robot execution failures up to 1 second in advance.