Sign Recognition System for an Assistive Robot Sign Tutor for Children

This paper presents a sign recognition system for a sign tutoring assistive humanoid robot. In this study, a specially designed 5-fingered robot platform with expressive face (Robovie R3) is used for interaction and communication with deaf or hard of hearing children using signs and visual cues. The robot is able to recognize and generate accurately a selected set of signs from Turkish sign language using various hand, arm and head gestures as relevant feedback. This paper focuses on the sign recognition system of the robot to recognize the human participant’s signing during the interaction. The system is based on two different approaches including a conventional method involving artificial neural network combined with hidden Markov model and a deep learning based method involving long short-term memory. The system is tested both on offline and real-time settings within an interaction game scenario with deaf or hard of hearing children. During the study, besides testing the sign recognition system, participants’ subjective evaluations and impressions were also collected and examined. The robot is perceived as likable and intelligent by the children, based on the questionnaires; and the proposed sign recognition system enables robust real-time interaction and communication of the assistive robot with children in sign language.