Unit:
Department of Informatics and TelecommunicationsΠληροφορική
Supervisors info:
Παναγιώτης Σταματόπουλος ,Επίκουρος Καθηγητής, Τμήμα πληροφορικής και τηλεπικοινωνιών, Εθνικόν και Καποδιστριακόν Πανεπιστήμιον Αθηνών
Original Title:
Dynamic sign language recognition using deep learning
Translated title:
Dynamic sign language recognition using deep learning
Summary:
Improving communication for hearing-impaired communities proves to be an extremely challenging task for scientists and researchers nowadays. Deaf people have to use an alternative to verbal speech communication such as sign language in order to interact with the world around them. In addition, the existence of over 200 sign languages around the world makes communication for people with hearing loss much harder. Thus, the need to develop an automatic dynamic sign language translator through AI becomes a necessity.
Vision-based sign language recognition aims to improve the accessibility and inclusivity of those who communicate through hand gestures. However, the recognition of complicated gestures that combine facial expression, hand gestures and body movements, in addition of an environment with background noises can be proved very challenging.
In this thesis we were able to experiment with multiple deep learning methods on the public DSL10_Dataset. Specifically, we applied the pretrained model VGG_16 for feature extraction based on pattern recognition and the MediaPipe framework for feature extraction based on the key-points of hands and pose. Finally, we evaluate both approaches using the proposed LSTM and GRU models for classification.
Our results show that the combination of the pretrained model VGG_16 for feature extraction and the proposed LSTM for classification achieved a recognition accuracy of 96.44% in comparison with others.
Main subject category:
Technology - Computer science
Keywords:
Artificial Intelligence, American sign language (ASL), VGG-16, MediaPipe, Long short-term memory (LSTM), Gated recurrent unit (GRU)
