Supervisors info:
Μανωλάκος Ηλίας, Καθηγητής, Πληροφορικής και Τηλεπικοινωνιών, Εθνικό και Καποδιστριακό Πανεπιστήμιο Αθηνών
Summary:
Deep Learning (DL), and especially Convolutional Neural Networks (CNNs), have been widely used to solve a large variety of problems in computer vision, including Sign Language Recognition (SLR). There have been many efforts towards designing systems using cameras that can translate signer gestures to text or even speech. However, these systems are very sensitive to factors such as light intensity, background color and motion occlusion etc.
In this thesis, we present the design of a simple endtoend embedded system that translates continuous American Sign Language (ASL) into text based on inputs received from an instrumented lowcost sensor glove that we have created using flex sensors and an IMU device. To prove the concept, we first generated a limited dataset of 20 random ASL sentences using a 20word vocabulary where we manually prelabel the time series data into 21 classes and simultaneously separate gesture and nongesture (transition class) movement periods, by using an external button. Subsequently, a sliding window technique was used to extract overlapping labeled samples (time windows) for continuous SLR. After standardization, the data samples are fed to a simple 3layer 1D CNN (conv1d conv1d fully connected) for classification.
Convolutional layers are useful for automated feature extraction and fully connected for classification. Our CNN achieves 93.40% accuracy on the test set (unseen data). For all practical purposes, the accuracy is actually 100% as vocabulary gestures are not confused for each other, and errors occur only in the transition from a gesture to a nongesture transition movement window and vice versa. The CNN was trained and its hyperparameters tuned using the ATOM pythonbased framework. Its accuracy was compared and found to be slightly higher than that of other popular machine learning methods, such as the Random Forests, Support Vector Machines, and Extreme Gradient Boosted Trees (XGBoost).
Finally, we have developed an allsoftware implementation of the designed CNN (inference part) for the ARM Cortex A9 processor on the Zybo development board. Using the Xilinx SDK and Eigen library, we managed to design a realtime embedded system that can achieve an operating frequency much higher than the sampling frequency. Optimization, training, and testing of the CNN were performed on a PC using ATOM and the Keras library with a TensorflowGPU backend.
Keywords:
SLR, sensor glove, ARM, CNN, Machine Learning
