Summary:
Sign Language constitutes the primary means of communication for the deaf and hard-
of-hearing individuals. Sign Language Representation is a complex task, which involves
human labor-intensive processes. To address this challenge, we propose an automated
method that maps skeleton keypoints to avatar motions by leveraging advanced deep
learning approaches. This mapping can be achieved by extracting accurate 3d body joints
coordinates from monocular videos using state-of-the-art human pose estimation
algorithms. In our study, we investigate certain approaches which detect the 2D body
joints in videos and subsequently convert them into 3D space, evaluated on a small
synthetic dataset of five videos, featuring the Paula avatar. Our work focuses on arm
motions, emphasizing on keypoints related to shoulders, elbows, and wrists,
acknowledging the significance of their movements in sign language understanding. Due
to the training of evaluated methods on generic dataset rather than those specific to sign
language, we had to make certain adjustments to ensure the accordance of skeleton
keypoints. We provide a comprehensive analysis of the benefits and drawbacks of each
method and report special patterns of performance on different axes. Notably, the
approach, which uses the BlazePose of Mediapipe as the 2D detector and the
VideoPose3D for 3D reconstruction, outperforms its competitors, achieving an average
Mean Per Joint Position Error (MPJPE) of 72.2 mm.
Keywords:
human pose estimation, 3D reconstruction, sign language representation, avatar motion
