Publication Date
Fall 2023
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
Ching-Seh Wu
Second Advisor
Navrati Saxena
Third Advisor
Mrunmayi Deshpande
Keywords
Sign language translation, deep learning, transformers, multi cue networks, temporal pooling, linear competing units, stochastic weights, weight compression, bilingual evaluation understudy
Abstract
Sign language is a form of visual language that uses face expression and hand gestures to communicate thoughts and concepts. The term refers to multiple visual languages that share some common visual cues but differ in their grammar and syntax. Sign language translation (SLT) is a crucial step in closing the communication gap between hearing and hearingimpaired people. The study of SLT using machine learning has gotten a lot of interest during the last three years but despite progress, SLT research is still in its early phases. Most of the previous approaches first convert the signs to glosses and then convert the glosses into meaningful sentences. A gloss is a word connected to a sign or can also be called a label of a particular sign. These words don't convey what the sign means but only roughly capture the sign's intent. Though such approaches have good accuracy, obtaining gloss sequence ground truth is a tedious job and it also increases memory requirements during translation. This research project proposes an ensemble transformer architecture by combining SpatialTemporal Multi-Cue Networks (STMC) with stochastic transformers to avoid the explicit use of glosses and directly perform video-to-natural spoken language conversion while reducing memory consumption during inference. The results show a 3% increase in translation accuracy in terms of BLEU-4 score compared to previous state of the art research along with 5 times lesser training time and 70.7% reduction in memory consumption during inference.
Recommended Citation
Ghegade, Sumeet, "ENSEMBLE TRANSFORMER ARCHITECTURE FOR EFFICIENT AND REAL TIME SIGN LANGUAGE TRANSLATION" (2023). Master's Projects. 1324.
https://scholarworks.sjsu.edu/etd_projects/1324
