Publication Date
Fall 2024
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
Saptarshi Sengupta
Second Advisor
Robert Chun
Third Advisor
Sayma Akther
Keywords
Indian Sign Language, ISL Gloss Translation, Rule-Based Model, Natural Language Processing, Syntax Transformation, Subject- Object-Verb Structure, Computational Linguistics
Abstract
In India, one of the most significant tools to communicate with the Deaf and Hard of Hearing (DHH) communities is Indian Sign Language(ISL). The issue of a scarcity of computational resources for ISL is even more pronounced when it comes to the translation of written or spoken English into ISL. This paper proposes a rule-based model for translation where the spoken english sentences are translated into ISL by focusing on the specific syntactic and grammatical differences between these two languages. ISL uses a simplified syntax unlike the nuanced sentence structures in English — we omit most of the auxiliary verbs and write in a subject-object-verb format. Our model is based on linguistic transformation rules that identify parts of speech and rearrange the English sentence to generate a grammatically correct ISL gloss with respect to these linguistic transformations. Since ISL is relatively new and lack a comprehensive dictionary of words, handling missing words has impacted the translation result. Here, we are aiming to use contextual embedding and semantic similarity measures to retrieve the best match of missing words from the dictionary. This approach will provide the users with a translation that is contextually accurate. Initial results show that the model is able to produce fluent ISL gloss, which will provide an easy-to-use tool and be a step toward investigating sign language translation.
Recommended Citation
Jaison, Dania, "A Rule-Based Hybrid Translation Model for Context-Aware Speech to Indian Sign Language Tasks" (2024). Master's Projects. 1426.
https://scholarworks.sjsu.edu/etd_projects/1426
