International Research Journal of Innovations in Engineering and Technology - IRJIET International Open Access, Monthly, Peer Reviewed, Reputed Journal ISSN (online): 2581-3048

Impact Factor (2025): 6.9

DOI Prefix: 10.47001/IRJIET

A Multimodal Approach for Real-Time Sinhala Sign Language Translation

Abstract

To facilitate communication between individuals with hearing impairments and those without, the use of language translation is necessary. This paper presents the development of a mobile-based solution to assist the communication between individuals having hearing impairments and those without. This study is based in Sri Lanka and the solution is designed to focus on the Sri Lankan Sign Language (SSL). There are three primary functionalities in this app: receiving input in English, converting it to a 3D avatar representing Sri Lankan Sign Language (SSL), and receiving input in Sinhala, converting it to an SSL picture, and receiving output in text form from Sri Lankan Sign Language. The Google Speech-to-Text API is used for translating English and Sinhala voices. The front end of the app was built in Unity and Blender for the Android platform, while the back end was built in Python using the Flask API, TensorFlow, and Kera. Artificial neural network (ANN) and Convolutional Neural Network (CNN) models tailored to the tasks of Natural Language Processing (NLP) and image processing are used during training.

Country : Sri Lanka

1 Seethoda Gamage2 Kavinda Lakshan3 Vinodya Wickramarathna4 Dr. Dasuni Nawinna5 Sathira Hettiarachchi

  1. Department of Computer Systems Engineering, Sri Lanka Institute of Information Technology, Sri Lanka
  2. Department of Computer Systems Engineering, Sri Lanka Institute of Information Technology, Sri Lanka
  3. Department of Computer Systems Engineering, Sri Lanka Institute of Information Technology, Sri Lanka
  4. Department of Computer Systems Engineering, Sri Lanka Institute of Information Technology, Sri Lanka
  5. Department of Information Technology, Sri Lanka Institute of Information Technology, Sri Lanka

IRJIET, Volume 7, Issue 10, October 2023 pp. 82-88

doi.org/10.47001/IRJIET/2023.710011

Full Paper
Download

References

  1. M. Rahaman, Md. P. Hossain, M. M. Rana, M. Rahman, and T. Akter, A Rule-Based System for Bangla Voice and Text to Bangla Sign Language Interpretation. 2020.
  2. V. Mueller, “American Sign Language (ASL),” 2013, pp. 133–136. doi 10.1007/978-1-4419-1698-3_1654.
  3. D. Patel and S. Naskar, “Image Classification Using Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN): A Review,” 2020, pp. 367–381. doi: 10.1007/978-981-15-1884-3_34.
  4. K. Saija, S. Sangeetha, and V. Shah, “WordNet Based Sign Language Machine Translation: from English Voice to ISL Gloss,” 2019 IEEE 16th India Council International Conference (INDICON), pp. 1–4, Dec. 2019, doi: 10.1109/INDICON47234.2019.9029074.
  5. A.Mehta, K. Solanki, and T. Rathod, “Development of an Automatic Translate Real-Time Voice to Sign Language Conversion for Deaf and Dumb People,” in 1st International Conference on Multidisciplinary Industry and Academic Research 2020, Institute of Industry and Academic Research Incorporated, Dec. 2020. Accessed: Jul. 05, 2023. [Online]. Available: https://www.neliti.com/publications/358242/
  6. R. Kumar Attar, V. Goyal, and L. Goyal, “State of the Art of Automation in Sign Language: A Systematic Review,” ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 22, no. 4, p. 94:1-94:80, Apr. 2023, doi: 10.1145/3564769.
  7. Z. Alsaadi, E. Alshamani, M. Alrehaili, A. A. D. Alrashdi, S. Albelwi, and A. O. Elfaki, “A Real Time Arabic Sign Language Alphabets (ArSLA) Recognition Model Using Deep Learning Architecture,” Computers, vol. 11, no. 5, Art. no. 5, May 2022, doi: 10.3390/computers11050078.
  8. A.Boukdir, M. Benaddy, A. Ellahyani, O. E. Meslouhi, and M. Kardouchi, “3D gesture segmentation for word-level Arabic sign language using large-scale RGB video sequences and autoencoder convolutional networks,” Signal, Image and Video Processing, vol. 16, pp. 1–8, Feb. 2022, doi: 10.1007/s11760-022-02167-6.
  9. K. Shenoy, T. Dastane, V. Rao, and D. Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition,” in 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Jul. 2018, pp. 1–9. doi: 10.1109/ICCCNT.2018.8493808.
  10. S. Katoch, V. Singh, and U. S. Tiwary, “Indian Sign Language recognition system using SURF with SVM and CNN,” Array, vol. 14, p. 100141, Jul. 2022, doi: 10.1016/j.array.2022.100141.
  11. R. Rajashekar, K. V. S. Hari, and L. Hanzo, “Antenna Selection in Spatial Modulation Systems,” Communications Letters, IEEE, vol. 17, pp. 521–524, Mar. 2013, doi: 10.1109/LCOMM.2013.012213.122650.
  12. R. M. Rishan, S. Jayalal, and T. Wijayasiriwardhane, Translation of Sri Lankan Sign Language to Sinhala Text: A Leap Motion Technology-based Approach. 2022, p. 223. doi: 10.1109/ICARC54489.2022.9754050.
  13. S. Das, Md. S. Imtiaz, N. H. Neom, N. Siddique, and H. Wang, “A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier,” Expert Systems with Applications, vol. 213, p. 118914, Mar. 2023, doi: 10.1016/j.eswa.2022.118914.
  14. S. K. Youme, T. A. Chowdhury, H. Ahamed, Md. S. Abid, L. Chowdhury, and N. Mohammed, “Generalization of Bangla Sign Language Recognition Using Angular Loss Functions,” IEEE Access, vol. 9, pp. 165351–165365, 2021, doi: 10.1109/ACCESS.2021.3134903.
  15. S. Majeed, H. HUSAIN, S. Samad, and T. Idbeaa, “Mel frequency cepstral coefficients (Mfcc) feature extraction enhancement in the application of speech recognition: A comparison study,” Journal of Theoretical and Applied Information Technology, vol. 79, pp. 38–56, Sep. 2015.
  16. K. Jain and V. Prajapati, “NLP/Deep Learning Techniques in Healthcare for Decision Making,” Primary Health Care, no. 3, p. 4, 2021.
  17. A.Núñez-Marcos, O. Perez-de-Viñaspre, and G. Labaka, “A survey on Sign Language machine translation,” Expert Systems with Applications, vol. 213, p. 118993, Mar. 2023, doi: 10.1016/j.eswa.2022.118993.
  18. Z. Liang, H. Li, and J. Chai, “Sign Language Translation: A Survey of Approaches and Techniques,” Electronics, vol. 12, no. 12, Art. no. 12, Jan. 2023, doi: 10.3390/electronics12122678.
  19. D. V. G and K. Goyal, “Indian Sign Language Recognition Using Mediapipe Holistic.” arXiv, Apr. 20, 2023. doi: 10.48550/arXiv.2304.10256.
  20. S. Hettiarachchi, “Machine Learning Approach for Real Time Translation of Sinhala Sign Language into Text,” 2020. Accessed: Aug. 03, 2023. [Online]. Available: https://www.semanticscholar.org/paper/Machine-Learning-Approach-for-Real-Time-Translation-Hettiarachchi/63e58eb09460d5028ebc42ca3e976a427cf13241
  21. F. M. J. M. Shamrat and S. Chakraborty, “Face Mask Detection using Convolutional Neural Network (CNN) to reduce the spread of Covid-19,” IEEE, Jan. 2021, Accessed: May 01, 2022. [Online]. Available: https://www.academia.edu/49065936/Face_Mask_Detection_using_Convolutional_Neural_Network_CNN_to_reduce_the_spread_of_Covid_19
  22. B.-J. Yoon, “Hidden Markov Models and their Applications in Biological Sequence Analysis,” Curr Genomics, vol. 10, no. 6, pp. 402–415, Sep. 2009, doi: 10.2174/138920209789177575.
  23. S. Duncan, “Gesture in Signing: A Case Study from Taiwan Sign Language,” Language and Linguistics, vol. 6, Jan. 2005.
  24. V. Sudha and D. Ganeshbabu, “A Convolutional Neural Network Classifier VGG-19 Architecture for Lesion Detection and Grading in Diabetic Retinopathy Based on Deep Learning,” Computers, Materials & Continua, vol. 66, pp. 827–842, Jan. 2020, doi: 10.32604/cmc.2020.012008.

Copyright @ 2026-2017 IRJIET. All Right Reserved.

Developed by Byzero Technologies