Traducción de lenguaje de signos a texto mediante python con redes neuronales LSTM

Luis Duván Torrado Mora; César Alberto Collazos; Dewar Rico Bautista

doi:10.24054/rcta.v2i46.4105

Authors

Luis Duván Torrado Mora Universidad Francisco de Paula Santander https://orcid.org/0000-0002-7534-4954
César Alberto Collazos Universidad del Cauca https://orcid.org/0000-0002-7099-8131
Dewar Rico Bautista Universidad Francisco de Paula Santander https://orcid.org/0000-0002-1808-3874

DOI:

https://doi.org/10.24054/rcta.v2i46.4105

Keywords:

LSTM, mediapipe, sign Language, gesture sign recognition

Abstract

There is a difficulty that deaf people with speech disabilities face in communicating effectively with those who do not know sign language. Despite the existence of methods such as writing and lip-reading, these have limitations and are not always effective. The proposed solution includes developing a real-time sign language recognition system using convolutional neural networks and the MediaPipe platform. It detects and classifies the positions of hand points to identify letters. Gestures made in front of the camera are translated into letters that are stored to form paragraphs in a text box. The type of research is quantitative and experimental. Ultimately, the importance of sign language recognition and teaching is highlighted, especially in countries such as Colombia, where it has received significant recognition.

Downloads

Download data is not yet available.

References

B. Villa, V. Valencia, and J. Berrio, “Digital image processing applied on static sign language recognition system/Diseño de un Sistema de Reconocimiento de Gestos No Móviles mediante el Procesamiento Digital de Imágenes,” Prospectiva, vol. 16, no. 2, pp. 41–48, 2018, doi: 10.15665/rp.v16i2.1488.

I. De Souza, P. Mattos, C. Pina, and D. Fortes, “ADHD: The impact when not diagnosed,” J Bras Psiquiatr, vol. 57, no. 2, pp. 139–141, 2008, doi: 10.1590/s0047-20852008000200010.

V. J. Schmalz, “Real-time Italian Sign Language Recognition with Deep Learning,” in CEUR Workshop Proceedings, 2022, pp. 45–57.

D. S. Breland, A. Dayal, A. Jha, P. K. Yalavarthy, O. J. Pandey, and L. R. Cenkeramaddi, “Robust Hand Gestures Recognition Using a Deep CNN and Thermal Images,” IEEE Sens J, vol. 21, no. 23, pp. 26602–26614, Dec. 2021, doi: 10.1109/JSEN.2021.3119977.

D. S. Breland, S. B. Skriubakken, A. Dayal, A. Jha, P. K. Yalavarthy, and L. R. Cenkeramaddi, “Deep Learning-Based Sign Language Digits Recognition from Thermal Images with Edge Computing System,” IEEE Sens J, vol. 21, no. 9, pp. 10445–10453, May 2021, doi: 10.1109/JSEN.2021.3061608.

I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine learning methods for sign language recognition: A critical review and analysis,” Nov. 01, 2021, Elsevier. doi: 10.1016/j.iswa.2021.200056.

R. Rastgoo, K. Kiani, and S. Escalera, “Hand sign language recognition using multi-view hand skeleton,” Expert Syst Appl, vol. 150, p. 113336, Jul. 2020, doi: 10.1016/j.eswa.2020.113336.

H. Liu, Z. Zhang, J. Qian, W. Wang, and K. Kang, “Hand gesture recognition based on deep neural network and sEMG signal,” in IEEE International Conference on Robotics and Biomimetics, ROBIO 2019, Institute of Electrical and Electronics Engineers Inc., Dec. 2019, pp. 3001–3006. doi: 10.1109/ROBIO49542.2019.8961445.

J. A. Zea Gutiérrez, M. J. Suárez Barón, and J. S. González Sanabria, “Aprendizaje por refuerzo como soporte a la predicción de la precipitación mensual. Caso de estudio: Departamento de Boyacá - Colombia,” TecnoLógicas, vol. 27, no. 60, p. e3017, Jun. 2024, doi: 10.22430/22565337.3017.

L. Antonelli, M. Lezoche, and J. Delle Ville, “Knowledge Extraction from the Language Extended Lexicon Glossary Using Natural Language Processing,” TecnoLógicas, vol. 27, no. 59, p. e2917, Apr. 2024, doi: 10.22430/22565337.2917.

L. D. Torrado-Mora and D. Rico-Bautista, “Systematic Mapping: Translator Language from Sign Language to Colombian Formal Language,” in Proceedings - 3rd International Conference on Information Systems and Software Technologies, ICI2ST 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 44–48. doi: 10.1109/ICI2ST57350.2022.00014.

S. Sharma and S. Singh, “Vision-based hand gesture recognition using deep learning for the interpretation of sign language,” Expert Syst Appl, vol. 182, p. 115657, Nov. 2021, doi: 10.1016/j.eswa.2021.115657.

L. D. Torrado-Mora, C. A. Collazos, and D. Rico-Bautista, “Sign Language to Colombian Formal Language Trans-lation Software,” in Human-Computer Interaction. HCI-COLLAB 2024. Communications in Computer and Information Science, V. Agredo-Delgado, P. H. Ruiz, and C. A. Meneses, Eds., SPRINGER, 2024, pp. 1–14.

L. A. Tovar, “Elaboración de tesis. Estructura y metodología,” Trillas, p. 384, 2017.

M. Gómez, Introducción a la metodología de la investigación científica, 1st ed. Argentina, 2015.

G. Briones, Metodología de la investigación cuantitativa en las ciencias sociales, vol. 1, no. 958-9329-09–8. 2002. doi: 10.1038/2191218a0.

E. Del-Canto and A. Silva, “Metodología Cuantitativa: Abordaje desde la complementariedad en ciencias sociales,” Rev Cienc Soc, vol. 0, no. 141, p. 45, 2013, doi: 10.15517/rcs.v0i141.12479.

A. Radmehr, M. Asgari, and M. T. Masouleh, “Experimental Study on the Imitation of the Human Head-and-Eye Pose Using the 3-DOF Agile Eye Parallel Robot with ROS and Mediapipe Framework,” in 2021 9th RSI International Conference on Robotics and Mechatronics (ICRoM), IEEE, Nov. 2021, pp. 472–478. doi: 10.1109/ICRoM54204.2021.9663445.

S. Adhikary, A. K. Talukdar, and K. Kumar Sarma, “A Vision-based System for Recognition of Words used in Indian Sign Language Using MediaPipe,” in Proceedings of the IEEE International Conference Image Information Processing, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 390–394. doi: 10.1109/ICIIP53038.2021.9702551.

T. Granollers i Saltiveri, “MPIu+a. Una metodología que integra la Ingeniería del Software, la Interacción Persona-Ordenador y la Accesibilidad en el contexto de equipos de desarrollo multidisciplinares,” 2004.

J. Lorés and T. Granollers, “Ingeniería de la Usabilidad y de la Accesibilidad aplicada al diseño y desarrollo de sitios web,” no. May, pp. 3–7, 2004.

T. Granollers, “Usability Evaluation with Heuristics . New Proposal from Integrating Two Trusted Sources 2 Combining Common Heuristic Sets,” pp. 1–16, 2018.

T. Granollers, “Diseño Centrado en el Usuario (DCU). El modelo MPlu+a,” p. 71, 2013.

S. Inform and L. Lleida, “MPIu + a . Una metodología que integra la ingeniería del software, la interacción persona-ordenador y la accesibilidad en el contexto de equipos de desarrollo multidisciplinares,” 2004.

S. Ikram and N. Dhanda, “American Sign Language Recognition using Convolutional Neural Network,” in 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies, GUCON 2021, Institute of Electrical and Electronics Engineers Inc., Sep. 2021. doi: 10.1109/GUCON50781.2021.9573782.

A. Radmehr, M. Asgari, and M. T. Masouleh, “Experimental Study on the Imitation of the Human Head-And-Eye Pose Using the 3-DOF Agile Eye Parallel Robot with ROS and Mediapipe Framework,” in 9th RSI International Conference on Robotics and Mechatronics, ICRoM 2021, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 472–478. doi: 10.1109/ICRoM54204.2021.9663445.

V. Chunduru, M. Roy, N. S. Dasari Romit, and R. G. Chittawadigi, “Hand Tracking in 3D Space using MediaPipe and PnP Method for Intuitive Control of Virtual Globe,” in IEEE Region 10 Humanitarian Technology Conference, R10-HTC, Institute of Electrical and Electronics Engineers Inc., 2021. doi: 10.1109/R10-HTC53172.2021.9641587.

N. F. Thejowahyono, M. V. Setiawan, S. B. Handoyo, and A. H. Rangkuti, “Hand Gesture Recognition as Signal for Help using Deep Neural Network,” International Journal of Emerging Technology and Advanced Engineering, vol. 12, no. 2, pp. 37–47, Feb. 2022, doi: 10.46338/ijetae0222_05.

S. Njazi and S. Ng, “Veritas: A Sign Language-To-Text Translator Using Machine Learning and Computer Vision,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Nov. 2021, pp. 55–60. doi: 10.1145/3507623.3507633.

M. L. Wehmeyer et al., “The intellectual disability construct and its relation to human functioning,” Intellect Dev Disabil, vol. 46, no. 4, pp. 311–318, Jan. 2008, doi: 10.1352/1934-9556(2008)46[311:TIDCAI]2.0.CO;2.

S. Srivastava, A. Gangwar, R. Mishra, and S. Singh, “Sign Language Recognition System Using TensorFlow Object Detection API,” in Communications in Computer and Information Science, Springer Science and Business Media Deutschland GmbH, Jan. 2022, pp. 634–646. doi: 10.1007/978-3-030-96040-7_48.

S. Gulati, A. K. Rastogi, M. Virmani, R. Jana, R. Pradhan, and C. Gupta, “Paint/Writing Application through WebCam using MediaPipe and OpenCV,” in Proceedings of 2nd International Conference on Innovative Practices in Technology and Management, ICIPTM 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 287–291. doi: 10.1109/ICIPTM54933.2022.9753939.

T. Granollers, “Usability evaluation with heuristics. New proposal from integrating two trusted sources,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer Verlag, 2018, pp. 396–405. doi: 10.1007/978-3-319-91797-9_28.

J. Lorés Vidal and T. Granollers, “La Ingeniería de la usabilidad y de la accesibilidad aplicada al diseño y desarrollo de sitios web,” 2004.