Pengenalan Alfabet Bahasa Isyarat Tangan Secara Real-Time dengan Menggunakan Metode Convolutional Neural Network dan Recurrent Neural Network

Abstract

Sign language is one of the communication tools commonly used by people with disabilities. The alphabet sign language is a basic tool used by teachers to teach people with hearing impairment and speech impairment to recognize basic alphabet letters. However, many people find it difficult to communicate with these groups because of a lack of community insight into hand sign language. Research on sign language has experienced much progress in processing static images but is still experiencing problems due to difficulties in processing dynamic images / video given that most of the sign language is represented by body, hand, and face movements.

This study uses Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) methods with video input. The CNN method will be used as a feature extraction in the spatial feature while the RNN is tasked to tolerate between frames extracted by CNN on the temporal feature.

The final result to be displayed is in the form of text alphabet which is the result of the recognition of the sign language alphabet. Based on the test carried out, obtained an average accuracy value of  60.58% for all letters while real-time testing has failed because the technology used cannot sustain the architecture created.