ACCESSIBLE COMMUNICATION WITH THE HELP OF AI: AN APP FOR TRANSLATING SIGN LANGUAGE

Most people take being able to hear and speak for granted, as an integral part of daily life. Yet for 70 million people worldwide^¹, the reality is different. Using sign language enables them to maintain social relationships and participate actively in public discourse. Nevertheless, deaf people often face challenges in their everyday lives: Very few people ever learn to use sign language. Furthermore, 80 per cent of deaf people live in developing countries, where access to education and technological aids is usually severely limited.

But what if sign language were just as easy to understand as spoken language? The use of technology and artificial intelligence (AI) can make this possible. We demonstrate how AI can improve sign language recognition (Continuous Sign Language Recognition, or CSLR for short) on mobile devices, both with and without an internet connection, and are developing an app to enable people with hearing impairments to participate fully in life on an equal footing.

1. Research relevance: Accessible communication has so far been limited

2. Three steps to a mobile solution

2.1 Analysing data sets

2.2 Selecting and integrating models

2.3 Developing a high-performance app

3. Outlook

1. Accessible communication has so far been limited

A solution for accessible communication is of great importance: in everyday life, there is often a lack of interpreters or technical aids that enable smooth communication with and for deaf people. In many public institutions and services, the requirements for accessible communication have not yet been adequately implemented. The challenges are also significant in the education sector. Access to high-quality education is often restricted because there are not enough teaching materials available in sign language. Furthermore, many teachers and lecturers are not sufficiently trained to work effectively with deaf or hard-of-hearing pupils and students.

2. A three-step guide to a mobile solution

A solution for people with hearing impairments is only useful if it can be used in everyday life, regardless of location. The aim was therefore to develop a smartphone app. The research was guided by the question of how machine learning models can be effectively utilised and optimised for sign language recognition on mobile devices.

We approached this question in several stages:

1. Analysing datasets: Firstly, the existing dataset had to be analysed and defined.
2. Selecting and integrating models: Suitable AI models were then evaluated and converted so that they could be seamlessly integrated into iOS applications
3. Developing a high-performance app: It was essential that the application could be used independently of an internet connection. Furthermore, various optimisation techniques significantly improved the efficiency and speed of the models on mobile devices. 

2.1 Analysing data sets

The project focused primarily on integrating existing, pre-trained AI models to create a mobile app. For this reason, we concentrated on the datasets used to train the models. As a first step, publicly available datasets were analysed and consolidated. Extensive sign language datasets, such as RWTH-PHOENIX_Weather 2014 T 2, were used for training the AI models and evaluating their performance . The data is currently still limited to specific scenarios, such as weather, and certain perspectives. In order to offer a comprehensive solution in future, it would be necessary to create more extensive datasets covering a wide range of scenarios, different perspectives and distances, in order to expand the models’ capabilities. For example: To train models as effectively as possible, close-up and long-distance shots, different angles and various signers are required, amongst other things. The sample data does not yet exhibit this level of variability. Nevertheless, the selected data and models have demonstrated that machine learning models can be used effectively for sign language recognition.

The figure shows examples from the RWTH-PHOENIX-Weather 2014 dataset with heatmaps. These highlight the most important areas learnt by the model by Hu et al.³  

Figure 1: Visualisation of the correlation maps used in CorrNet

2.2 Selecting and integrating models

In the next step, promising AI models were identified. These were converted to the ONNX format to enable their seamless integration into iOS applications. ONNX is a specialised format for representing and standardising deep learning models. This enables cross-platform portability, e.g. from server to app (in this case, from Python to iOS).

The models used, including CorrNet, are specialised in processing videos as a sequence of images and, using modern computer vision and time-series techniques, predicting the translation into natural language. CorrNet reads the video data and uses complex algorithms to interpret the movements and gestures in sign language and convert them into text.  

Figure 2: Architecture of CorrNet, showing the integration of multiple CNN layers and the correlation module, which is crucial for processing continuous sign language.

2.3 Development of a high-performance app

During the development of the application, various technical and practical challenges were addressed. When integrating the models onto the iOS target system, hardware-specific hurdles, amongst other things, arose. The greatest difficulty lay in the fact that the image processing steps in iOS proved to be significantly more complex than in Python, the industry standard for machine learning.  

The subsequent optimisation of the image processing introduced a further layer of complexity. The aim was to find the best method for reducing processing time without compromising accuracy – a process that required a great deal of patience and a willingness to experiment.  

Another problem was memory bottlenecks with large image sequences, particularly at higher frame rates. This required careful adjustments and numerous tests to achieve the optimal balance between image quality and memory usage.  

A fast, internet-independent solution

The results show that the optimised models achieve an accuracy on mobile devices that is comparable to that of the server-based variants. The app’s local processing offers significant speed improvements and is independent of the internet connection, which considerably enhances usability. Compared with the server-based solution, local processing is faster and offers more robust performance overall. These advances help to break down communication barriers and improve the quality of life for deaf people.  

Outlook

The project demonstrates that the use of artificial intelligence for this purpose could significantly improve the social inclusion and autonomy of deaf people. The idea of simply pulling out a smartphone and understanding a conversation in sign language would be a major step towards a more inclusive society.  

The research described here, which was carried out as part of a student research project at Landshut University of Applied Sciences and supported by iteratec, is just the beginning. To further improve sign language recognition, the datasets must be expanded and diversified by collecting data from different perspectives and contexts. There is also a need to further optimise the models to increase their efficiency and accuracy. The path taken to make this technology accessible to all and break down communication barriers is promising and offers great potential for further development.

Do you have any questions or need assistance?

You can find out more about the possibilities of artificial intelligence for your business on our website.