Improving Sign Language Detection Through Better Datasets

Sign language detection is about figuring out if someone is using sign language. This is becoming really important, especially for video calls and for gathering data to train systems that recognize or translate sign language. However, we believe that the current datasets used to test sign language detection might not give accurate results because they often mix signers in the training and testing phases.

#What is Sign Language Detection?

Sign language is a natural language used by the deaf community. It involves various body movements to convey messages, including hand shapes, facial expressions, and body posture. Traditional computer systems mainly focus on spoken languages, making sign language less visible in many platforms.

As more people start working remotely, software that allows video conferencing has become common. These programs often highlight the active speaker so that the audience can focus on them. Unfortunately, when a sign language user starts to sign, they can become hidden, making it challenging to communicate effectively. To bridge this gap, sign language detection tools need to be improved so that they can identify when someone is signing versus when they are not.

#Current State of Sign Language Detection

There is a growing amount of research in sign language detection, yet it remains a challenging area. Most systems designed for sign language detection follow a two-step process: first, they extract important features from the video, and then they classify the video as containing a sign language user or not.

For example, some studies use models that take video frames and extract features to determine if a person is signing. They have achieved fairly high accuracy rates, but upon closer inspection, it appears that these results may not be reliable. This is mainly because the same signers often appear in both the training and testing phases, causing overlap.

#The Problem with Signer Overlap

When we say "signer overlap," we mean that the same person might be included in both the training and testing datasets. This can make it seem like the system is working better than it really is. If a model has seen a particular signer during training, it is likely to do well when it encounters the same signer again during testing. This is a problem because it does not show how well the system can perform on new or different signers.

To illustrate this issue, we analyzed two major datasets used for sign language detection: the DGS Corpus and Signing in the Wild. Both of these datasets showed significant overlap in signers between the training and testing groups. We measured the impact of this overlap and found noticeable drops in accuracy when we compared results from datasets with and without overlapping signers.

#Proposed Solutions

To improve the way we evaluate sign language detection systems, we suggest creating new datasets that do not have signer overlap. By ensuring that a signer appears in only one dataset (either training, development, or testing), we can achieve a more realistic assessment of how well these systems are performing.

For the DGS Corpus, we developed a new way to split the data, ensuring that there were no signers in both the training and testing sets. This arrangement will help provide a clearer picture of the system’s capabilities. Similarly, we did the same for the Signing in the Wild dataset.

#Examining Datasets: DGS Corpus

The DGS Corpus is a collection of videos featuring German sign language, with over 1150 hours of recorded material. Only a portion of this data is annotated to show when signing occurs. Existing splits of the DGS Corpus suggested a mix of signers, which we identified and quantified.

By analyzing the original splits suggested by earlier research, we found that the same signers appeared in both training and testing sets. We broke down the original dataset to show how many signers overlapped between these phases. Noticing that 88 signers were common between the training and development sets was alarming. To demonstrate the effect of this overlap, we split the original test set into parts with and without overlap.

#Analyzing Signing in the Wild Dataset

The Signing in the Wild dataset consists of videos collected from YouTube, aiming to include a diverse range of sign languages and settings. This dataset also incorporates both signing and non-signing examples, such as speaking and other activities.

Similar to the DGS Corpus, the Signing in the Wild dataset showed that videos from the same signer might appear in multiple splits, which skews accuracy results. Initial experiments using the original splits indicated better performance due to the overlap. However, by creating a new split without overlap, we expected to find a drop in accuracy, reflecting a more honest performance evaluation.

#Clustering Signers for Better Data Management

One of the challenges in working with the DGS Corpus is that there is no labeling for the signers within the videos. To tackle this, we employed a method called face clustering, which groups similar faces based on extracted features. By using a clustering algorithm, we identified and grouped signers based on the videos where they appeared.

The results showed varying accuracy depending on the number of images used for clustering. We found that using more images led to better accuracy. However, we still faced challenges in identifying all signers perfectly.

#Conclusion

The results from analyzing the DGS Corpus and Signing in the Wild datasets indicate that signer overlap significantly impacts the effectiveness of sign language detection systems. To enhance the accuracy and ensure generalization, we proposed new datasets that eliminate this overlap.

Moving forward, reducing signer overlap is essential for establishing fair, accountable, and transparent systems for sign language detection. Additionally, the clustering method will help improve the management of sign language data while addressing privacy concerns.

Overall, by creating more reliable datasets and assessing sign language detection performance without overlap, we can work towards better tools for the deaf community and improve accessibility in various settings, especially in remote communication.