Challenges in Spoken Language Understanding Systems

In the world of technology, spoken language understanding (SLU) systems play a crucial role in how we interact with devices. When we talk to smart assistants or voice-controlled gadgets, they need to understand what we say before they can perform tasks. However, these systems can face challenges when they encounter speech data that is different from what they were trained on. This situation is often called out-of-distribution (OOD) generalisation.

When we say that data is OOD, we mean that it varies unexpectedly from what the system has learned. This can happen due to many reasons, like variations in accents, new words, or different speaking styles. While there has been growing interest in studying how systems can handle this type of data, there hasn't been enough focus on SLU tasks regarding OOD generalisation.

To help further research in this area, we have developed a modified version of a popular SLU dataset called SLURP. Our new dataset, which we call SLURP for OOD Generalisation (SLURPFOOD), includes specific ways to test how well models can handle OOD data.

#The Importance of Generalisation in Spoken Language Understanding

SLU systems are essential for devices that listen and respond to our commands. For these systems to work correctly in real-world situations, they must perform well even when the data they encounter is different from what they learned during training.

There are several kinds of generalisation capabilities that are important but often not reached by SLU systems:

• Length Generalisation: This ability allows the system to understand sentences that are longer or shorter than those it was trained on.

• Out-of-Vocabulary (OOV) Generalisation: This is necessary when the test data includes words that the system has never seen before.

• Compositional Generalisation (CG): This ability is required when the data presents familiar words in new ways. For instance, combining known phrases in different contexts can be challenging for SLU systems.

These types of generalisation are necessary for handling various speaking styles, accents, and settings.

Traditional SLU systems typically involve two parts: one that converts speech to text (automatic speech recognition or ASR) and another that interprets the text to understand the meaning (natural language understanding or NLU). Most studies on SLU generalisation focus on the text output instead of the original audio input. However, evaluating these systems based only on text can misrepresent their capabilities, as audio processing presents unique challenges.

#Our Approach to Testing Generalisation

To study how well SLU systems manage OOD data, we have created new data splits for SLURP. These splits allow us to test the models on three main aspects: OOV generalisation, CG, and mismatched acoustic environments.

Our dataset contains thousands of recordings with different types of annotations, such as transcripts and action labels. Each recording gives a context or situation, such as asking a question or giving a command. We designed our splits to assess how well systems can handle situations they were not trained on.

#OOV Splits

For the OOV splits, we selected a test set that includes new intents that were not present in the training data. This way, we can see how well the model understands commands it has never encountered before.

#Compositional Generalisation (CG) Splits

For the CG splits, we use a method to assess how well the model combines familiar elements. We focus on creating splits where the combination of words might be new, even if the individual words were previously seen.

#Microphone Mismatch Splits

We also account for the various environments in which audio recordings can occur. By creating splits based on recordings made with headsets versus those made without, we can assess how well the models adapt to changes in the audio environment.

#Experiments and Results

To evaluate the capabilities of SLU models on our new splits, we created baseline systems trained on the scenario classification task. We utilized a pre-existing model that has shown good performance on speech-related tasks.

For all of our experiments, we used a consistent setup, allowing us to focus on how well the models performed under different conditions. We trained our models and calculated their performance using a metric called the micro F1 score, which helps measure their accuracy.

In our findings, we noticed significant performance drops when the models were evaluated on OOD data. For instance, in the OOV split, the models performed much worse than on the non-OOV data, indicating a struggle with generalisation.

#Performance on Different Splits

• The model showed a drop in performance when handling OOV data, indicating challenges when faced with new commands.
• On the CG splits, the difference in performance was less severe, but still noticeable.

Additionally, we tested how models fared with audio samples that did not match the training environment. Here again, we saw a drop in performance, showing that models struggle to adapt to different acoustic conditions.

#Investigating the Reasons for Poor Generalisation

To better understand why these models faced challenges with OOD data, we explored which words were most important for their predictions. We used a technique to identify which words contributed significantly to the model's output.

Our analysis revealed that models often relied too heavily on less meaningful words, known as stopwords, such as "a" or "the." This reliance suggests that the models may not be effectively learning the important parts of the input data, which can lead to poor generalisation to new situations.

When comparing predictions made on OOD and traditional data, we noticed that successful predictions often used more relevant words in OOD contexts. This observation indicates that models may struggle when they encounter commands that have different word combinations than they were trained on.

#Improving Generalisation

In our efforts to enhance generalisation, we experimented with two techniques: TOPK and segmented processing.

#TOPK Approach

The TOPK method involves focusing only on the most significant losses within a training batch. By averaging the top losses rather than all losses, we aimed to encourage the model to prioritize more significant errors during training.

#Segmented Processing

For segmented processing, we took the audio data and divided it into smaller overlapping segments. This way, we aimed to gather more context and improve the final representation of the input.

Both approaches showed promise in improving generalisation in various splits, although they did not consistently yield better results across all scenarios.

#Conclusion

In this study, we highlighted the importance of testing SLU systems on diverse data types to understand their generalisation capabilities better. Through our new splits, we provided valuable insights into how well models can adapt to OOD situations.

Our results show significant room for improvement in SLU models when faced with unseen commands or different audio environments. By examining the contributing factors to performance, we identified weaknesses in how models learn and apply knowledge to new inputs.

As a future direction, we plan to build on these findings and develop new methods that can help SLU systems generalise more effectively to different contexts and types of data.