Advancements in Speech Emotion Recognition with Pre-trained Models

Understanding emotions is something humans do naturally, but it can be very hard for machines. This is important because machines are used to predict emotions in many situations. As technology becomes more common in everyday life, from smartphones to smartwatches, machines now interact with humans continuously. Therefore, accurately detecting human emotions is essential for effective communication between people and machines.

Emotions can be identified in numerous ways, such as facial expressions, body language, physiological signals, and speech. Speech Emotion Recognition (SER) is the process of identifying emotions through spoken words. This method has gained popularity due to its various applications in fields like psychology and healthcare. For example, in mental health care, SER can help psychologists better understand the emotional state of their patients, which can lead to more effective treatment. SER is also useful in customer service, where understanding emotions can improve interactions between service providers and customers.

Detecting emotions through speech involves analyzing different aspects of the speech signal, including pitch, intensity, and duration. Various approaches have been developed for SER, ranging from simple fuzzy methods to more complex Hidden Markov Model (HMM)-based methods. Traditional machine learning methods, like random forests and support vector machines, have been used but often rely on features that need to be extracted by experts. To avoid this, researchers have turned to deep learning techniques, which can automate the feature extraction process. Some of these deep learning models include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory Networks (LSTMs).

Recently, transformer-based architectures have also been used for SER. Models like Wav2Vec 2.0 and HuBERT, which were originally designed for automatic speech recognition, have been adapted for SER. Other advanced models, like the MLP mixer, have also been explored in this domain.

#Challenges in SER

SER faces some challenges. One main issue is the differences in how people express emotions and variations in voices. Different backgrounds and experiences lead to unique emotional expressions, making it more difficult for machines to accurately recognize emotions.

To address these challenges, researchers train and evaluate different models to improve SER effectiveness. Various Audio Pretrained Models (PTMs) are available that can help in recognizing emotions in speech. These models have been trained on extensive speech and audio datasets and are designed to capture intricate details of human emotions in spoken language.

#Previous Research

Numerous studies have examined PTMs in the context of SER. Initially, many studies used Hidden Markov Models (HMMs) and classical machine learning algorithms with manually extracted features. As deep learning gained traction, researchers started employing CNNs, especially after AlexNet's success in image recognition competitions. Various innovative approaches have been proposed, including architectures that combine CNNs with LSTMs and attention mechanisms.

Over time, transformers have gained prominence in SER research. Models that stack multiple transformer layers have shown great potential in capturing emotional nuances in speech. Additionally, models trained on vast amounts of speech data, like Wav2Vec 2.0 and HuBERT, have been fine-tuned for SER tasks, leading to improvements in performance.

While there are many studies on PTMs for SER, there is still a lack of comprehensive comparisons between the Embeddings derived from various models and architectures. Understanding the best embeddings for SER is crucial for optimizing performance in real-world applications.

#Speech Emotion Data

In order to effectively train and evaluate SER systems, various speech emotion datasets are used. These datasets contain audio clips labeled with different emotions, allowing researchers to build and test their models. Here are some commonly used datasets:

1. Crowd-Sourced Emotional Multimodal Actors Dataset (CREMA-D): This dataset includes audio clips from male and female speakers expressing different emotions. Each clip is linked to multiple emotions and emotion intensities, providing a rich source of data for SER studies.

2. Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): This corpus includes both speech and song data, offering varied emotional expressions from trained actors.

3. Toronto Emotional Speech Set (TESS): Comprised of recordings from two female actors, TESS includes various emotional expressions across a range of words.

4. Surrey Audio-Visual Expressed Emotion (SAVEE): This dataset features recordings from male actors, providing phonetically balanced phrases representing different emotions.

5. German Emotional Speech Database (Emo-DB): This corpus includes recordings from male and female speakers, contributing to multilingual studies in SER.

These datasets are crucial for training SER models as they provide the necessary labels and examples of emotional speech.

#Pre-trained Model Embeddings

Embeddings are representations derived from models that capture important features from the input data. They are essential for training Classifiers that make predictions about emotions. Several PTMs generate embeddings that can be used in SER tasks.

For effective evaluation of these embeddings, researchers consider various benchmarks. One benchmark is the Speech processing Universal PERformance Benchmark (SUPERB), which measures various speech-related tasks. Models ranking high on SUPERB are often selected for SER studies.

Another benchmark is the Holistic Evaluation of Audio Representations (HEAR), which assesses different audio models on their performance across various tasks. Models like Wav2Vec 2.0, data2vec, and UniSpeech-SAT are popular choices due to their strong performance in these benchmarks.

Empirical evidence suggests that embeddings from models trained for speaker recognition may also enhance SER performance. Knowledge acquired from recognizing different speakers can support machines in recognizing emotional nuances in speech.

#Classifiers for SER

When it comes to classifying the emotions detected in speech, various approaches can be employed. Researchers commonly use classical machine learning algorithms, such as XGBoost and Random Forest. Simple neural networks, known as Fully Connected Networks (FCN), are also utilized for this purpose.

In training these classifiers, data is split into training, validation, and testing sets. Hyperparameters are optimized through validation to ensure the models perform well. The classifiers are trained on different embeddings from various PTMs to evaluate their effectiveness in recognizing emotions.

#Experimental Results

The performance of different PTM embeddings can be compared based on metrics such as accuracy and F1-score. These metrics help understand how well each embedding performs on various datasets.

In studies, embeddings from speaker recognition PTMs often show superior performance over other embeddings. This suggests that the ability to recognize unique speech features contributes positively to understanding emotions. Among other embeddings, some models, like UniSpeech-SAT and WavLM, stand out for their performance in SER tasks.

Additionally, visualizations like t-SNE plots help illustrate how well the embeddings cluster by emotion. Clusters for different emotions can reveal how effectively models distinguish between them, providing insights into model performance.

#Conclusion

Pre-trained models have made significant advancements in the fields of speech and audio processing. These models, trained on large datasets, provide valuable embeddings that can enhance SER systems. However, previous studies have often focused on specific models without a thorough comparison across various architectures and techniques.

This study aimed to fill that gap by comparing embeddings from multiple PTMs, using different classifiers across various speech emotion datasets. The findings emphasize that embeddings from models trained for speaker recognition consistently outperform those from other types of PTMs.

In the future, there is potential for further exploration by incorporating more diverse models and databases, broadening the scope of SER research. As machine learning technologies evolve, the landscape of SER will continue to develop, leading to more effective human-machine interactions. The results from this study can guide future research in selecting the most suitable embeddings for speech emotion detection tasks.