Advancing Emotion Recognition with EEG Signals

Emotion recognition using brain signals, specifically EEG (electroencephalogram) signals, is an important area of study. It has many potential uses, such as in brain-computer interfaces, diagnosing emotional disorders, understanding Emotions in patients, monitoring drivers’ feelings, estimating mental workload, and in studies of the mind. Emotions can influence our actions and decisions every day. They come from a mix of thought and sensory information and can respond to things happening both inside and outside of us.

Physiological signals like ECG (electrocardiogram), EMG (electromyography), and EEG reflect the body's response to emotions. Among these, EEG signals stand out because they can provide quick and detailed information about emotional changes, making them highly suitable for emotion detection. Research has shown a clear link between EEG signals and emotions, leading to the idea that techniques based on EEG can work better and be more objective. Nevertheless, combining different types of information from EEG signals to improve emotion recognition remains a challenge.

#Background

Convolutional Neural Networks (CNNs) are known for their ability to extract features from data effectively. Many researchers have begun using CNNs for recognizing emotions in EEG signals. Some examples include models that focus on feature extraction and classification of emotional characteristics from EEG signals. However, these models often struggle to capture both detailed (local) and broad (global) features of the data.

With recent advancements in Transformer models, researchers have successfully applied them in many areas. Some methods combine different types of networks to enhance feature extraction, especially in tasks like emotion recognition. However, they often miss capturing local features. Hence, combining the advantages of CNNs and Transformers has become a goal for researchers in the field.

#Proposed Approach

To tackle these challenges, a new method called CIT-EmotionNet was developed. This method takes EEG signals, processes them into segments, and extracts features from each segment. The first step involves taking the raw EEG signals and breaking them into small parts of a few seconds. For each part, the features related to specific frequency bands are extracted. These features are then organized based on the brain's electrode positions.

CIT-EmotionNet has two main parts: the CNN branch and the Transformer branch. The CNN branch is designed to capture local features, while the Transformer branch focuses on global features. The two branches run side by side, and a special module called the CIT module helps them work together. This CIT module allows local and global features to interact, improving the network's ability to recognize emotions.

The results show that CIT-EmotionNet outperforms several leading methods, achieving high accuracy rates on two commonly used EEG datasets.

#Emotion Recognition

Recognizing emotions through EEG signals presents various benefits. There are numerous applications, such as improving brain-computer interfaces, aiding in diagnosing emotional disorders, and even enhancing safety for drivers by monitoring their emotional state. Emotions can be complex, arising from both internal thoughts and external stimuli. Various physiological signals can indicate emotional responses better than mere facial expressions or speech, which can sometimes be misleading.

EEG signals, in particular, provide real-time, high-resolution data on brain activity. This makes them ideal for capturing the subtle changes occurring during emotional responses. Techniques based on EEG signals can yield better accuracy due to the established correlation between brain activity and emotional states.

#CNN Branch

The CNN part of CIT-EmotionNet uses a specific structure known as ResNet, which helps solve some common problems faced in deep learning, such as vanishing gradients. It connects different layers selectively to maintain the flow of information. The input to this CNN structure is the EEG feature representation, and it uses several stages to process the data.

Initially, the EEG features are processed to extract important details without losing critical information. Each stage will refine the data further, outputting various feature representations. The goal here is to ensure that the model can focus on the fine details of the EEG data, which are essential for accurate emotion recognition.

#Transformer Branch

Unlike the CNN branch, the Transformer branch of CIT-EmotionNet focuses on the broader context of the EEG signals. The Vision Transformer (ViT) is used for this purpose, which helps in understanding the relationships between different areas of the brain's activity simultaneously.

As with the CNN branch, the EEG data are processed in steps. Each step includes breaking the data into smaller units (patches), which allows the model to perform calculations that capture relationships across the entire dataset. This is particularly important because it allows the model to gain insights into how different parts of the brain might interact during emotional experiences.

#CIT Module

To merge the strengths of both branches, CIT-EmotionNet includes the CIT module. This module is where local features from the CNN branch and global features from the Transformer branch can interact. There are two key blocks within this module: the L2G block and the G2L block.

The L2G block helps transform local features from the CNN to a form that can be understood by the Transformer. It takes the local features, reduces their size, and adjusts them to fit into the space needed by the global features.

On the other side, the G2L block takes information from the Transformer and converts it back into a format that the CNN can use. This interaction ensures that both types of features complement each other, leading to better performance in recognizing emotions overall.

#Experimental Setup

In practice, CIT-EmotionNet was tested using powerful GPUs and specific software tools for implementation. The model was trained with a variety of settings, including a learning rate, batch size, and dropout rate. The purpose was to enable the model to handle different types of EEG data efficiently.

Two widely used datasets, SEED and SEED-IV, provided a rich source of data for testing. These datasets contain labeled emotional responses, allowing for effective evaluation of the model's performance.

#Results

CIT-EmotionNet showed impressive results when compared to other prominent methods. The model achieved accuracy levels higher than many existing models on the same datasets. This ability to accurately recognize emotions demonstrates the effectiveness of integrating local and global features in EEG signal analysis.

The results indicate that CIT-EmotionNet not only excels in combining features but also in capturing the nuances of EEG signals. These findings support the model's potential for practical applications, from enhancing computer interfaces to assisting in emotional health diagnostics.

#Ablation Studies

To further validate the effectiveness of CIT-EmotionNet, researchers conducted ablation studies. This approach involved systematically removing certain components from the model to see how they influence overall performance. These experiments helped clarify which parts of the model are most beneficial for achieving high accuracy.

By testing variations of the model, researchers discovered important insights regarding the contributions of each feature extraction technique. The experiments confirmed that the interplay of local and global features significantly boosts the recognition capability.

#Conclusion

The CIT-EmotionNet model represents a significant advancement in emotion recognition from EEG signals. By effectively combining local and global features using a parallel approach of CNNs and Transformers, the model sets a new standard in the field. Its high accuracy rates on standardized datasets indicate strong potential for real-world applications in areas such as emotional health monitoring and responsive brain-computer interfaces.

The research emphasizes the importance of integrating diverse information sources in understanding emotions, highlighting opportunities for further exploration and development of emotion recognition technology. Future work could expand on the findings and adapt them for use in various practical settings, continuing to improve how we understand and respond to human emotions.