Federated Learning for Multilingual Emoji Prediction

Federated Learning is a way to train machine learning models where data remains on individual devices instead of being sent to a central server. This method is useful for keeping user information private while still allowing the model to learn from a variety of data sources. In this article, we focus on predicting emojis in different languages using federated learning, specifically looking at how well this approach works both when the data is clean and when it is under attack.

Emojis play a significant role in how people communicate online. They express emotions and add depth to messages. In recent years, the use of emojis in social media has increased. This growing usage makes emoji prediction a valuable tool for improving online conversations. By predicting the right emoji for a given text, we can enhance the communication experience for users.

#Data Collection

To train our models, we collected two million tweets from Twitter that included emojis. Additionally, we used a standard emoji dataset known as the SemEval dataset, which contains labeled emoji data for training and testing. The tweets were gathered in three languages: Spanish, Italian, and French, while the SemEval dataset is in English. We filtered this data to eliminate unnecessary elements such as stop words and links, ensuring we focused on what matters most for emoji prediction.

#Different Training Methods

We set up several training methods:

1. Centralized Training: This is the traditional way of training where all data is gathered in one place to build a model.
2. Federated Learning with IID Data: In this setup, each participant has a random mix of data, mimicking a more balanced distribution.
3. Federated Learning with Non-IID Data: Here, each participant only has data from one language, which simulates a more realistic scenario where data types can vary significantly.

For our federated learning experiments, we chose four clients based on previous research and to keep a consistent approach. We tested both clean scenarios, where all data is accurate, and attack scenarios, where some of the data is manipulated through a process called Label Flipping.

#Label Flipping Attack Explanation

Label flipping is a type of attack where the labels (in our case, the emojis) associated with the data are changed. For example, if a tweet should have a smile emoji, it might instead be labeled with a sad emoji. We tested two levels of attack: in one scenario, we altered the data of 25% of clients, and in another, 50% of clients. This helped us understand how well our models could handle such attacks.

#Federated Learning Methods

In federated learning, models on individual clients train on their own data and send their updates back to the server, which combines them into a global model.

1. Federated Averaging (FedAvg): This is a straightforward method where model updates from all clients are averaged to create a new model.
2. Krum: This is a more complex method that selects the most accurate updates from clients, ignoring data from any clients that may have sent misleading information.

While Krum requires more computational effort, it can offer better performance in scenarios where data may be compromised.

#Training Models

We used several transformer models of different sizes for our studies. Some of the models were derived from popular sources known for their effectiveness in handling multi-language tasks. We trained these models in both centralized and federated learning settings, focusing on how well they predicted the right emojis.

Training took place for 30 epochs, a term that means we went through the entire dataset repeatedly to train the model. We used a friendly framework for federated learning which helped make the training process smoother.

#Results from Experiments

We structured our experiments into three key stages:

1. Baseline Training: Here, we assessed how well our models performed under traditional centralized conditions to establish a performance benchmark.
2. Federated Training: We then took the trained models and ran them through federated learning setups, testing both clean and attacked scenarios.
3. Fine-tuning: In this final stage, we fine-tuned the models in a centralized setting using all the combined data.

Through these experiments, we measured performance using two main metrics: Macro-F1 and Micro-F1 scores. The Macro-F1 score treats each class equally, which is helpful when dealing with unbalanced datasets.

#Performance Analysis

#Centralized vs Federated Learning

In our tests, centralized training generally produced strong results. However, when we applied federated learning, we found that the models still maintained competitive performance. For instance, federated learning setups with clean data often matched the accuracy of centralized setups, even when languages and data distributions varied.

#Impact of Label Flipping

When label flipping was introduced, we observed significant drops in performance. The models trained using FedAVG struggled, particularly when facing substantial portions of toxic data. However, Krum performed much better, demonstrating that it could resist the negative effects of attacks effectively.

Our results showed that Krum could restore accuracy even in scenarios where 50% of the input data was compromised. This outcome highlights the importance of choosing the right aggregation method in federated learning.

#Multilingual Predictions

We also tested how well our models performed across different languages. Although there was a slight decline in performance when multiple languages were involved, the models still managed to handle the multilingual input reasonably well. This indicates that our approach is versatile and capable of working across diverse languages.

#Unseen Language Performance

Testing our models on languages they had not seen during training illustrated that they could still perform adequately. While there was some decline in accuracy, the federated models maintained competitive performance compared to traditional models.

#Conclusion

Our research into federated learning for multilingual emoji prediction shows promising results. The ability of federated learning to maintain privacy while learning from diverse data sources is valuable, especially in a world where data security is paramount.

By training models in both clean and attacked scenarios, we learned that using sophisticated methods like Krum can help protect against data poisoning attacks. The results suggest that federated learning can serve as a strong alternative to centralized approaches, particularly for tasks like emoji prediction where privacy and data diversity are key.

In future work, we hope to focus on improving communication efficiency in federated learning and exploring how to better personalize models based on user interactions with emojis.

Through ongoing research, we aim to create more robust systems that adapt to user needs while ensuring their data remains secure.