Enhancing Language Models for Uralic Languages

The training of language models using many languages at once often leads to poor results for less-used languages. While there are many languages in the world that do not have enough data for effective training, research shows that these less-used languages can improve when models are trained using languages closely related to them. This paper tests how to best adapt a pre-trained language model to a specific language family, focusing on the Uralic family, which includes languages like Finnish and Hungarian, along with more endangered languages like Sámi and Erzya. The goal is to train models that work well for as many of these languages as possible.

#Background

Most language models today rely on data from widely used languages, particularly English. This often causes challenges for languages with fewer resources. To solve this, multilingual models pool data from different languages to train a single model. However, these models tend to struggle with less-used languages. The idea of "targeted multilinguality" suggests that training on languages that are similar can lead to better outcomes for these less-used languages.

While many studies have looked at training language models from scratch for groups of related languages, this paper takes a different approach. Instead, it investigates how to take existing multilingual models and adjust them to focus on a smaller, more manageable set of languages.

#Methodology

In this study, we focus on adapting the XLM-R model, which has already been trained on many languages, to the Uralic language family. The Uralic family includes both mid-resource languages like Finnish and low-resource languages like Komi and Sámi. The two main methods for adapting the model are:

1. Multilingual Language-Adaptive Pre-Training (Lapt)
2. Vocabulary replacement and specialization

Through experiments, we analyze how well these methods work for the Uralic family.

#Data Collection

To prepare for training, we collected text data from various sources, including the OSCAR corpus, OPUS translation corpus, and the Johns Hopkins University Bible Corpus. For high-resource languages such as Finnish and Estonian, we gathered all available training data. For low-resource languages, we had to rely on smaller datasets from different sources.

The gathered data shows a massive difference between high-resource and low-resource languages. For instance, data for Estonian far exceeds that for Komi, showcasing the challenges faced by less-used languages.

#Adapting Vocabulary

To make the model fit the Uralic languages better, we trained a new vocabulary based on a subset of data. This new vocabulary was initialized using a method called the Focus algorithm, which helps the model understand the structure of the language better. With this approach, we tested different vocabulary sizes to see how it affects performance.

#Experiments

#Evaluation Tasks

Our analysis focused on two main tasks:

1. Part Of Speech (POS) tagging
2. Unlabeled Attachment Score (UAS) for syntactic parsing

Both tasks were evaluated using the Universal Dependencies treebanks, which provide high-quality data for many languages.

To assess model performance, we tested three evaluation settings:

• Few-shot: Fine-tuning the model on a small amount of data (512 sentences).
• Full-finetune: Fine-tuning the model on all available data for a language.
• Zero-shot: Testing the model on a language without any fine-tuning, relying on training data from related languages.

#Baselines

We compared our adapted models to:

• The original XLM-R model with no modifications.
• An XLM-R model adapted with Lapt but without changes to the vocabulary.

#Results

#Multilingual Adaptation

Our results showed that adapting the model for the Uralic language family led to significantly better performance than adapting models for individual languages alone. The multilingual models outperformed both the original and Lapt-only models.

#Specialized Vocabulary

Having a specialized vocabulary proved beneficial, particularly for low-resource languages. Smaller Vocabularies performed well and were more computationally efficient, requiring less processing power and memory.

#Analysis of Hyperparameters

We found that several factors influenced the success of the adaptations:

1. Lapt Steps: More training steps generally improved performance.
2. Vocabulary Size: Larger vocabularies helped but not as much as increasing training steps.
3. Sampling Alpha: Using a lower sampling alpha during training led to better results for low-resource languages without harming high-resource language performance.

#Evaluation of Language Performance

When analyzing how different languages performed, we noted that some high-resource languages also benefited from the multilingual approach. However, certain low-resource languages, such as Skolt Sámi, consistently struggled across different tasks.

#Discussion

#Challenges with Skolt Sámi

The low performance on Skolt Sámi suggests that the training data for this language did not align well with the evaluation tasks. Lack of quality data can hinder model training, especially when there is a significant difference in how the language is written compared to training data.

#Recommendations for Future Work

From our findings, we have several recommendations for adapting models to less-used languages:

1. Emphasize Multilingualism: It is more effective to adapt models for groups of related languages rather than training each one separately.
2. Focus on Vocabulary Size: Start with smaller, specialized vocabularies to ensure computational efficiency.
3. Use Lower Sampling Alpha: In multilingual training, applying a lower sampling alpha encourages better performance for low-resource languages.

#Conclusion

In summary, adapting a pre-trained multilingual model to a specific language family can significantly enhance performance for less-used languages. Our study underlines the importance of targeted multilingual adaptation, which avoids the problems seen in massively multilingual models, while maximizing the benefits of multilingual training. By leveraging existing models and focusing on better vocabulary management and adaptive training techniques, we can better support the linguistic diversity of the world through natural language processing advancements.

This work highlights the path forward for improving the applicability of language models for languages that have been historically underrepresented in the field and underscores the need for ongoing research into effective multilingual strategies.