Boosting Icelandic Language Models: Insights and Techniques

Smaller language models (LLMs) can do amazing things, but they still have some hiccups, especially in languages that are not as widely spoken. When these models try to handle languages like Icelandic, they often struggle. This is mainly because they lack the specific knowledge needed to generate good text in those languages. Even if you feed them machine-translated text, it doesn’t always solve the problem.

#The Case of Icelandic

In our quest to improve these models, we set our sights on Icelandic. The goal was to take an LLM and make it better at generating Icelandic text by training it on a bunch of unstructured text. However, we had to be careful. Too much tinkering could mess up the model’s ability to handle longer pieces of text. Think of it like trying to improve a car’s speed while also making sure it can still turn corners without flipping over.

#Parameter Efficient Fine-Tuning (PEFT)

One of the key techniques we used in this project is called parameter-efficient fine-tuning (PEFT). It’s a fancy term for a method that allows us to train our model without changing too many of its settings. We discovered that making more parameters trainable generally led to better results.

We tried different styles of PEFT, including adding special components called LoRAs and Bottleneck Adapters in various parts of the model. LoRAs in certain layers of the model showed great promise, while other methods, like prefix tuning, seemed to cause more harm than good. It’s kind of like trying to find the best places to add turbo boosters to a car—some spots just make things worse.

#Instruction Tuning and Model Performance

We also took a look at how well these models were performing when we used machine-translated data for training. While this method improved performance compared to only using English, it still didn’t quite hit the mark when it came to the actual Icelandic benchmarks. It became clear that something was missing—namely, specific knowledge about the Icelandic language.

Collecting a huge amount of native instruction-tuning data could fix this issue, but let’s be real—it’s often easier said than done. This is where the techniques we explored using unstructured text data become very useful.

#The Experiment Setup

For our experiments, we used the smallest version of the LLaMA 3.2 model, which has 1 billion parameters and has been fine-tuned for instructions. We chose a dataset that was focused on Icelandic, consisting of text chunks we felt were good quality. To make sure we had enough material, we grabbed 250,000 text segments, each up to 1,024 tokens long, resulting in a massive pile of 12.5 million tokens.

We also used data from another source, the Icelandic Gigaword Corpus (IGC), but our findings didn’t show any benefits from it. It seems that using a wide range of data might yield better results than sticking to a smaller set of curated content.

#Different Adaptation Methods

We tried various methods to adapt our language model, including:

1. LoRA: This approach added low-rank matrices to certain parts of the model. The cool thing is that you can merge these matrices back into the model, which makes things faster.

2. Bottleneck Adapters: These add smaller layers in between the main layers of the model, but they can also increase the total number of parameters and slow down the model a bit—like adding too many snacks to your backpack for a hiking trip.

3. Prefix Tuning: This method inserts a string of learnable vectors at the beginning of input sequences. It’s like adding a catchy intro to a song, but sometimes it just confuses the listener instead of drawing them in.

#Generating and Evaluating Text

To see how well our models did at summarizing texts, we used a popular dataset of news articles. We filtered out pieces missing key information, so we were left with articles that met our standards.

We tested how our models performed in different scenarios, such as 0-shot, 1-shot, and 5-shot setups. Think of this as preparing for a quiz where you might have zero hints, one hint, or five hints to help you out.

#Results: The Good, The Bad, and The Ugly

Our experiments revealed several interesting findings. When we looked at how well the language models adapted, 0-shot summarization scores consistently improved. However, in the 1-shot and 5-shot scenarios, some setups actually performed worse than when we didn’t use any adaptation at all. This led us to think that in-context learning might sometimes work just as well—like a student acing a quiz without studying!

#Best Methods

The standout performer was LoRA placed in the feed-forward layers of the model. The bottleneck adapters also boosted scores, although not as dramatically. We found that when the LoRA ranks increased or the bottleneck reduction factors decreased, our scores improved.

However, prefix tuning didn’t help our models at all. It caused some serious drops in performance, especially when the model was asked to summarize more complex inputs.

#The Placement of LoRAs

During our experiments, we dug deeper into where LoRAs should be placed. It turns out that having LoRA in the feed-forward module performed better than placing it in the self-attention module. We were surprised to find that adding LoRA to both modules didn’t really make a difference.

This has some implications for our understanding of how to get the best results out of our models. If you can boost performance without losing efficiency, why not do it?

#Layer Exclusion Experiment

We next experimented to see if leaving out the final layers during adaptation would help maintain the model’s original abilities. To our surprise, this didn’t improve performance at all. Instead, when we focused the LoRA modules on just the last two layers, we started to see better results in the 5-shot tests, although we lost a bit in the 0-shot performance.

This suggests that focusing our efforts on the right layers can lead to improvements, especially in cases where the model struggles.

#The Role of Data Quality

When we looked at the quality of our data, we didn’t see any advantage in using the Icelandic Gigaword Corpus. In fact, performance was generally lower with that dataset. This highlights the need for diverse and high-quality training data.

#Future Directions

We plan to take our findings and apply them to other languages and larger models in the future. Expanding our testing to see if longer context lengths improve performance is also on our to-do list.

One interesting idea is to use episodic memories to boost performance. Think of it as sprinkling in some examples from previous tasks to remind the model of what it learned before.

#The Need for Better Evaluations

We’ve realized that while using automated metrics like BERTScore and ROUGE-L gives us some insights, they might not give us the full picture. It might be worth conducting human evaluations on our model outputs for a broader understanding of how well it's performing.

This will help us assess different aspects of language quality and the content being generated, giving us a clearer understanding of what works and what doesn’t.

#Conclusion

In summary, adapting smaller language models for languages like Icelandic comes with its share of challenges. However, through careful tuning and innovative approaches, we can improve their performance. It's a bit like teaching a dog new tricks—you have to find the right treats to motivate them!

With further research and a focus on using high-quality data, these models could become even more capable and reliable. And who knows? Perhaps one day they'll be able to chat with you in Icelandic without missing a beat!