Evaluating Multimodal Language Models with CoIN Benchmark

In recent years, large language models that can handle both text and images have gained a lot of interest. These models, known as Multimodal Large Language Models (MLLMs), have shown great promise in understanding and generating content that involves both visuals and text. A common method to improve these models is called Instruction Tuning, where the model learns to follow human commands better and adapt to various tasks based on instructions.

However, these models face challenges in keeping their existing knowledge while also learning new information or commands from users. This is where the concept of Continual Learning comes into play. Continual learning focuses on the ability of a model to learn new things without forgetting what it has already learned. The goal is to balance the ability to learn new tasks (plasticity) with the need to remember previous knowledge (stability).

This article presents a new benchmark called Continual Instruction tuNing (CoIN), designed to evaluate how well current MLLMs perform in this continual instruction tuning process. CoIN consists of ten datasets covering eight different tasks, aiming to offer a diverse set of instructions. The trained models are assessed based on two key aspects: how well they follow instructions and how much General Knowledge they retain for reasoning.

#The Challenge of MLLMs

MLLMs have the capability to combine visual and textual information, making them quite powerful. They usually undergo a two-phase training approach. First, they align visual data with text data to create a foundational understanding of the two modalities. In the second phase, they are fine-tuned using carefully designed instruction data to help them follow human commands better.

Despite their advanced abilities, these models still struggle to update their knowledge and adapt to new instructions effectively. It's been found that multi-task training, where models are trained on both old and new commands, is a promising approach. However, starting the training process from scratch with each new instruction can be expensive and time-consuming. Therefore, finding ways for MLLMs to learn new information while keeping their old skills is essential.

#A New Benchmark: CoIN

To better understand how MLLMs perform in a continual instruction tuning environment, the CoIN benchmark has been created. This benchmark includes ten commonly used datasets that cover a range of tasks like visual question answering, image classifications, and more. By having a variety of tasks and instructions, CoIN aims to provide a comprehensive evaluation of MLLMs.

In the CoIN assessment, models are evaluated based on two perspectives: Instruction Following and General Knowledge. Instruction Following measures how well the model aligns with human intent, while General Knowledge assesses how much knowledge the model retains for reasoning tasks.

#Findings from CoIN Experiments

Initial experiments using CoIN indicate that many MLLMs still experience significant forgetting, where they lose the ability to follow previous instructions rather than losing knowledge itself. This issue of "catastrophic forgetting" happens when learning new tasks interferes with the model's ability to remember older tasks.

To address this, a method called Mixture-of-Experts (MoE) was introduced to MLLMs. This method allows the model to utilize separate experts that specialize in different areas of knowledge. By leveraging these experts, the model can retain its ability to follow previous instructions while also learning new ones. Results from experiments show that this method effectively reduces forgetting.

#The Importance of Instruction Tuning

Instruction tuning is vital for MLLMs because it helps them follow natural language commands. Various strategies have been employed to create instruction data, ranging from using existing datasets to generating new instructions based on powerful language models. However, the focus on traditional task types can limit the diversity of the instructions.

CoIN attempts to overcome this limitation by incorporating a wide range of tasks and instruction templates. This diversity aims to test the models thoroughly and understand how they adapt to different types of instructions.

#Evaluation Methods in CoIN

The evaluation of MLLMs in CoIN is based on two main aspects: Instruction Following and General Knowledge.

#Instruction Following

This aspect examines how well the model can generate the correct response in the desired format to meet human intent. To evaluate this ability, the outputs of MLLMs are compared directly to the ground truth, which serves as the correct response. Various metrics are used to measure accuracy for different tasks.

For example, in visual question answering tasks, accuracy is calculated based on how many answers the model gets right. For classification tasks, performance is assessed by comparing predicted labels with actual labels.

#General Knowledge

General knowledge assesses the understanding that models possess beyond merely following instructions. Evaluating general knowledge involves analyzing the predicted results at a semantic level, considering whether the information contained in the model's response is logically accurate.

To do this, another powerful language model is used to evaluate the outputs without focusing on the structure, looking instead at the core information. This allows for a more nuanced understanding of what the model knows beyond just following commands.

#Key Insights from CoIN

The results from CoIN reveal several important insights regarding MLLMs and their Instruction-following capabilities.

1. Significance of Diverse Instructions: Models perform better when trained on a variety of tasks and instructions. The ability to adjust to various instructions leads to enhanced performance compared to using a single type of instruction.

2. Impact of Training Data Volume: The volume of training data influences performance, where more data tends to improve results up to a certain point. However, if too much new information is introduced too quickly, it can lead to forgetting previously acquired knowledge.

3. Role of Experts: The number of experts used in the MoE framework significantly affects the model’s ability to learn and retain diverse knowledge. More experts allow for better specialization, decreasing interference from unrelated tasks.

4. Forgetting Dynamics: It was observed that the forgetting of general knowledge is more manageable than the forgetting of instruction following. This indicates that while models can retain information, they can struggle to align with specific human intents.

#Conclusion

The CoIN benchmark opens up new avenues for evaluating MLLMs in the context of continual instruction tuning. By focusing on diverse tasks and applying evaluation methods that consider both instruction following and general knowledge, researchers can better understand how these models function and how to improve their capabilities.

As MLLMs continue to evolve, the insights gained from benchmarks like CoIN will help guide the development of better strategies for instruction tuning, ultimately leading to more robust models that can adapt to changing user needs without losing what they have already learned.

This ongoing research into how MLLMs learn and remember will be crucial in advancing the field of artificial intelligence, particularly in applications that require a deep integration of text and visual information.