Evaluating Language Models: A New Approach

Large Language Models (LLMs) are becoming important tools in many areas. They show real skill in tasks such as understanding and generating human-like text. Because of this, many people are interested in using these models in real-world situations. However, assessing how well these models perform is not straightforward.

#Evaluating Large Language Models

Evaluating LLMs is tricky for several reasons. One issue is that the test data used to assess the models may not be perfect, often tainted by information from the model's training. Another problem is that traditional methods of evaluation, which often rely on specific metrics, may not truly reflect how well the model performs. Human evaluations are helpful, but collecting these ratings can be hard. Therefore, some researchers want to use LLMs themselves to evaluate text.

#Challenges with LLM Evaluators

Using LLMs as evaluators has its own challenges. Previous studies found that when LLMs evaluate text, their judgments sometimes do not align well with human opinions. LLMs can also display bias in their assessments. Additionally, many evaluations lack the depth needed to provide a complete picture of quality. This raises questions about whether LLMs can accurately replace human evaluations, especially in multilingual situations.

#Our Framework for Evaluation

To address these challenges, we created a structured way to evaluate LLMs in multilingual contexts. Our approach includes building a dataset with human ratings from native speakers in multiple languages. This dataset focuses on summarization tasks and is designed to compare how well different LLMs perform as evaluators.

#Dataset Creation

We developed a special dataset that contains 1,000 summaries in 10 different languages. Each summary was judged by native speakers on five different quality metrics. The languages included in our dataset were English, French, Chinese, Hindi, Arabic, Bengali, Russian, Turkish, Japanese, and Swahili. We chose these languages to ensure a broad coverage of scripts and cultural contexts.

#Summary Generation Process

To create this dataset, we started with source text and used an LLM (GPT-4) to generate good and bad summaries. For good summaries, we provided instructions for the model to create concise and informative texts. For bad summaries, we prompted the model to produce lower-quality content. We controlled the generation process to ensure a range of quality in the outputs.

#Annotation by Native Speakers

Once the summaries were generated, we had them rated by three native speakers for each of the five evaluation metrics. These metrics were:

1. Linguistic Acceptability - Whether the summary sounds natural to a native speaker.
2. Output Content Quality - The overall quality of the summary, considering repetition and clarity.
3. Task Quality - How well the summary matches the key points of the original text.
4. Problematic Content - Checking if the summary includes any offensive or misleading content.
5. Hallucinations - Assessing if the summary strays from the actual information in the original text.

#Analysis of Evaluators

We tested several LLMs, including GPT-3.5 Turbo, GPT-4, and PaLM2, to see how well they performed as evaluators. Our results showed that GPT-4 was the most accurate evaluator across different languages. In contrast, GPT-3.5 Turbo performed poorly.

#Reasoning Behind Evaluations

After analyzing the evaluations, we noticed that even though some LLMs did well in matching human ratings, their reasoning often did not align with the justifications provided by human annotators. This inconsistency raises concerns about relying solely on LLMs for text evaluations.

#Related Work

Numerous studies have looked at how human evaluations can help in assessing various language models. Some focused on automated metrics like ROUGE and BLEU, but these methods often fail to capture the nuanced quality expected in human judgment. Our work builds on these previous efforts by creating a more systematic approach.

#Limitations of Existing Metrics

Traditional metrics such as ROUGE or BLEU focus on exact matches of phrases, but they do not consider aspects such as coherence or overall quality. This limitation can lead to unreliable evaluations. New metrics that account for subjective quality aspects are gaining popularity as a way to improve the evaluation process.

#Findings and Results

From our experiments, we found significant differences in how the LLMs evaluated summaries. For most of the metrics we studied, human evaluations showed the best agreement. In cases where human ratings varied, GPT-4 performed better when given detailed instructions, suggesting the importance of instructive clarity.

#Challenges in Multilingual Evaluations

One major takeaway from our study is that LLMs often perform inconsistently across different languages. Although some models did well in high-resource languages, their performance dipped sharply in low-resource languages. This presents a clear challenge for using LLMs as universal evaluators.

#Future Directions

To improve upon the current framework, future research should aim to develop more comprehensive evaluation methods that consider the unique challenges associated with multilingual data. Further studies can also explore how to refine LLM prompts to enhance consistency in evaluations.

#Ethical Considerations

Creating a dataset like ours requires careful ethical considerations. We ensured that all annotators were compensated fairly and trained properly. Furthermore, we made sure that the data used was public and appropriate for the task at hand.

#Conclusion

In summary, our framework for evaluating LLMs in multilingual contexts opens up new avenues for assessing how these models function as evaluators. While we found that GPT-4 performed best under certain conditions, the need for further research and improvement is evident. Our study highlights the potential and pitfalls of using LLMs for evaluation, urging the community to proceed with caution.

#Acknowledgments

We appreciate the contributions of everyone involved in creating and evaluating the dataset. The collective effort in this study underscores the importance of collaboration in achieving meaningful results in the field of natural language processing.

This research presents both opportunities and challenges in the ongoing development of LLMs, paving the way for future advancements in language understanding and generation technology.