Reforming Language Models for Diverse Opinions

When we ask a group of people what they think about a topic, we often get a mix of answers. This shows that preferences aren't just one-size-fits-all; they vary. Current ways of teaching language models to reflect these opinions, like Direct Preference Optimization (DPO), often miss the mark. They tend to focus too much on the majority opinion, leaving minority voices unheard.

To tackle this issue, we propose a new approach called Group Distribution Preference Optimization (GDPO). This method aims to align language models with the wide range of opinions within a group, by considering the beliefs that drive those opinions. By using statistical techniques to represent the group's beliefs, GDPO offers a better way to include everyone's views, compared to older methods.

#The Problem of Diverse Preferences

Imagine asking people in a town whether they like a new park. Some might love it, some might think it’s okay, and some might dislike it altogether. Current methods often focus on the majority opinion, ignoring those who feel differently. This creates a problem when trying to create a fair representation of opinions in language models.

For example, if we ask a group, "Is the availability of foreign products good for our country?" the responses could vary widely, even among family members. The challenge arises when people can't agree, leading to conflicting preferences. Existing algorithms like DPO often treat these differing opinions as noise rather than meaningful variations, which can skew results toward dominant views.

#Research Question

Given these challenges, we ask: How can we make language models align with the diverse preferences of a group?

#Introducing GDPO

To answer this question, we propose GDPO. Our approach focuses on two main goals: first, improving the model's ability to reflect diverse beliefs in a group, and second, resolving conflicts among differing preferences.

GDPO uses a concept called belief, which indicates how strongly individuals agree with certain opinions. By understanding these beliefs, we can better capture the complexity of human preferences.

#How GDPO Works

1. Belief Calibration: The model first predicts a belief for a given input. This belief is then used to generate responses that express it.

2. Preference Alignment: Instead of treating all preferences equally, GDPO prioritizes responses based on their associated beliefs.

This dual approach helps to ensure that the model reflects a wider range of opinions while managing conflicts.

#Demonstration of GDPO

#Training Dataset

To implement GDPO, we create datasets that link beliefs to preferences. First, we generate opinions based on questions about global issues. Then, we construct preference pairs based on what people believe.

#Training Objective

GDPO doesn’t try to optimize all preferences at once. Instead, it first focuses on calibrating the beliefs and then aligns the generated responses accordingly.

#Inference Time

When a new question comes in, the model predicts a belief and generates an answer based on it.

#Experimental Results

We apply GDPO in two main tasks: producing opinions on synthetic data and generating movie reviews based on real-world data.

#Controllable Opinion Generation

For this task, the model generates an opinion based on a question and then follows up with a response that aligns with that opinion. We use synthetic data that simulates conversations on worldwide issues.

#Feedback and Results

Our results show that while DPO struggles with minority preferences, GDPO effectively increases representation for both majority and minority views. This is an important step in making sure all voices are heard.

#Movie Review Generation

In another task, we assess how well GDPO can generate accurate rating scores and reviews for movies. Here, the model starts by predicting a score based on user reviews and then creates a review that matches it.

GDPO shows outstanding performance, consistently aligning with both the expected score distribution and the generated reviews.

#Related Work

#Preference Alignment with Language Models

Current alignment techniques often fail to consider that preferences can vary greatly. While methods like Reinforcement Learning from Human Feedback (RLHF) and DPO have advanced the field, they tend to focus on majority views.

#Pluralistic Preference Alignment

Some researchers have tried to address these limitations by proposing methods for aligning multiple group preferences. However, these efforts often overlook how to accurately reflect the range of opinions within a single group.

#Conclusion

Our work highlights a fundamental issue in aligning language models with human preferences: existing methods often overlook the richness of opinions within a group. GDPO offers a fresh approach, emphasizing the importance of beliefs in preference alignment. Our findings suggest that GDPO can effectively capture this diversity while producing coherent responses.

#Limitations to Consider

Even with these advancements, we recognize certain limitations. This study focuses mainly on preferences within a single group. Future work should explore how to accommodate preferences across different groups.

Additionally, while our experiments utilized datasets where beliefs were explicit, many real-world scenarios may not have such clear belief statements. We suggest using advanced techniques to better infer these implicit beliefs from preference data.

Through GDPO, we have taken important steps toward a more inclusive representation of group preferences in language models, ensuring that everyone's voice can be heard, even in a crowded room!