Rationales in Argument Ranking by Language Models

Large Language Models (LLMs) have become good at generating free-text explanations, called Rationales, to support their decisions. These rationales are important because they can help users understand why the model made a certain choice. Recently, there has been a lot of interest in how these rationales can be used in tasks where the answers are not clear-cut or factual. This study looks at rationales in situations where opinions matter, focusing on a specific task called pairwise argument ranking. This task involves comparing two arguments on the same topic and deciding which one is stronger.

#Importance of Rationales

When models provide rationales, they add clarity and trust to their decisions. This is especially helpful in areas like debate support, where understanding the reasoning behind an argument is crucial. By giving persuasive reasons for their choices, LLMs can be more effective and reliable in various applications.

#The Task of Pairwise Argument Ranking

In pairwise argument ranking, a model looks at two arguments that have the same position or viewpoint on a topic and selects the better one. The model then generates a rationale explaining its choice. This task is subjective, meaning people might disagree on which argument is superior. Given the subjective nature of this task, we will assess how persuasive the generated rationales are.

#Research Questions

To guide this study, we raised several important questions:

• How do different LLMs stack up against each other in generating persuasive rationales?
• Can we automatically find out which rationales are more persuasive?
• What features of a rationale make it more convincing?
• Can we make the rationales generated by models more persuasive?

#Methodology

We prompted various LLMs to perform pairwise ranking without any prior training (zero-shot) and to provide rationales for their choices. We also used human Evaluations to assess the persuasiveness of the rationales and examined ways to enhance their persuasive qualities.

#Selection of LLMs

We looked at several LLMs, including some that are open-source and others that are closed-source. The open-source models included popular versions like Llama2, while Closed-source models included the well-known GPT series. We used different versions of the models to see if size and training made a difference in the persuasive ability of the generated rationales.

#Dataset Preparation

To evaluate the rationales, we used two main Datasets that contained pairs of arguments. The first dataset, IBM-ArgQ-9.1kPairs, had pairs of arguments on various topics, while the second dataset, IBM-30k, included arguments each rated for quality. From these datasets, we filtered and selected pairs of arguments for analysis, ensuring that we focused on high-quality examples.

#Evaluation Stages

Our evaluation process consisted of three key stages:

1. Basic Evaluation: We checked the rationales to see if they were clear and coherent. If a rationale didn’t make sense or repeated the argument without adding anything new, it was ignored.

2. Content Evaluation: Here, we looked at the substance of the rationale. We analyzed whether the rationale offered contrasting views on the arguments and whether it introduced new ideas.

3. Persuasiveness Evaluation: This final stage assessed how convincing the rationales were. We asked human reviewers to rate the rationales in pairwise comparisons, allowing us to determine which rationale was more persuasive.

#Findings

#Overall Performance

Our results showed that Llama2-70B-chat generated the most persuasive rationales, even outperforming the well-known GPT models. This highlights the potential of open-source models in generating effective explanations for their decisions.

#Human and Automatic Rankings

In most cases, GPT4 closely matched human rankings of rationales, although it did have some discrepancies in cases where rationales were similar in quality. This indicates that while automatic evaluations can be helpful, human judgment still plays an important role in assessing persuasiveness.

#Key Features of Persuasiveness

We identified several characteristics that contributed to the persuasiveness of rationales. The most important feature was contrast. Rationales that explained why an argument was stronger than its counterpart were found to be significantly more persuasive. Length also mattered; longer rationales that provided detailed support for the model's choice were often more convincing.

#Enhancing Persuasiveness

To enhance the persuasiveness of rationales, we tested methods such as re-prompting the models to focus on contrast and detail. This technique improved the persuasiveness of the outputs from models that initially struggled to generate compelling rationales. However, even with these improvements, the results still fell short compared to the outputs generated by more advanced models.

#Conclusion and Future Directions

This study offers valuable insights into the persuasive abilities of rationales produced by various LLMs. The findings suggest that open-source models, specifically Llama2-70B-chat, can create persuasive rationales that are practically useful for subjective tasks. The importance of contrast in rationales was emphasized, along with the potential to improve outputs through specific prompting techniques.

Future work will investigate user acceptance of model-generated arguments and explore other subjective tasks where understanding the reasoning is critical. We also aim to consider additional factors that may influence rationales, seeking a deeper understanding of how different models support their choices.

As we continue this research, it is crucial to remain aware of the ethical implications of persuasive rationales, particularly in how they might influence decision-making and the potential for misuse.

#Ethical Considerations

While persuasive rationales can improve transparency and user acceptance, they also carry the risk of being used to support biased or false arguments. It's essential to develop responsible practices for deploying these models to prevent any potential harm.

#Results on Dataset Quality

An analysis of our datasets showed that the number of agreement among models decreases with the inclusion of more models. This reinforces the idea that some models may not align as well when assessing argument quality, necessitating careful curation of datasets used for evaluation.

In summary, our study confirms that while there are variations among LLMs in generating persuasive rationales, some models show significant promise for supporting subjective decision-making tasks. Further investigation into the factors that contribute to effective rationales will be beneficial as the field continues to evolve.