What does "WSC+" mean?
Table of Contents
WSC+ is a new dataset created to improve how machines understand and generate questions. It builds on an existing test called the Winograd Schema Challenge, which checks how well machines can answer tricky questions that involve understanding language.
How Does WSC+ Work?
To make WSC+ better, a new method called Tree-of-Experts was used. This method helps create a larger number of valid questions. In fact, WSC+ has many more valid questions than earlier efforts. It includes over 3,000 sentences made by a machine.
What Makes WSC+ Different?
WSC+ stands out because it adds new types of tricky questions that machines can face, including some that are ambiguous or offensive. This helps researchers see how confident machines are in their answers and where they might make mistakes or show bias.
Results and Insights
When the top machine, GPT-4, was tested on the WSC+ dataset, it got 68.7% of the questions right. This is quite a bit lower than the human benchmark of 95.1%, showing that while machines are getting better, they still have a long way to go.