Harnessing Expert Knowledge for Better Data Labeling

For machine learning to work well, it needs a lot of good quality data that has been labeled correctly. This labeled data tells the machine what the information means, which helps it learn better. However, getting this kind of data can be challenging and expensive, especially when specialists are needed to label the data. In some cases, it may even be impossible if the task requires a single expert to do all the work.

#Using Experts to Source Knowledge

In order to tackle this problem, we can use a method that gathers knowledge from experts in the field. It’s essential to design this process in a way that encourages experts to take part and share their knowledge. In our work, we are focusing on finding true Synonyms from a list of possible synonyms. This is particularly important in fields where people from different backgrounds need to work together, like in business settings where clear communication is crucial.

#The Role of Synonyms in Communication

When different teams collaborate, they often use different terms to describe the same things. This can lead to misunderstandings. For example, if one team refers to a certain object using a specific term, and another team uses a different term, they might not realize they are talking about the same thing. By identifying true synonyms, we can improve communication and collaboration among teams.

#The Challenge of Synonym Detection

Getting a good list of synonyms can be tough. In our case, we have a system that tries to identify synonyms based on examples from a specific area of work, in this case, Construction. The system looks at how terms are used in a written collection of texts and establishes possible synonyms. However, early tests showed that only a small percentage of the identified synonyms were actually correct. Only about 1% of the candidates were true synonyms. While this may sound disappointing, finding synonyms using machines is not easy and should be considered against the time and effort needed for manual searches.

#Gathering Expert Input Through Crowdsourcing

Going through all the potential synonym candidates-over a million-would be an overwhelming task for a single person. While it’s possible to ask the general public to help, doing so without proper guidance would likely lead to unreliable results, especially since the task requires knowledge of the specific language used in the construction industry. Instead, we opted for a more controlled approach using a crowdsourcing platform that allows us to manage the task effectively. This platform lets us choose who will participate, how the data will be stored, and how tasks are designed.

#Designing Effective Tasks

We have structured the synonym validation task into two clear phases. The first phase involves selecting synonyms from a list of candidates. In the second phase, the expert receives Feedback on their choices. This feedback is crucial because it helps experts improve their understanding of the synonyms and see how their choices compare with those of others.

To make the task easier, we show context for each term, including a definition and its place in a larger classification system. Providing this background not only aids experts in their selection but also promotes learning within the organization, helping everyone speak the same language.

#Feedback and Results

After completing the selection, we analyze the results immediately. Experts can see how well their choices matched with those of previous users. This immediate feedback can help them learn and adjust their selections in future tasks. If most experts agree on a particular synonym, it gets identified as a new synonym through a simple voting process.

#Motivating Experts to Participate

Motivation is key in crowdsourcing tasks. We designed the synonym validation task to create a win-win for everyone involved-experts, management who pays for the task, and researchers. Past studies show that people are generally more motivated by internal factors, like learning and community connection, than by external rewards like money.

To keep experts engaged, we provide them with information about the terms they are working with, show them how their choices align with others, and let management track how much time is being spent on these tasks. This data can help justify the funding for such projects.

#Keeping Experts Engaged

To keep experts interested, we also randomize the tasks they receive. This helps reduce bias and keeps them attentive. If an expert hasn’t selected a synonym after a certain number of tasks, we randomly introduce a true synonym to encourage engagement and ensure they are paying attention.

#Looking Ahead

As we continue refining this process, our goal is to use it with a larger group of experts. We want to collect synonyms not just for a specific project, but also to create a framework that can be applied to other data labeling tasks. For example, we could adapt this method to assess the quality of different project documents or to evaluate the clarity of code.

#Conclusion

By using expert knowledge in a systematic way, we can gather high-quality labeled data for machine learning tasks. Our approach shows that with the right design, crowdsourcing can be an effective way to label domain-specific data. The lessons learned here are applicable to various projects in software engineering and beyond, paving the way for better collaboration and clearer communication among teams.