Advancements in Semantic Textual Relatedness with RoBERTa

Semantic Textual Relatedness (STR) is important in the field of Natural Language Processing (NLP). It helps to find out how closely two pieces of text are related in meaning. STR has many uses, including spelling correction, figuring out the meaning of words, plagiarism detection, opinion analysis, and retrieving information. Traditionally, people used knowledge-based and statistical methods for STR tasks. However, the arrival of Large Language Models has changed the way these tasks are approached, leading to new methods and techniques.

#Methodology

In this study, we focused on sentence-level STR by using a specific technique. We worked on Track A (Supervised) and used fine-tuning methods on a popular model called RoBERTa, which is a type of transformer. Our goal was to see how well this method works across different languages.

We made some interesting discoveries during our experiments. Our results showed improvements in STR Performance, especially in Latin languages like English and Spanish. We achieved a strong correlation of 0.82 in English and ranked 19th. In Spanish, our score was 0.67, putting us in 15th place. However, we faced difficulties with Arabic, where our correlation dropped to 0.38, ranking us 20th.

#Understanding STR

STR shows how the meanings of language units are connected, showing how closely they relate. For example, the words "cup" and "coffee" are related but not the same. STR is important for various NLP tasks, but it has received less attention compared to Semantic Textual Similarity (STS) because there are not many Datasets available.

To address this issue, we created the first datasets for sentence-level STR. We tackled the STR problem within Task 1, using the datasets for English, Spanish, and Arabic provided for the project. We also looked at Track C and included extra information as a secondary aim.

#Model Design

Our system uses a pre-trained RoBERTa model for making predictions about how related two sentences are. We fine-tuned this model to give a score that shows the relatedness of the input text. During its initial training, RoBERTa was taught to understand language by seeing many different contexts.

We also added a Classifier Head to our system, which helps classify sentences. This part is essential as it allows the model to determine how related the sentences are.

Our results show good performance on the English and Spanish datasets, with scores of 0.82 and 0.67 respectively, surpassing the baseline scores. Unfortunately, our performance on Arabic was not as strong, with a correlation score of only 0.38. This could be due to different training methods used for Latin and non-Latin languages.

To support other researchers, we made our complete code available online.

#Datasets and Structure

The SemEval-2024 Task 1 has three tracks, with our main focus on Track A, which used labeled data for training STR systems. The datasets for Task 1 consist of training, development, and test sets in 14 languages, with each pair of sentences getting a score from 0 to 1, where 0 means no relatedness and 1 means high relatedness.

Participants must predict these scores based on the sentence pairs, advancing research in NLP.

#Related Research

Research in sentence-level STR has been limited due to a lack of datasets. Existing datasets have mostly focused on simpler text connections, but earlier works laid the groundwork for further analysis by creating the first sentence-level STR datasets.

Both STR and STS have been studied using knowledge-based and statistical methods. These include using resources like thesauri and dictionaries for STR. Statistical methods rely on features from larger language collections.

Recently, deep learning methods have gained popularity for STS tasks, leading to better results. Different models, including Tree-LSTM, Bi-LSTM, and hybrid models combining LSTM and CNN, have been introduced and improved upon. These innovations highlight the changing landscape within NLP.

Studies show that fine-tuning transformer-based models leads to great results in interpreting text. The transformer model uses attention mechanisms to understand word meanings. Variations of transformer models, such as BERT and its successors, have demonstrated increased effectiveness when trained on larger datasets.

#System Challenges

One challenge we faced was using T5 to create more training data. The original dataset was validated by humans, while the additional data was not, raising concerns about its quality. Moreover, changing the data diversity during augmentation was another hurdle, as we tried to keep the dataset representative.

Deciding to use data augmentation only for testing raised concerns about the possible impact on model quality. These challenges need to be addressed to improve our model's effectiveness.

#Experimental Setup

We used a portion of the dataset for training and reserved part for evaluation. The limited amount of training data required careful testing, as it could affect the model's performance. We also utilized the entire development set for selecting the best model.

Our pre-processing focused on converting labels into float values to match the model's requirements. We also tokenized the input texts using the RoBERTa tokenizer.

Tuning hyper-parameters is crucial to maximizing model performance. We tested various rates and sizes to find the best combination. Our final settings achieved the best results across all languages.

#Evaluation Metrics

We evaluated our model's performance using Mean Squared Error (MSE) to measure the prediction accuracy. The Mean Absolute Error (MAE) and R-squared score were also used to assess the quality of our predictions.

These evaluation methods provide a complete picture of how well our regression model predicts the relatedness between different text samples.

#Results

When comparing our model's performance on English, Spanish, and Arabic datasets, we noticed differences. The English model scored the highest correlation points, showing that it captured semantic links well. The Spanish model also performed decently, but the Arabic model lagged significantly.

The performance gap could be due to the availability of training data or the structural differences between Arabic and Latin languages. This indicates the need to understand the unique characteristics of each language when developing models.

The analysis of the Arabic model showed that it tended to classify most inputs as highly related, demonstrating its struggles in differentiating between varying degrees of relatedness.

#Visual Analysis

We also examined scatter plots that showed relationships between our predictions and human ratings. The English model aligned closely with human judgments, while the Spanish model performed well on some inputs. The Arabic model showed discrepancies, indicating areas for improvement.

We learned that dataset size and language specifics affect model performance. More investigations are essential to find ways to enhance the model's results, especially in languages with less training data.

#Error Analysis

Through our error analysis, we highlighted the model's strengths and weaknesses. The confusion matrices displayed how the English model performed well in certain score ranges but needed improvement for highly related scores.

The Spanish model showed proficiency in predicting less related sentences but struggled with more related ones. The Arabic model had different issues, as most predictions fell in the middle but were frequently incorrect.

Observations showed that the models had difficulty with scores at both ends of the scale, indicating a need for enhancement in future training methods.

#Conclusion

In our work, we focused on fine-tuning RoBERTa for STR tasks, achieving notable results in English and Spanish. However, the Arabic model performed less well, reflecting the challenges posed by different data qualities. Our insights into Track C broadened our understanding of the challenges facing STR systems.

We suggest that developing more transformer models for various language families could lead to better outcomes. A thorough examination of top-performing models in future studies will aid in advancing STR research and improving model accuracy. By pursuing these recommendations, we can help shape the future of STR applications in NLP.