Advancing Code Clone Detection in COBOL

The world of programming languages is constantly changing. Some languages, like COBOL, have been around for decades, while newer ones like Python and JavaScript are more popular today. However, a lot of COBOL code is still used in companies and organizations. This article talks about a new approach to find similar pieces of code (known as Code Clones) in COBOL programs, even when there is little to no available data for training models.

#The Importance of Code Clone Detection

Code clone detection is essential for several reasons. First, it helps developers reuse code, which saves time and effort. Second, it can compress code by replacing longer pieces with shorter, similar ones. Lastly, it can help in identifying copyright issues, where one piece of code may be too similar to another. The goal is to determine if two different code segments behave the same way, regardless of their actual appearance.

Even though there has been plenty of research on code cloning for popular languages, COBOL has been somewhat neglected. This is problematic, as millions of lines of COBOL code are still in use today, with estimates suggesting over 800 billion lines globally.

#Challenges with Legacy Programming Languages

One of the main challenges with working on COBOL code is the lack of available training data. Unlike more popular languages where massive datasets exist, the datasets for COBOL are very small. For example, the CodeNet dataset contains only 727 COBOL codes related to given problem descriptions. This makes it hard to train models effectively.

In contrast, languages like Java and Python have access to datasets that are hundreds of gigabytes in size. This means that while developers can create powerful models for Java or Python, similar efforts for COBOL fall short due to the lack of data.

#Our Approach: Neuro-Symbolic Method

To tackle the challenge of detecting code clones in COBOL, we use a method combining symbolic reasoning with neural networks, which we refer to as a “neuro-symbolic approach.”

We create a framework that can convert code from both C (a more widely used language) and COBOL into a common format known as Intermediate Representation (IR). This IR is based on something called Abstract Syntax Trees (ASTs), which represent the structure of code. By transforming COBOL and C code into the same IR, we can apply one model to both languages, allowing for the transfer of knowledge gained from C to COBOL.

#Steps Involved in Our Approach

1. Transforming Code to IR: We begin by transforming the C codes into IR using a process called Structure Based Traversal (SBT). This turns the code into sequences that can be easily analyzed by our model.

2. Fine-Tuning the Model: We take a model that has already been trained for code searching and fine-tune it using our transformed C codes. This model is called UniXcoder and is known to perform well in identifying code clones.

3. Testing on COBOL Codes: Having fine-tuned our model with C codes, we then test it on COBOL codes in a way that does not require prior training on COBOL data. This is known as a zero-shot approach.

#Results of Our Method

Our results show that this method is effective. When we tested our model on COBOL codes, we gained a noticeable improvement in performance compared to using the pre-trained model without any fine-tuning. Specifically, we achieved a performance increase of over 12% when looking for similar code segments in COBOL, showing that our approach can bridge the gap between C and COBOL effectively.

#Code Representation Learning

To understand code more deeply, we analyzed its representation. Different representations can highlight various aspects of the code. For example, while the AST gives an overview of the structure, other methods might focus on control flow or the data used in the program.

Research has shown that using neural networks to learn these representations can benefit tasks like code cloning and searching. Many recent models rely on large amounts of data to learn how to analyze code better, but as we have seen, COBOL is at a disadvantage due to its lack of available data.

#Related Work

Numerous efforts have been made to detect code clones and improve code understanding in various programming languages. Some models work well with languages like Java, JavaScript, and Python by utilizing vast datasets. However, for something like COBOL, most existing models are not suitable due to the scarcity of data.

Projects like UniXcoder have made strides in zero-shot code-to-code searching, showcasing the potential for models trained with more commonly used programming languages to assist with understanding older languages like COBOL.

#Creating the Dataset

For our project, we formed our dataset using the CodeNet dataset, which was modified to only include C codes relevant to problem descriptions. After filtering the dataset, we were left with over 300,000 C codes to use for training our model.

When creating test datasets for COBOL, we focused on accepted submissions that had several code variations for the same problem. We selected a smaller set of COBOL codes to ensure we had enough data to evaluate the model’s performance accurately.

#Challenges with Code Length

When transforming code into the SBT-IR format, there were cases where the length of the transformed code exceeded the model's processing limit. To address this, we filtered out overly long codes and focused on datasets that fit within a manageable length. This careful selection allowed us to assess how well our approach worked under various conditions.

#Evaluating Performance

To evaluate the performance of our model, we used a metric known as Mean Average Precision (MAP). This score helps to understand how well the model identifies similar code segments accurately.

Our results indicated that the model trained with the C-SBT representations performed significantly better than a simpler model trained from scratch. This improvement highlights the benefits of using existing knowledge gained from more abundant languages to assist with less common ones.

#Conclusions and Future Work

In conclusion, our neuro-symbolic approach shows promise in addressing the challenges associated with code clone detection in COBOL. By using a common IR representation for C and COBOL, we could leverage the extensive data available for C to enhance performance in COBOL without requiring additional data specific to it.

Looking ahead, we plan to explore newer models that can further improve our results. Large Language Models (LLMs) have shown advanced capabilities in various tasks and may provide better ways to approach code understanding, especially in low-resource settings like COBOL. We aim to compare our methods with those using these newer models to see how they perform against one another in this unique context.