Improving Execution Time Predictions with Dual-Transformer Models

In recent years, the field of deep learning has made significant advancements in analyzing source code. One crucial aspect of this analysis involves understanding how well different models can predict execution time from source code. Execution time prediction helps developers get an idea of how their code will perform before actually running it.

#Background

Source code can be represented in various ways, but one effective method uses Abstract Syntax Trees (ASTs). An AST is a tree-like structure that represents the different components of the source code, stripping away unnecessary details such as punctuation. By using ASTs, models can focus on the core elements of the code, which makes analysis easier and more precise.

Traditionally, many approaches to source code analysis have treated code as simple text. These methods often struggle to capture the deeper relationships and semantics inherent in programming languages. Because of this, newer techniques that utilize ASTs and advanced machine learning approaches are becoming increasingly popular.

#Tree-Based Neural Networks

Tree-Based Neural Networks (TBNN) are one type of model that has gained traction in this area. They specifically work with tree structures and have been used mostly for classification tasks. However, their ability to handle regression tasks, such as predicting execution time, is limited.

In this study, we focused on improving the performance of TBNN models for regression tasks by developing a new approach called a dual-transformer model. The dual-transformer model works with both the source code's textual representation and its AST representation, providing a more comprehensive way to analyze execution time.

#Dual-Transformer Model

The dual-transformer model consists of two main components: one processes the source code as sequences of tokens (words, variables, etc.), and the other processes the AST. The synergy between these two components allows for better understanding and prediction of execution time. The model uses a technique called cross-attention to integrate information from both sources effectively.

The idea behind cross-attention is to allow the two components of the model to 'talk' to each other. This communication helps the model figure out how different parts of the source code influence each other and how they relate to the structure represented by the AST.

By employing this dual approach, we can leverage the strengths of both textual representations and structural representations, ultimately improving execution time predictions.

#Experimentation

To evaluate our dual-transformer model, we carried out extensive experiments using two different real-world datasets. The first dataset, OSSBuild, contains execution data collected from open-source projects. The second dataset, HadoopTests, involves execution data obtained by running unit tests on the Hadoop open-source system.

In our experiments, we compared the performance of our dual-transformer model to several other existing models, including other TBNN models and Graph Neural Networks (GNN). GNNs are another class of models that have shown promise in handling graph-structured data, and in this case, they were adapted to work with ASTs.

We measured the performance of the models using three main metrics: Mean Squared Error (MSE), Mean Absolute Error (MAE), and Pearson correlation. Lower values in MSE and MAE indicate better performance, while a higher Pearson correlation indicates a stronger relationship between predicted and actual Execution Times.

#Results

Our experiments indicated that the dual-transformer model consistently outperformed the other models across various metrics for both datasets. For instance, in the OSSBuild dataset, the dual-transformer model achieved the lowest MSE and MAE, and a high Pearson correlation score, demonstrating its predictive capacity.

In contrast, traditional TBNN models, while effective in classification tasks, showed significant limitations in the regression context. They struggled to accurately predict execution time when compared to the dual-transformer model. The GNN models also performed reasonably well but did not match the dual-transformer’s performance.

Through these experiments, we observed that the dual-transformer model, by integrating both textual and structural information, could effectively manage the complex nature of source code, leading to improved predictions.

#Importance of Data Diversity

The diversity of datasets also plays a vital role in training machine learning models. In our study, the OSSBuild dataset includes data from multiple projects, which can better expose models to various code patterns. This exposure ultimately helps models generalize better across different scenarios. On the other hand, the HadoopTests dataset is limited to a single project, leading to a more homogenous set of data, which can restrict a model's learning capabilities.

The results showed that models trained on diverse datasets, such as OSSBuild, tend to perform better when faced with new tasks or data. This finding emphasizes the need for diverse training data in machine learning applications, especially in the context of source code analysis.

#Model Adaptability

When testing our dual-transformer model, we also explored its adaptability across different datasets. The goal was to see how well the model could generalize when trained on one dataset and then evaluated on another. This adaptability is crucial in real-world applications, where conditions often change, and models must remain effective.

In our findings, the dual-transformer model demonstrated strong adaptability. It was able to leverage knowledge from one dataset and perform well on another, highlighting its potential for practical applications in software development. GNN models showed some capability for transferring knowledge across datasets, but the performance was not as robust compared to our dual-transformer model.

#Limitations and Future Work

While the dual-transformer model has shown significant improvements over traditional approaches, it is essential to recognize its limitations. The model's complexity may lead to longer training times and require more computational resources. Additionally, as with any machine learning model, it relies heavily on the quality and quantity of the training data.

Future work could focus on optimizing the dual-transformer model to address these limitations. Researchers could explore ways to reduce the computational load while maintaining prediction accuracy. Investigating different architectures and training techniques could also lead to further enhancements in performance.

Moreover, expanding the dataset diversity and incorporating real-world scenarios that mimic various coding environments can help improve the model's robustness and applicability.

#Conclusion

In summary, our study presents a dual-transformer model that effectively combines the structural representation of source code through ASTs with the textual representation of code tokens. This combined approach leads to more accurate execution time predictions, improving on traditional models that either rely solely on textual or structural data.

Our experiments demonstrate the model's superior performance across various datasets, emphasizing the importance of data diversity and adaptability. As this field continues to evolve, the findings from our work pave the way for future research into advanced neural network architectures that can enhance source code analysis in practical settings.

By continuing to refine these models, we can help developers gain deeper insights into their code, ultimately leading to better performance and more efficient software development processes.