Evaluating ChatGPT's Debugging Skills for Deep Learning

ChatGPT is a tool that has changed many areas, including software engineering. It shows promise in helping with tasks like fixing codes and understanding them. However, there is still uncertainty about how well it can fix deep learning programs, as these programs work differently than traditional ones.

Deep learning programs do not have their logic clearly written out in the code, which makes fixing them more challenging. To successfully repair these programs, ChatGPT must understand not only the code structure but also the purpose behind the code. Current methods for finding faults in such programs are not very effective, with performance levels only around 30%.

This study investigates how well ChatGPT can fix deep learning programs by answering three main questions:

1. Can ChatGPT effectively debug deep learning programs?
2. How can its ability to fix issues be enhanced through better Prompts?
3. How can conversations with ChatGPT help in the repair process?

We break down helpful aspects for crafting prompts aimed at fixing deep learning programs and suggest various templates for those prompts. We also summarize what ChatGPT does well and where it struggles, focusing on finding bad code, improving code quality, and handling outdated or deprecated functions.

#The Role of Large Language Models

Large language models (LLMs), such as ChatGPT, have gained attention for their effectiveness in many tasks. They have been used in several studies that analyze the role of LLMs in programming tasks. Findings suggest that these models outperform older methods in understanding and fixing programs.

In one recent study, ChatGPT was tested on a group of Python programs to evaluate its repair abilities. The results showed that it performed comparably to some top methods. The study also revealed that using the right prompts could lead to even better results.

This paper focuses on the strengths and weaknesses of using ChatGPT to debug deep learning programs. We chose ChatGPT for its popularity, effectiveness, and accessibility compared to similar models.

#Why Focus on Deep Learning Programs?

Unlike traditional programs, deep learning programs function by guiding the training of deep neural networks (DNNs), which means their logic isn't always directly apparent in the code. Therefore, to fix a deep learning program, ChatGPT must be able to interpret the code as well as grasp its intended meaning.

This study differs from previous works because it uses programs with more complex dependencies and features. The aim is to examine how well ChatGPT can handle this complexity.

#Research Questions

#RQ1: Can ChatGPT Debug Deep Learning Programs Effectively?

This question seeks to understand how well ChatGPT performs in debugging compared to two state-of-the-art methods. We break down the debugging process into three steps:

• Finding faults: This looks at how many faulty programs ChatGPT can identify.
• Localizing faults: This measures how many faults ChatGPT can pinpoint correctly.
• Repairing faults: This checks how many faults ChatGPT can fix.

By examining these steps, we can better understand ChatGPT's strengths and weaknesses.

#RQ2: How Can We Improve ChatGPT's Repair Performance with Better Prompts?

Recent studies highlight how well-crafted prompts can enhance the responses from LLMs. However, it is still not clear what information is most useful in these prompts. To answer this question, we analyze real questions asked by developers and categorize them. Based on this analysis, we propose improved prompt templates and test their effectiveness.

#RQ3: How Can Conversations with ChatGPT Aid in the Repair Process?

ChatGPT allows for interactive dialogue. However, it is unclear if and how this feature can help in fixing programs. We explore whether providing hints about fault locations can improve the repair performance.

#Study Design

#Benchmark

To assess how well ChatGPT can fix deep learning programs, we use a benchmark formed from buggy programs found on platforms like StackOverflow and Github. Each of these programs contains common issues like incorrect data processing or faulty model setups.

The programs we chose are longer than those usually tested. They contain several functionalities and depend on more libraries, making them closer to real-world applications.

#Comparison Methods

In our study, we compare ChatGPT's performance against two leading tools: AutoTrainer and DeepFD. AutoTrainer focuses on identifying and fixing training issues in deep learning models, while DeepFD specializes in monitoring runtime features to locate faults.

#Setting Up ChatGPT

We run our tests using the latest version of ChatGPT's API. To account for its non-deterministic nature, we make five requests for each program, keeping track of all the inputs and outputs.

#Metrics for Evaluation

For evaluating Fault Detection, a correct identification occurs if a fault is reported on a buggy program. We repeat the detection requests five times and count it as correct if the majority of the replies confirm a fault's existence.

In the case of Fault Localization, the results are deemed correct if the specific faults reported match the expected ones. For repair evaluation, we check if the faults ChatGPT repaired align with the benchmark.

#Findings

#RQ1: Debugging Deep Learning Programs

Using a basic prompt, we tested ChatGPT on finding and fixing faults. The results are shown in a table that details its performance alongside the two main comparison tools.

1. Fault Detection: ChatGPT detected 27 out of 34 buggy programs. The baseline tool, DeepFD, detected all buggy programs.
2. Fault Localization: ChatGPT could correctly localize 23 out of 72 faults, while the top comparison method localized 29 faults.
3. Program Repair: ChatGPT repaired 16 out of 72 faults correctly, showing improvement over 7 repairs made by AutoTrainer.

ChatGPT was particularly good at finding syntax errors and suggesting improvements, even if these improvements were sometimes not the most critical fixes needed.

#RQ2: Improving Repair Performance with Better Prompts

To enhance ChatGPT's performance, we analyzed the types of information developers typically provide when seeking repair help. This included aspects like the symptoms of bugs, tasks intended for the program, and details about the datasets being used.

From our analysis, we created an enhanced prompt template that includes more context about the program. After testing with this template, we found that ChatGPT's detection rate hit a remarkable 34 out of 34 programs. The number of correctly localized faults rose to 50, and the number of repairs improved to 43.

This significant leap in performance shows that providing clearer context and intention in the prompts benefits ChatGPT's ability to aid in debugging.

#RQ3: Using Dialogue in Repairs

We also explored whether using dialogue with ChatGPT could further enhance its repair performance. In our tests, we provided hints about fault locations in successive rounds of dialogue.

Our results showed that with improved prompts and fault location hints, ChatGPT was able to repair a total of 55 faults. However, some faults remained unrepaired due to ChatGPT sometimes ignoring provided information or misunderstanding the program's intent.

#Conclusion

In this study, we examined ChatGPT's ability to debug deep learning programs. We found that:

1. ChatGPT can find and repair faults better when given enhanced prompts.
2. Dialogue can help guide the repair process, but the effectiveness depends on the clarity of the hints provided.
3. There are still areas where ChatGPT's understanding can improve, particularly concerning the intent behind code.

#Contributions

This research sheds light on how ChatGPT can assist developers and highlights the need for better prompt design to maximize its potential in software engineering tasks, especially in debugging complex programs.

#Future Directions

In the future, we aim to further refine the ways ChatGPT engages in dialogue and understand how to better leverage its capabilities for more effective program repair. We also seek to continue exploring variations in prompts to achieve the best results in different programming contexts.