Challenges and Insights on Small Language Models for Coding

Large Language Models (LLMs) like GPT-4 and LLaMA-405b have made waves in the world of coding. They can help write code, finish code, and even create test cases for software. However, making these models bigger also makes them more power-hungry and expensive to run. This isn’t just a problem for giant tech companies; small businesses and academics may struggle to keep up. One way to manage the size of these models is through Quantization, which helps reduce the memory they need. But, here's the catch: it might also make the code they produce less reliable.

#The Problem with Model Size

When you hear about LLMs, think of them as giant brains with billions of parameters. The bigger the brain, the better it can think, right? But just like a brain needs food, these models need a ton of power to work. They leave behind a significant carbon footprint, which isn't great for the environment. For many academic researchers and small companies, using these large models is like trying to run a marathon with a pebble in your shoe-it's just not practical.

#Can Smaller Models Help?

This is where smaller LLMs come in. They might not be as talented as the big guys, but they can still help with coding tasks. However, there’s a catch: while they are cheaper and easier on resources, their work might not be up to par. They may produce code that doesn’t work at all or is riddled with errors.

#Evaluating Code Generation

To see how well these smaller models perform, we conducted a study using two benchmarks called HumanEval Plus and MBPP Plus. Think of these benchmarks as test papers for our models. Each model takes a coding prompt, writes some code, and we check how many of its answers are correct.

We looked at four smaller models and ran them through different tests. The tests focused on two main areas: how well the models could generate code and whether quantization affected the quality of their output.

#What Did We Find?

Well, the results weren’t all that promising. While the smaller models showed some potential, they frequently produced code that didn’t pass basic tests. When we quantified their performance, the results were less than stellar. Mistral Instruct 7B seemed to do the best but was still struggling with simpler tasks. Meanwhile, the other models generally flopped.

For instance, Mistral Instruct managed to pass 25% of the tests on the HumanEval Plus benchmark, which sounds decent until you realize that it passed none on the MBPP Plus. That's a bit like being a star student in advanced math but failing gym class!

In terms of structure, the code these models produced wasn’t all that bad. They often generated code that looked fine on the outside but didn’t work well. Imagine a beautifully wrapped gift that turns out to be an empty box inside.

#The Role of Quantization

Next, we tried quantizing the models, which is just a fancy way of saying we shrunk their brain size. Some models, like Mistral Instruct and CodeLlama, did a bit better with quantization, especially on easier tasks. Think of it as making a model wear glasses to help it see better. But for models like WizardCoder 3B, quantization was more like shouting at someone to make them hear better-it didn't really help.

#Quality Issues in Generated Code

We wanted to dive deeper into the quality of the code produced by these models. So, we took a closer look using a tool called SonarQube. This is like a health checkup for code; it tells you what’s wrong with it. What we found was a little alarming: over 80% of the problems we detected had to do with maintainability!

Most of the issues involved Code Smells-which is a funny way of saying there are signs of bad code practices. These included things like messy formatting, inconsistent naming, and overly complicated structures. Many of the models even forgot to include necessary bits of code like import statements, which is a bit like baking a cake without flour.

#Manual Inspection of Code

A manual examination of the generated code revealed more issues. For example, Mistral Instruct consistently overlooked important import statements in the code it generated. CodeLlama, on the other hand, was a repeat offender, generating duplicate sections of code. So, if you asked it to solve a problem, it might give you an answer along with ten variations of it, wasting your time and space.

#Conclusion and Future Directions

In the end, our study highlighted the strengths and weaknesses of these smaller models for code generation. While they can produce some decent code, they also produce a lot of junk that needs fixing. And if you’re hoping to use these models in real-world situations, prepare to roll up your sleeves and do some serious cleaning up afterward.

The findings also call for better evaluation methods that don’t just count how many tests the models pass but also consider the code’s long-term maintainability. More research is needed into improving the training processes for these models, especially as they become more common in everyday software development tasks.

To wrap it all up, smaller LLMs have a long way to go. They may be easier to use but are often less reliable, meaning that developers should be ready to put in extra work to clean up the code they generate. It’s always great to have help from this new technology, but don’t forget to check under the hood!