Improving Vision Language Models with Directional Guidance

In today’s world, we often need help answering questions using pictures. Imagine a visually impaired person attempting to take a picture for their question but not getting it quite right. Wouldn’t it be helpful if a computer could tell them how to adjust their photo to get the answer they need? This is where Vision Language Models (VLMs) enter the scene. They are computer programs designed to understand both images and language, but they are not perfect yet.

While humans can think about whether they have enough information to answer a question, VLMs generally just give quick answers. This study looks to see if we can make VLMs better by teaching them how to say, “Hey, you might need to change the angle of that picture," instead of just guessing.

#The Problem with VLMs

When you ask a computer a question with a picture, it should ideally check if the picture has all the needed information. Humans can do this pretty well. If someone asks, "What color is my shirt?" while showing a blurry picture, they can realize that they might need to take another picture. However, VLMs sometimes just provide a single answer without checking if the image has the right view.

So, how do we tackle this? We need to make VLMs think more like humans. They should be able to say something like, “I can’t see your shirt well enough to tell you the color. You might want to move the camera left.”

#Setting Up a New Task

To bridge this gap, we created a new challenge for VLMs called Directional Guidance. The idea is simple: When a VLM is faced with a question and an image, it should recognize if the image is good enough to answer the question. If not, it should provide advice on how to improve the image.

Think of it like giving someone directions to take better selfies. If they are holding the camera too close, you might tell them to step back. If they need to show more of the scene, you could say, "Take the picture to the left!"

#Getting Feedback from Real People

To test how well VLMs can give Directional Guidance, we created a Benchmark Dataset with images and questions. Our research team gathered a bunch of real-world images from the VizWiz dataset, which includes questions asked by visually impaired individuals. We had a team of human annotators check these images and provide advice on framing-like where to move the camera or if the picture needed to be retaken.

Using this helpful input, we gathered examples where moving the camera would help reveal answers and also examples where no amount of moving would change things.

#Training the VLMs

To teach VLMs how to give Directional Guidance, we needed to create training data. Instead of just asking models to make correct guesses based on available images, we played around with the images to make them harder.

If an image had enough clear information, we might crop out some parts to make it seem less complete. For instance, if the original image showed a bright blue sky and a tree, we would clip a portion of the sky to create confusion. This way, the models could practice improving images, rather than just guessing answers blindly.

#What We Found

When we put our new method to the test, we checked how well several popular VLMs performed in the Directional Guidance task. To our delight, we found that VLMs showed real improvement when trained with our synthetic data. The models could not only answer questions better after fine-tuning but also gave more accurate guidance on how to adjust camera angles.

Essentially, when VLMs learned from the right examples, they became more like helpful friends who provide thoughtful tips instead of just shouting random answers.

#Understanding Self-Knowledge in VLMs

Part of teaching VLMs is helping them gain a sense of self-knowledge. This means they should know what they can and can't see. Humans are aware of when they don't have enough information to make a smart guess, and VLMs need this awareness too.

When faced with an unclear image or an ambiguous question, VLMs should be able to admit, "I can't answer that right now." Then, they could suggest actions to take, like "Try taking a picture from a different angle."

#The Cognitive Process

To explain how VLMs can improve, think of a process similar to how humans learn and solve problems:

1. Getting Information: VLMs look at an image and see what they can figure out from it, just like we do when asked to recall known facts.
2. Recognizing Gaps: They should also see when they don’t have enough info to answer a question-like when a person realizes they can’t see their friend clearly in a crowd.
3. Seeking Answers: Finally, they should learn to suggest where to go next for new information, similar to how humans might search online or ask someone for help.

#Expanding the Training Framework

Our training framework focuses on mimicking this cognitive process. In the Directional Guidance task, VLMs must learn when and how to suggest reframing an image.

We created a user-friendly classification system where VLMs can pick from a list of direction options: keep the image the same, move left, move right, up, or down. There’s also an option for when no adjustment will help.

#Real-World Examples

To see how well our VLMs performed, we included examples from our benchmark dataset in the training phase. Some models were able to determine direction quite accurately, while others had trouble with certain categories.

Even with these hiccups, we saw progress. When the models were fine-tuned, they provided better directional advice, proving our framework's effectiveness.

#Moving Forward

While our focus was on guiding reframing directions, we recognize that there are additional aspects of taking better pictures we could explore. What if VLMs could also help with exposure or focus adjustments? Our automated training framework could easily adapt to cover these other areas in the future.

Fine-tuning to accommodate complexities like needing to move up and left simultaneously will also be a topic worth investigating. The aim is to provide richer guidance, making the experience as smooth as possible for users.

#Conclusion

The Directional Guidance task offers an exciting new approach to enhancing VLMs, especially for assisting visually impaired users. With clever adjustments and thoughtful training, VLMs can become better at understanding the limits of their visual information and improving their responses.

As we aim for a world where technology can smoothly assist and empower people, developing models that think more like humans brings us one step closer. With ongoing improvements, VLMs have the potential to become indispensable tools for answering questions effectively.

Let’s keep pushing boundaries and creating systems that make our lives just a little bit easier-even if it means telling someone to move left or right for that perfect snapshot!