Comparing Traditional and LLM-Based Search Engines for Image Geolocation

Web search engines have been crucial in helping people find information online. People have developed specific ways to ask questions and look for what they need. Recently, search engines powered by large language models (LLMs) have started to offer a more conversational way to search and different ways to ask questions.

This work looks at how traditional search engines compare to LLM-based search engines for the task of figuring out where a photo was taken, known as image geolocation. The study focuses on how users interact with these search engines and their different ways of asking questions. A group of 60 participants was divided into two groups, one using a traditional search engine and the other using an LLM-based search engine. The results showed that those using the traditional search engine were better at guessing the image locations.

#Background

For many years, web search engines have been the go-to tools for various information-seeking tasks. Users have learned to optimize their searches based on keywords rather than natural language. The rise of artificial intelligence and language models, such as BERT and GPT-3, has led to new search methods that allow for more natural interaction.

These newer systems allow users to have more conversational exchanges with the search engine. This shift could improve how users find information, but there are still questions about how well people adapt to this change.

In this study, we focused on the task of image geolocation, which is important in fields like law enforcement and journalism. Historically, this task required expert analysis and high-quality reference tools. While fully automated systems have been developed, they often rely on well-defined visual clues. This task is inherently challenging because users must not only find visual clues but also know how to turn them into effective search queries.

#Research Questions

This study aimed to answer several key questions:

1. How do the querying strategies differ between LLM-based search and traditional search for image geolocation tasks?
2. What Challenges do users face when using LLM-based search engines compared to traditional search engines?
3. How do users adapt their query formulation strategies when using these different search tools?

#Methodology

To investigate these questions, we conducted a study with 60 participants, each assigned randomly to use either a traditional search engine or an LLM-based search engine for image geolocation tasks. The experiment involved six rounds of guessing the location of different images, which varied in difficulty.

Participants used a dual-screen setup, with the search engine on one screen and the geolocation task on the other. They were instructed to use only the provided search tool and were given two minutes for each round to make their guesses. After completing the tasks, participants filled out a survey regarding their experience.

#Results

#Performance Comparison

The results indicated that participants using the traditional search engine performed significantly better than those using the LLM-based search engine. On average, those in the traditional group scored higher points based on the accuracy of their guesses. The traditional search users issued more queries, while LLM users preferred longer, more natural language questions.

#Query Formulation Strategies

The ways participants formed their queries differed between the two groups. Participants using the traditional search engine tended to issue shorter, more focused queries utilizing specific terms. In contrast, LLM-based search users formulated longer queries and made more conversational requests.

Furthermore, when reformulating their queries, traditional search users often added more terms to their initial queries. In contrast, those using LLMs preferred to rephrase their questions without adding much new content.

#Challenges Identified

Participants also reported facing challenges with the LLM-based search engine. Many found it difficult to effectively communicate their needs, leading to frustration. Several participants mentioned issues with language, particularly when they used terms in languages other than English.

#Strategies for Finding Clues

Participants utilized different strategies for interpreting cues from the images. Common strategies included identifying street names, locating businesses, and describing geographic features. Although the specific strategies did not vary significantly between groups, how participants translated these clues into queries differed.

Those using the traditional search engine focused on keyword-based strategies, while LLM users tended to seek conversational guidance, which sometimes did not yield effective results.

#Discussion

#Differences in User Interaction

The study highlights distinct differences in how users interacted with LLM-based and traditional search engines. Traditional search users were more focused on simple and direct queries, while LLM users adopted a more conversational tone. This difference in approach may account for the performance disparities observed.

Additionally, the feedback from participants indicated that many struggled to adapt to the LLM's capabilities. Some were unclear on how to formulate effective queries, particularly as they faced tasks that required specific knowledge.

#Implications for Design and Use

Understanding the differences in user behavior has important implications for designing LLM interfaces. As these technologies become more prevalent, it is crucial to ensure users can effectively communicate their needs. Reducing barriers to effective prompting will enhance user experience and outcomes.

The lack of effective results from LLM searches, especially when users sought specific details like maps or clear location information, underscores the need for better guidance in how to use these tools. In many cases, users were left to figure out how to ask effective questions without adequate support.

#Conclusion

This study sheds light on the differences between traditional and LLM-based search engines in the context of image geolocation. Participants using traditional search tools performed better and articulated their queries in ways that aligned with expected keyword-based searches. In contrast, LLM-based search users faced challenges in accurately formulating their questions, which affected their performance.

Future work should focus on improving LLM interfaces and enhancing the overall user experience. This includes understanding how users form mental models of these systems and how to guide them in crafting effective queries. Equipping users with the tools they need to prompt LLMs effectively will be essential as these technologies continue to evolve.

By addressing the challenges identified in this study, we can work towards developing more user-friendly search technologies that meet the needs of diverse users. Enhanced LLM interfaces that combine conversational elements with structured prompts may provide a more effective balance, guiding users toward successful interactions.

#References

(References would be added in a formal publication but are not included here.)