Dynamic Few-Shot Learning for Enhanced KGQA

Large language models open new possibilities for answering questions using Knowledge Graphs (KGs). These models aren’t always ideal for generating queries, leading to efforts that improve their performance. While some methods have yielded positive results through fine-tuning or custom setups, they still struggle when dealing with different types of data, especially when the data is not similar to what they were trained on.

This article discusses a new method called Dynamic Few-Shot Learning (DFSL). This approach uses examples from previous questions to help generate better answers, showing improved performance compared to existing methods.

With the growth of the Semantic Web, a lot of structured knowledge has been created and stored in large KGs like Wikidata, DBpedia, and FreeBase. These KGs organize significant amounts of information, making it harder to retrieve relevant data as they expand. Knowledge Graph Question Answering (KGQA) helps to respond to natural language questions posed over KGs. A typical KGQA system has three main parts: Entity Linking (EL), Relation Linking (RL), and Query Generation (QG).

Starting from a natural language question, EL and RL identify entities and relations in the question. The QG part then uses this data to create a SPARQL query that fetches the answer. This paper focuses on the QG aspect. Current top methods for generating SPARQL queries often involve adjusting language models like T5 or creating custom models that use dependency trees. Although successful, these methods lack flexibility and scalability, with fine-tuning often being costly and ineffective when facing different types of data.

This paper suggests that better use of existing examples in training sets can reduce errors. DFSL combines semantic search with examples from the training data to improve question answering. To assess how well DFSL works, tests were run on several benchmark datasets using popular knowledge bases like DBpedia and Wikidata.

The investigation included four publicly available datasets: QALD-9 (based on DBpedia), QALD-9 Plus, QALD-10, and LC-QuAD 2.0 (based on Wikidata). The model was tested using three advanced language models: Mixtral 8x7B, Llama-3 70B, and CodeLlama 70B. The results showed that DFSL reaches new performance heights in speed and efficiency.

A typical KGQA system requires identifying relevant entities and relations from a question before generating a SPARQL query. The study introduced DFSL as a method combining semantic search to dynamically pull similar examples from the training data, improving the prompt used by language models-helping them craft better queries.

Previous research in KG query generation has often relied on methods that involve creating rules or templates. These older techniques often struggle to keep up with the larger KGs as they grow, leading to calls for advancements that could better tackle these challenges. Recent developments have been split between methods focused on information retrieval and those that translate text into SPARQL queries directly.

In information retrieval-based methods, the aim is to find sub-graphs relevant to a given question. Some techniques involve more complex methods, including relation prediction and structural modeling. The rise of modern language models like GPT has also enabled new ways to translate natural language into SPARQL queries. However, many existing methods tend to produce systematic errors, such as flipping the subject and object in generated queries, which can lead to incorrect answers.

DFSL takes advantage of in-context learning (ICL), where reasoning through examples is a key feature. This method allows existing examples to enhance the query generation process. The idea is to pull examples that closely match the current question to improve performance. The retrieval process is based on the similarity of a question to examples stored in the training data.

DFSL dynamically chooses demonstrations to improve the prompt. The prompt includes three parts: a description of the task, retrieved examples, and the input question with associated entities and relations. After the language model generates a query, it is executed to fetch results.

A challenge within SPARQL query generation includes determining subjects and objects in relationships-a process LLMs may not handle efficiently. DFSL offers a solution by allowing the generation of multiple SPARQL queries, retaining all hypotheses generated, increasing the chances of producing a correct and relevant query.

When executing multiple queries, different answers may arise, necessitating a method to choose the best one. Two methods were designed for answer selection: Largest Set (LS) and First Set (FS). The LS method chooses the largest answer set among all generated answers, while FS simply picks the first query that returns an answer.

In experiments, DFSL showed improved performance compared to previous methods across multiple benchmark datasets. This included an evaluation of how many examples should be retrieved, with findings suggesting a larger number of examples tends to yield better results.

The study also looked into how DFSL performed against other approaches. DFSL consistently outperformed zero-shot learning methods and even many few-shot learning models. However, results varied somewhat in different datasets, especially with lower performance noted when training data did not align well with testing data.

The analysis of DFSL compared to other state-of-the-art methods indicated that DFSL operates effectively even without additional training on specific tasks. Although it showed strong performance generally, there remains room for improvement in handling cases where only partial information is available, reflecting a still-present challenge in the field.

In summary, DFSL is a new method for Knowledge Graph Question Answering that enhances the language model's ability to generate SPARQL queries by making use of relevant examples dynamically. It provides improved performance on various benchmarks without requiring extensive fine-tuning. Future work will examine its application in similar areas like text-to-SQL and aim to explore the method's applicability beyond English datasets.

The code for implementing DFSL will be shared publicly following the paper's acceptance, allowing for broader use and exploration of its capabilities. By addressing current limitations and investigating further variations, this approach could lead to better understanding and handling of knowledge graphs in answering complex queries.