Improving Forensic Investigations with Excalibur

In today's world, forensic investigations face a big problem: dealing with a large amount of digital evidence, especially images. These images can hold significant information that is crucial for solving crimes. Therefore, having a system that can efficiently find and identify relevant images is essential. This article discusses a new approach that combines different types of data, making it easier for investigators to find important images quickly.

#The Challenge of Digital Evidence

White-collar crime is on the rise and can be very costly. For example, in 2021, a Dutch government agency looked into more than 900 cases, recovering over 520 million euros. This situation highlights the need for effective methods for investigating financial crimes.

With the rapid increase in digital devices, the amount of evidence has grown tremendously. Most phones and cameras store countless images, many of which may have crucial details for investigations. However, current systems often just list images without helping users sort through them efficiently. This method is slow, making it hard for investigators to analyze evidence quickly and bring cases to court in a timely manner.

To tackle these issues, there is a need for systems that not only retrieve images but also allow investigators to express their specific needs while searching.

#What is Cross-modal Retrieval?

Cross-modal retrieval allows users to use different types of queries to search for images. For instance, one can use natural language questions or even other images as queries. This flexibility is important because every investigation is unique, and investigators may have different ways of describing what they are looking for.

However, existing systems have limitations when it comes to understanding user queries fully. Sometimes, users have vague explanations of what they seek, making it hard for systems to find the right images. A useful method to improve the search process is called Interactive Learning. This approach lets users give feedback on the images they receive, helping the system learn and adapt to their needs more effectively.

#Introducing Excalibur: A New Retrieval System

To improve image retrieval in the forensic field, we developed a system called Excalibur. This system combines the concept of cross-modal retrieval with interactive learning, allowing investigators to search through big collections of images.

Excalibur works by taking an initial query from users, whether it's text or an image. It then fetches a list of images that match the query as closely as possible based on similarity scores. Once the user gets the initial results, they can provide feedback by marking certain images as relevant or not relevant. The system then uses this feedback to improve future searches, refining the list of images it presents.

#The Use of Image Encoders

At the core of Excalibur is an image encoder, a tool that helps translate images into a format that the computer can understand. For Excalibur, we utilized a model called CLIP, which can analyze both images and text, comparing them and determining how similar they are.

The advantage of using CLIP is that it has already been trained on a vast amount of image and text data. This means it can effectively handle various queries without needing extra training specific to different types of images. Consequently, Excalibur can adapt to new cases quickly.

#Query Processing and Strategies

Excalibur provides two main ways for users to query the image database: using natural language or an actual image.

1. Natural Language Queries: Users can type in their queries using everyday language. This option is particularly helpful when they are unsure about how to describe what they're looking for.

2. Image Queries: Users can upload an image that they believe is similar to what they want to find. This option is often more effective when users have a specific image in mind.

By allowing both types of queries, Excalibur caters to various preferences and needs among investigators.

#Improving Performance Through Feedback

After the initial search, Excalibur enables users to provide relevance feedback. This process means users can indicate which images were helpful and which were not. The system then uses this feedback to adjust its future results based on user preferences.

The method relies on a binary classifier, a tool that sorts images based on the user’s feedback. In Excalibur, this classifier is designed to find patterns in the images that users identified as relevant.

With repeated interactions, the system can effectively improve its performance, helping investigators find relevant images more quickly.

#Testing Our System

To evaluate how well Excalibur works, we conducted several tests using both simulations and real user feedback. Simulations involved having actors, who mimicked real investigators, search through a collection of images based on a specific scene.

We gathered a dataset of images depicting various scenes, from airports to restaurants. The goal was for actors to retrieve images that matched their query.

During these tests, actors marked images as either relevant or irrelevant. By measuring how well the system improved over time with user feedback, we could assess its effectiveness.

Additionally, we conducted a user study involving actual investigators from the field. They were asked to use Excalibur on their datasets to see how well it met their needs.

#Results of Our Testing

The results from both simulations and user studies showed that Excalibur significantly enhanced retrieval performance over time. For example, the accuracy of images retrieved dramatically increased after just a few rounds of user interaction.

The Feedback Mechanism helped the system adapt quickly to the specific needs of investigators.

Furthermore, participants expressed satisfaction with the usability of Excalibur. They found it intuitive and appreciated the ability to search for images using both natural language and other images.

#Practical Applications and Deployment

We plan to make Excalibur available for investigators to use in real-life situations. A demo version will allow users to explore the features before larger-scale deployment.

Since security and privacy are crucial in forensic work, we will implement the system locally rather than in the cloud. This approach will help mitigate risks associated with sensitive data.

As interest in Excalibur grows, we hope to provide training for investigators to help them use the system effectively in their work.

#Conclusion

The challenges posed by the increasing volume of digital evidence in forensic investigations require innovative solutions. Excalibur presents a promising approach to enhancing image retrieval through interactive learning and cross-modal querying.

By allowing users to provide feedback and using both natural language and image-based queries, Excalibur stands out as a tool designed for investigators' unique needs.

The success seen in both simulations and user studies highlights its potential for real-world applications, ultimately aiding investigators in their pursuit of justice.