Changing Neural Network Perceptions

Artificial Neural Networks are computer systems that learn and make decisions based on data. They are now widely used in many areas, including speech recognition, image processing, and medicine. Despite their effectiveness, these networks are often seen as "black boxes." This means that it is hard for people to understand how they arrive at their conclusions or decisions.

In this article, we introduce a method to change how a neural network perceives specific ideas defined by humans. This method can create imaginary scenarios that help users understand how these networks work and can even help find errors in them. We tested our method using a special dataset and a popular dataset called ImageNet, to see how well it works with different models and how they respond to our changes.

Our goal is to change how a neural network sees certain ideas, which are not clearly stated in the data it processes. This can help us better understand how different ideas impact the predictions made by a model. Our findings show that we can do this with only a small amount of labeled data, without needing to retrain the existing model.

Over the past few decades, neural networks have transformed many fields, leading to impressive advancements, from understanding text and images to improving healthcare and finance. As these models become more common, the challenge remains that they lack clarity about how they reach their outcomes. This has led to the invention of many methods to address this issue. Most of these methods focus on showing which inputs influence the output or offering an alternative model that is easier to understand.

However, these methods still require users to make sense of the explanations. For instance, if a model looks at a picture of a train, it might point to specific pixels to show how it reached its conclusion. But most people think about images using broader ideas, such as the fact that a passenger train has windows.

When humans attempt to understand complex situations, they often think about different possibilities and how changes in those situations lead to different outcomes. This way of thinking can also help us make sense of how neural networks work since it emphasizes the changes made and the results of those changes. For example, a user could question how the model would behave if the train image had a passenger wagon instead of a freight wagon.

To aid our understanding of artificial networks, we want to create these “what if” scenarios based on how specific human-defined ideas affect a model’s predictions. Focusing on human-defined concepts is crucial since it allows us to discuss these scenarios in ways that make sense to users.

There has been work on creating methods to generate scenarios for artificial neural networks. Some approaches allow for creating scenarios based on specific attributes. However, they usually focus on how changes to the input images affect the model's output without looking at what the model perceives from those images.

Additionally, current methods for generating scenarios often require complicated setups, involving training extra models or using particular architectures, such as invertible networks.

In our approach, we address the creation of scenarios at a more straightforward level. We focus on what a model perceives regarding specific human-defined ideas and examine how that perception affects the model's output. The goal is to convince the model that it is seeing a particular idea, like a passenger wagon, without actually changing the input image, and then checking how this affects the model’s output.

By focusing on how we can change what a model perceives concerning human-defined concepts, we offer a clearer way to see how the information in a neural network influences its predictions.

By changing the way a model perceives different concepts, we can let users see how the output of a neural network relies on these concepts. For our method to create scenarios based on perception rather than input content, we must first understand how a model perceives information and how to adjust that perception for a specified concept.

Our method is inspired by findings from neuroscience, where some neurons are thought to represent specific ideas. These specialized neurons, referred to as Concept Cells, are believed to provide clear and stable representations of concepts. The discovery of these neurons has helped us better understand how the human brain-a complex and intricate system-works and how it links different ideas.

Moreover, research in neuroscience shows that we can influence how specific neurons function, which helps reveal their roles. In a similar way, assigning meanings to certain neurons in a neural network can help clarify what information is stored in a model and how it connects different concepts.

Since we generally have access to a neural network's internal workings, we can also alter the outputs of neurons that we've assigned meanings to and see how these changes affect the entire model in various scenarios, allowing us to create imagined scenarios.

We believe that neurons in neural networks can act like concept cells for various human-defined ideas. By identifying which neurons relate to specific human-defined concepts and adjusting their activities, we can modify a neural network's perception regarding those concepts.

This article presents and evaluates a method to create imagined scenarios for artificial neural networks by adjusting how they perceive specific human-defined concepts.

#A Method to Manipulate a Neural Network's Perception

To create imagined scenarios based on a neural network's perception of human-defined ideas, we propose a method with three main steps for each concept of interest:

1. Estimating Sensitivity: Determine how sensitive each neuron is to that idea. This means assessing how well the activation of a neuron differentiates between samples where the idea is present and those where it is not.

2. Selecting Concept Neurons: Based on the sensitivity values of the neurons, choose which ones are regarded as "concept neuron-like."

3. Computing Activation Values: For each concept neuron, calculate two activation values: one representing when the idea is present and another representing when it is absent.

In the first step, we analyze how sensitive each neuron is to the idea in question. We define a function that uses a set of samples where the idea is present and another set where it is absent, producing a value to show how sensitive that specific neuron is to the idea.

Then, in the second step, we determine which neurons to consider as concept neurons. For this, we define a threshold function that indicates whether a neuron's sensitivity value is high enough to qualify as a concept cell.

Finally, for each chosen neuron, we compute two activation values: one for when the idea is present and one for when it is absent.

When a specific sample is fed into the neural network, creating a scenario where a concept is present or absent requires replacing the activation value of each neuron with the respective activation values calculated earlier. This step is referred to as injecting a concept into a neural network model.

To test our method, we utilized a synthetic dataset called the Explainable Abstract Trains Dataset (XTRAINS). This dataset contains representations of trains and includes labels for various visual concepts. Our experiments employed a neural network trained to identify different types of trains.

#Identifying Concept Neuron-like Neurons

Before changing how a neural network perceives human-defined concepts, we need to identify which neurons in the model correspond to those concepts.

Based on our hypothesis that these models contain information related to human-defined concepts, we aim to identify neurons that act as concept cells for ideas relevant to their tasks. To achieve this, we assess how well a neuron's output can separate samples where the idea is present from those where it is not.

While some existing methods can help identify a model's sensitivity to specific concepts, we wish to find neurons that are sensitive to a particular idea, rather than checking the model's overall sensitivity.

We assess neurons based on three different metrics to gauge their suitability as concept cells:

1. Spearman Rank-Order Correlation: A statistical measure indicating how well the neuron's activations correlate with the dataset's labels.

2. Accuracy of a Linear Classifier: Evaluates how accurately a simple classifier can predict the dataset's labels using the neuron's activations.

3. Probability Density Function Intersection: Compares the distributions of the neuron's activations for positive and negative samples of the idea.

By applying these metrics, we can gather evidence that neurons in neural networks can encode relevant concepts.

#Manipulating a Neural Network's Perception

Next, we test whether we can modify a neural network's perception of specific human-defined concepts by adjusting the activations of the identified concept neurons.

To verify our hypothesis, we analyze model outputs for specific samples. For instance, when inputting a train image that includes a passenger car but lacks a reinforced car, we expect the model to classify the image accordingly if it comprehends the concept of a passenger car. If we then modify the relevant concept neurons to indicate the presence of a reinforced car, we anticipate an appropriate shift in the model's output.

We applied our method to the XTRAINS dataset and gathered various sample sets to evaluate how well our approach worked for different concepts. Our results indicated that it is indeed possible to adjust a neural network's perception of certain concepts by altering the activations of specific neurons linked to those concepts.

Overall, the high success rate of our method suggests that changing how neurons perceive specific concepts can lead to expected changes in the model's output.

#Importance of Selected Neurons

We also examined how crucial the selection of concept neurons is for the performance of our method. Specifically, we tested how varying the threshold value impacts results.

The performance of the method relies heavily on the number of concept neurons chosen. If the threshold excludes too many neurons that act as concept cells, we may see lesser performance. Conversely, if we include too many irrelevant neurons, performance may also decline.

In essence, fine-tuning the threshold for selecting neurons is essential to maintain high performance.

#Counterfactual's Cost

So far, the results indicate that it is practical to manipulate a neural network's perception of human-defined concepts. However, we also looked at how much labeled data is needed to execute this method properly.

By comparing results while varying the amount of labeled data used to identify neuron sensitivity, we found that having even a limited number of samples (for both present and absent cases of the concept) allowed us to manipulate the model's perception effectively. However, with insufficient labeled data, identifying relevant neurons becomes challenging, which ultimately leads to lower performance.

#Interpreting Neural Networks

The method we introduced can generate imagined scenarios for neural networks by altering the activation of neurons that act as concept cells. This allows us to understand how different ideas influence the model's outputs.

What if one wants to know how a model links different concepts that do not appear in its output? For instance, we may want to confirm if a model understands that empty trains do not have passenger cars.

To assess whether a model has identified such non-output concepts, we can use additional mapping networks. These small neural networks help determine if a certain human-defined concept has been recognized in a model's activations.

#Correcting a Neural Network's Misunderstandings

When a neural network makes an incorrect prediction, it can be challenging to discern the cause of the error. Often, this arises because the model fails to recognize some details in the input. Our proposed method can help "correct" a model's mistakes by testing whether different concepts affected its outputs.

We can examine all samples where the model provided an incorrect result and analyze whether certain concepts were not perceived as they should have been. Our experiments indicate that for many false negatives where the model failed, injecting the missing concept could lead to correction in a significant number of cases.

#Validation with Real World Data

After testing our method with a synthetic dataset, we turned our attention to real-world data to see if similar results could be achieved. We used the ImageNet dataset and a well-known model called MobileNetV2.

For our validation, we selected a specific dog breed, the Rhodesian ridgeback, and defined the concept of its face by selecting relevant images where the dog's face was visible. When we injected this concept into the model, we found that it could significantly improve the model's classification accuracy for Rhodesian ridgebacks.

Moreover, when we censored images of the dog's face in the dataset, the model's accuracy dropped. However, when we injected the concept after the censorship, it helped restore accuracy, which confirms that the concept is vital for the model's understanding.

#Conclusion

In summary, our method allows us to change how neural networks perceive specific human-defined concepts without retraining or altering the original model. With only a small amount of labeled data, we can manipulate the model’s perception, leading to better understanding and performance.

We hope to further explore this method to identify biases or undesired associations between concepts in the future. This exploration could open new doors for improving artificial intelligence systems and making them more reliable.