Modeling Mouse Vision: Insights from Deep Learning

Deep Convolutional Neural Networks (CNNs) are tools used in computer science that can model how the brain processes visual information. These networks have structures similar to the visual cortex in animals, which is the part of the brain responsible for sight. Researchers aim to create accurate models of brain function by understanding how different network features can improve these predictions.

Two major aspects to consider are the network's structure and the goals set during training. Networks can be adjusted to reflect properties of brain activity better. Some researchers have modified networks to mimic processes found in neuroscience and to make the responses of these networks more realistic.

The type of visual input given during training also plays an important role. For example, when networks are trained with videos of moving objects, they can better match the activity in specific parts of the brain responsible for motion detection. Also, when the input has details that match the visual abilities of the animal being studied – such as using video resolutions suited for mice rather than humans – the network performs better in mimicking the mouse brain's activity.

In this study, researchers used a model called MouseNet, built to resemble the mouse visual cortex. They trained this model using a self-supervised learning method. The team varied several aspects of the videos used for training and looked at how these variations affected how well the model matched actual brain activity. They created videos showing a mouse's perspective in both natural environments (like meadows) and artificial settings (like spaceships) and studied how these differences impacted model performance.

#Methodology Overview

To carry out this study, researchers created a virtual mouse agent in a game development engine called Unity. This agent had two cameras set up similarly to a mouse's eyes, allowing it to capture a field of view and resolution that matched what a mouse would see. The team recorded videos that the agent produced while navigating various environments, using these recordings to train their deep neural network, MouseNet.

They generated eight different Training Datasets. Each dataset was either naturalistic or non-naturalistic based on three categories: the environment, the optical properties of the mouse's eyes, and how the mouse moved through the environment. The study found that these differences significantly affected how well the model's activity matched that of the mouse brain.

Among the three factors studied, the environment had the biggest and most consistent influence. The most similar brain activity was observed when all three conditions (environment, optics, and motion) were naturalistic. Interestingly, while naturalistic motion generally decreased overall similarity to the brain, it improved similarity for a specific brain area known for processing motion.

The researchers sought to understand what specific differences between natural and non-natural environments contributed to varying similarity in brain activity. They initially thought simpler statistical properties like frequency, color, orientation, and temporal correlations might explain the differences. However, tests using more abstract environments showed that those properties did not fully account for the brain similarity differences observed between the two types of environments.

#Visual Cortex Model

The MouseNet model is designed to reflect the anatomy of the mouse visual cortex closely. It includes several layers that correspond to specific parts of the visual cortex. The design of the model is based on available neuroanatomical data, ensuring that the network's structure is similar to the real visual processing pathways found in mice.

#Training Datasets

Using Unity, the researchers created a mouse agent that could collect video recordings in various environments. They devised two cameras, which resembled mouse eyes, to capture video. The videos were recorded at a resolution that matched mouse vision and at a specific frame rate.

The team produced both natural and non-natural versions of three factors affecting the visual stimuli: the environment, the optics of the mouse's eyes, and the mouse's movement patterns. By experimenting with various combinations, they created eight unique datasets representing different training conditions.

The natural environment included elements such as grass, trees, and rocks, while the artificial setting resembled a spaceship with metallic walls and bright lights. The visual properties of how each environment was represented were critically analyzed to understand their impact on the model's performance.

#Mouse Motion

The researchers developed two distinct motion models for the mouse agent. The artificial model included general forward movements and simple turns, while the naturalistic model was based on the actual movement data of a freely moving mouse. This data was gathered in a controlled environment and provided more realistic representations of mouse motion, allowing the researchers to create diverse and complex trajectories.

The team used a specific modeling approach to generate motion trajectories that resembled how mice typically navigate their environments. By ensuring that the model captured more natural and complex movement patterns, the researchers aimed to enhance the model's ability to simulate actual brain activity.

#Synthetic Environments

To further investigate the observed differences in brain similarity, the researchers created synthetic environments consisting entirely of simple geometric shapes. They aimed to match these synthetic environments to the natural and artificial environments based on key statistical properties.

By utilizing this controlled approach, the researchers were able to systematically analyze how different visual features influenced brain activity similarity. This investigation aimed to clarify whether features like spatial frequency, color distribution, and orientation can help explain why the natural environment performed better in aligning with brain activity.

#Training Procedure

The training of MouseNet involved using a technique known as self-supervised learning. Essentially, the network learns to predict future video frames based on the prior frames it has seen. This method allows the model to develop a deep understanding of the hidden structures within the input data.

The training process involved breaking down the video datasets into smaller clips and organizing these clips into batches for the network to analyze. The model was repeatedly trained on each dataset to enhance its learning and adjust its predictions based on the network's performance.

After extensive training, the researchers compared the network's outputs with actual brain activity data collected from mouse Visual Cortexes. They used techniques to calculate how well the model's activity aligned with the recorded brain data. This comparison aimed to measure the success of the model in mimicking real visual processing in the mouse brain.

#Impact of Stimulus Properties

The results of the training showed that the models performed well, exceeding random chance in prediction accuracy. However, the types of environments and motion patterns used during training affected how accurately the model mirrored actual brain activity.

Conditions with natural environments led to the highest similarity scores compared to artificial ones. This implies that the characteristics of the training stimuli have a significant impact on how well the model aligns with brain activity. The findings showed that not only did the natural environment improve similarity, but also there were nuanced interactions between the various factors that influenced the overall performance.

#Time Course during Training

Throughout the training process, changes in both prediction accuracy and model alignment with brain activity were observed. The models exhibited a decrease in prediction loss over time, indicating improved performance. However, similarities with the brain didn't always follow a clear trend. Some areas showed rapid changes initially, followed by stabilization, while others demonstrated fluctuations throughout the training period.

This led to the understanding that the connection between loss and similarity is intricate, dependent on a variety of factors, including the training conditions. The time course of brain similarity development varied across different cell populations, highlighting the complexity of aligning model outputs with actual brain function.

#Conclusion

In summary, this research investigated the effects of various training conditions on a deep neural network's ability to model mouse visual processing. Through the use of naturalistic and non-naturalistic environments, along with realistic motion statistics, the study revealed that environment played a significant role in improving alignment with mouse brain activity.

The results indicate that creating a deep network with features reflecting naturalistic experiences can enhance its performance and understanding of brain function. Future research aims to delve deeper into the properties of environments that may further optimize brain similarity, allowing for better models of visual processing in the brain. The findings will help shape our understanding of both artificial intelligence and neuroscience, providing insights into how to develop models that better mimic real-world visual processing systems found in animals.