Optimizing Weight Packing for In-Memory Computing in Neural Networks

In-Memory Computing (IMC) hardware accelerators have shown to greatly improve the efficiency and performance of tasks like matrix-vector multiplications, which are crucial for running Neural Networks. Neural networks are used in many applications, such as voice recognition, image processing, and more. However, to get the most benefits from IMC, it's important to make sure that the resources are used effectively and to reduce the Energy Costs that come from loading weights into memory.

#The Challenge of Neural Network Workloads

Neural networks that run on edge devices, like smartphones or smart cameras, often have limited computing and memory resources. Traditional processors are often not powerful enough for the complex tasks needed by modern artificial intelligence models. This is especially true for tasks that involve matrix-vector multiplications.

A significant issue in using IMC for deep neural networks (DNN) is the overhead caused by loading weights into memory. Each time weights are loaded, it takes extra energy and time, which affects overall performance. The goal is to reduce this overhead while maximizing the stability of operations by efficiently packing weights in the IMC.

#Advantages of In-Memory Computing

IMC has several features that make it suitable for hardware acceleration. First, it allows for many matrix-vector multiplications to happen at the same time because of its memory structure. Second, it enables efficient movement of data since the same operands can be used repeatedly across different operations. This setup helps reduce the time and energy spent on fetching data from memory.

Despite these advantages, real workloads often reveal two main problems for IMC systems: underutilization of computational resources and the overhead from loading weights. The way weights and data are stored in memory affects these issues. By arranging the data wisely in the IMC, we can lessen both problems.

#Need for Optimized Data Mapping

To make the most of IMC's potential, a good approach is needed for how weights are arranged in the memory. This method should not only aim to improve memory usage but also enhance computing efficiency without sacrificing performance. Currently, there is no ideal way to arrange data that maximizes both aspects.

#The Weight Packing Algorithm

To address the challenges of loading weights without losing computing power, a weight packing algorithm has been developed. The aim is to arrange weights tightly within the IMC memory while running a neural network. The overall objective is to minimize energy and delay when using the network for inference.

The efficiency of this system heavily relies on how well it can capitalize on available space. More spatial reuse of input and output data leads to lower energy costs for moving data and using peripheral elements.

#Steps of the Algorithm

1. Weight Tile Pool Generation: The first step is to create a pool of weight tiles based on the dimensions of the IMC. These tiles are uniform and are defined for each layer of the neural network.
2. SuperTile Generation: SuperTiles combine several weight tiles and maximize spatial parallelism by ensuring different layers are stacked without losing efficiency.
3. Column Generation: This phase focuses on finding the best allocation of these SuperTiles in the IMC to maximize memory usage while keeping compute efficiency high.
4. Column Allocation: Finally, the columns of SuperTiles created are allocated across the available space in the IMC.

#Examining the Results

The proposed weight packing method has been integrated into a system and tested under different scenarios. The results show that this new approach offers several benefits compared to traditional methods.

In these tests, the packed method outperformed previous techniques, especially for networks with small weight tensors. However, the packing process can sometimes increase computation time because of the folding operations involved, which convert spatial computations into sequential ones.

#Energy and Delay Trade-offs

Analyzing energy and delay trade-offs is crucial. The tests demonstrated that the loading of weights from external memory greatly hampers the performance of IMC accelerators. While storing activation data internally reduces some of these issues, loading weights from external sources still presents significant challenges.

Increasing the number of processing units can help improve efficiency but does not eliminate the bottlenecks associated with loading weights. The weight packing method provides a solution that allows most weights to remain inside the IMC, significantly reducing the need to fetch from external memory.

#Conclusion

This work presents a method for effectively packing weights for neural networks in IMC systems, addressing the challenges of weight loading and Computational Efficiency. The new approach not only minimizes energy and delay costs but also enhances the overall performance of neural network workloads on edge devices. By utilizing a systematic way to organize data, the method demonstrates significant improvements in performance and energy efficiency compared to traditional mapping techniques, making it a promising solution for future applications in edge AI systems.