Training neural networks to operate object recognition or self-driving car navigation tasks utilises a large amount of computing power and time. Broad computers with at least hundreds of processors are required to learn these tasks, and training times can vary between weeks and months.
This has to do with the fact that the computations include several operations of data transfer and reception between the memory and the processor, which eats up most of the energy and time according to Yuhan Shi, a professor of electrical and computer engineering at the Jacobs School of Engineering at UC San Diego.
In an attempt to reduce such power loss, a team led by the University of California San Diego has developed a neuro-inspired hardware-software co-design approach that could make neural network training faster and more efficient. The target of this study is to allow computations to be performed directly in the memory unit, doing away with the need for repeated data shuffles. In practice, the problem was tackled from two perspectives: the device and the algorithm.
The hardware component is a super energy efficient type of non-volatile memory, a 521 Kbit subquantum CBRAM (Conductive Bridging RAM) array. This technology enables an energy consumption drop of around 10 to 100 times compared to conventional memories. While the CBRAM is a digital storage device with only two states (0 and 1), Kuzum and her team demonstrated that it can use analog states to emulate biologic synapses in the human brain. In-memory computing for neural network, training can be carried out by means of this so-called synaptic device. Such a high capacity memory allows training without data transfer to an external processor.
Even better performance is provided through spiking neural networks (SNNs) for implementing unsupervised learning in the hardware. In fact, artificial neural networks are series of neurone layers connected to each other. The strength of these connections is determined by what is commonly called “weights”, which are updated in the training phase. In spiking neural networks, only weights tied to spiking neurones are updated hence helping to save a considerable amount of power and time. The research team also applied soft-pruning, an algorithm they developed in order to make training more energy-saving without sacrificing much in terms of accuracy.
In order to test the efficiency of their system, the team trained the network to classify handwritten digits from MNIST database. The results revealed an accuracy of 93% even when 75% of the sample weights were soft pruned. The subsequent energy saving is estimated between two and three orders of magnitude in comparison with the state of art.
Islam, Consultant, Leyton France
Read more on the same subject: