Understanding neuromorphic computing: from basic principles to experimental verification

The human brain has long been an inspiration to researchers because it supports our computing power in part with efficient bioenergy and uses neurons as the basic firing unit. Neuromorphic chips, inspired by the low power and fast computing characteristics of the human brain, are not a new topic in the computing world. Due to the rapid development of complex algorithms and architectures, heat dissipation has become a major challenge. Neuromorphic computing may be the cornerstone of the future of hyperscale machines and artificial intelligence applications (such as autonomous driving).

"Somehow, the human brain—our own biological body—has figured out how to make it a million times faster than a traditional supercomputer at delivering AI operations." Neuromorphic is an opportunity to come up with a CMOS-based architecture that can mimic the human brain, benefit from the human brain model, and remain energy efficient and cost-effective. —Mark Seager, Intel Fellow, CTO of Scalable Data Center HPC Ecosystem

The original idea of ​​neuromorphic chips can be traced back to a paper published by Professor Carver Mead of Caltech in 1990. In the paper, Mead proposed that analog chips can mimic the activity of neurons and synapses in the human brain. Unlike the binary nature of analog chips, analog chips are chips whose outputs can change. The significance of mimicking human brain activity is that we can learn from them. Traditional chips maintain a fixed specific voltage in each transmission. As Mead mentioned in the 2013 dialogue, heat dissipation has become the biggest challenge in the chip industry when faced with complex algorithms and architectures used in today's machine learning tasks.

In contrast, neuromorphic chips require only a low level of energy consumption due to their biological nature. One reason why the human brain is so energy efficient is that nerve impulses only release a small amount of electricity during transmission. Only when the accumulated power exceeds a set limit will the signal pass through. This means that neuromorphic chips are event-driven and only operate when needed, which leads to a better operating environment and lower energy consumption.

Several companies have invested in research into brain-inspired computing. Wireless technology company Qualcomm demonstrated a fascinating robot based on a neuromorphic chip in 2014. The robot was able to perform tasks that would normally require a specially programmed computer using modified software on a smartphone chip. IBM's 2014 SyNAPSE chip was also built using a brain-inspired computing architecture and has incredibly low power consumption, only 70mW in real-time operation. Recently, neuromorphic has again attracted the interest of companies such as IBM and Intel. Unlike the previous attempts to create commercial products in 2013 and 2014, this time they hope to explore it for research purposes.

In 2012, Intel proposed a design example of a spin-CMOS Hybrid ANN similar to a biological neural network as one of the first prototypes. In this design, neuron magnets constitute the triggering site. The magnetic tunnel junction (MTJ) is similar to the cell body of the neuron, and the domain wall magnets (DWM) are similar to the synapse. The spin potential energy in the central area of ​​the channel is equal to the electric potential energy of the cell body that controls the activation/inactivation state. The detection and transmission unit of CMOS can be compared to the synapse that transmits electrical signals to the receiving neuron (as shown in Figure 1).

In addition to the advantage of low power consumption, neuromorphic devices are also good at tasks that require pattern matching in addition to supercomputing, such as autonomous driving and real-time sensor-fed neural networks. In other words, applications that need to simulate human brain thinking or "cognitive computing" rather than simply more powerful complex calculations. As Mark Seager suggested, the development of neuromorphic should focus on architectures with a large number of floating-point vector units and higher parallelism, and can handle highly hierarchical memories in a fairly uniform way. More specifically, with regard to neural networks, the focus of research is on how to parallelize machine learning tasks through interconnects, such as OmniPath developed by Intel, to solve larger and more complex machine learning problems, thereby scaling on multiple nodes. Currently, scalability is limited to tens to hundreds of nodes, which limits the potential of neuromorphic chips. However, one thing is reasonable, that is, with the advancement of computational neural network algorithms and models, scalability can be greatly increased, which will allow neuromorphic chips to have more room for improvement.

However, we must acknowledge that, despite being a promising direction for the future of computing, neuromorphic technologies are still theoretical and have not yet been mass-produced. There are several devices that are said to have elements of neuromorphic chips, such as the noise suppressor produced by Audience, but they have not yet succumbed to the current large-scale stimulation required to obtain their performance evaluation. Ongoing research has demonstrated progress in overcoming the difficulties encountered in realizing neuromorphic chips and promises a bright future for neuromorphic computing.

experiment

“This architecture can solve a wide range of problems from vision and sound to multi-scene fusion, and has the potential to revolutionize the computing industry by integrating brain-like performance into devices where computing is limited by power and speed.” —Dharmendra Modha, IBM Fellow

The goal of neuromorphic is to use neuroscience as inspiration for algorithms to abstract key ideas to guide the future development of neuromorphic computing architectures. However, converting our biological structures into electrical devices such as oscillators and semiconductors is not an easy task.

To reap the benefits of neuromorphic chips, a large number of oscillators are needed to emulate them. Today’s deep neural networks already have millions of nodes, not to mention the ongoing push toward more complex neural networks with even more nodes. To reach capabilities comparable to the human brain, billions of oscillators are needed. Using software to stimulate such a huge neural network is extremely energy-intensive, but it would be much better to handle it with hardware. In order to arrange all the nodes on a chip the size of a fingertip, nanoscale oscillators are essential.

That’s a problem, because nanoscale oscillators are very susceptible to noise. Such oscillators change behavior under thermal perturbations, and their characteristics can drift over time. Neuromorphic computing doesn’t do a very good job of dealing with noise in processing circuits, although it can tolerate unreliable inputs. Take classification tasks, for example, where a neural network needs to classify the same thing every time when presented with similar inputs. Because of noise, there have only been theoretical approaches to implementing neuromorphic chips with nanoscale oscillators, but no empirical implementations. However, a recent paper proposes a solution that overcomes this difficulty and successfully simulates the oscillatory behavior of an ensemble of neurons using specialized nanomagnetic oscillators.

Researchers have found that under specific dynamic conditions, the use of spin torque oscillators can achieve the best classification results with high signal-to-noise ratio. As shown in Figure 2, the spin oscillator consists of two magnets and a normal spacer component sandwiched in the middle, which has exactly the same structure as the current magnetic memory unit. As shown in the figure above, the magnetization oscillations generated by the charging current are converted into voltage oscillations. Later experiments on speech digit recognition proved that the spin torque oscillator can achieve the current best performance on neuromorphic tasks.

A relatively simple waveform recognition task was used to investigate the role of spin oscillators in pattern recognition. Each sine or square wave was marked with 8 discrete red dots, and the task required distinguishing between sine and square waves at the red dots. Figure 3b shows that many nonlinear neurons are needed to create the path depicted in blue in the spatial neural network. As shown in Figure 3c, the path can also be defined in time, for example by the nonlinear trajectory of each oscillator's amplitude. Each input will trigger a specific path of oscillator amplitudes, and if the time step is set as a fraction of the oscillator's relaxation time, this will generate a transient dynamic state. This means that, in contrast to a traditional neural network where neurons are spatially separated, a single oscillator acts as a group of virtual neurons connected in time. This function creates a memory pool about past events and allows the oscillator to respond differently to the same input if the previous input was different. Due to the finite relaxation time of the oscillator, the perfect separation of sine and square waves is also possible.

Simulating the iterative training of neural networks on hardware can also compensate for anomalies in the processing. As mentioned above, in the case of simulated hardware, distortions can play an important role in the dynamics. Controlling these anomalies is important because the performance of the network depends fundamentally on training with accurate parameters.

The compensation provided by online training was demonstrated using a spiking network on a BrainScaleS wafer-scale neuron system converted from a software-trained deep neural network. A training cycle was then performed, with activity recorded after each training phase. The network activity was first recorded in hardware and processed using a back-propagation algorithm to update the parameters. The researchers found that the parameter updates did not have to be exact during the training step, but only needed to roughly follow the correct gradient trend. Therefore, the calculation of updates in this model could be simplified. Despite the inherent variations in the simulated substrate, this approach allowed for rapid learning, reaching accuracy close to that of an ideal software-emulated prototype in just a few dozen iterations.

Neuromorphic hardware implementations often face another major challenge in terms of system accuracy. The limited precision of synaptic weights reduces system accuracy, which has hindered the widespread application of neuromorphic systems.

Nanoscale oscillators should achieve continuous analog resistance, but only a few stable resistance states can be achieved in practical devices. A recent work proposed three correction methods to learn synapses with one-level precision:

1. Distribution-aware quantization (DQ) discretizes the weights in different layers to different values. This method is based on the observation of the weights of different layers of the network.
2. Quantization regularization (QR) directly learns the discrete weights of the network during training. Regularization can reduce the distance between a weight and its nearest quantization level with a fixed gradient.
3. Dynamic bias tuning (BT) can learn optimal bias compensation to minimize the impact on quantization. This can also mitigate the impact of synaptic changes in memristors in neuromorphic systems.

These three methods enable the model to achieve image classification accuracy comparable to the current state-of-the-art. Experiments were conducted on the MNIST dataset and CIFAR-10 dataset using multi-layer perceptron and convolutional neural network respectively.

The results in Table 2 show that when only one of the three accuracy improvement methods is used, the accuracy is greatly improved compared to the baseline accuracy (1.52%, 1.26% and 0.4% respectively). When two or three methods are used together, the accuracy is even higher and close to the ideal value. The same observation is made when using convolutional neural networks. Some combinations, such as QR+BT, do not improve the accuracy compared to using only QR (as shown in Table 2). This is probably because MNIST is a relatively simple database and the accuracy improvement of the three methods has quickly reached saturation. In the multi-layer perceptron and convolutional neural network, the accuracy drop is controlled to 0.19% (MNIST dataset) and 5.53% (CIFRAR-10 dataset), which is significantly lower than the accuracy drop of the system without using these three methods.

in conclusion

As machine learning algorithms and models advance, novel architectures will become increasingly needed. Neuromorphic devices have great potential in artificial intelligence and cognitive computing applications due to their low power consumption and highly parallelized fast computing speeds. Although current neuromorphic chips are still at the theoretical level, they are moving towards practical applications and marketable products, and researchers have demonstrated some promising research. This is a direction for the future, a direction with the potential to greatly revolutionize the computing world.

“I’ve been thinking about how people create massively parallel systems, and the only examples we have are animal brains. We’ve built a lot of systems. We’ve made retinas, cochleas—a lot of things that work. But creating massively parallel systems is a much bigger task than I’ve ever thought about.” —Marver Mead

[[195949]]

References:

• https://web.stanford.edu/group/brainsinsilicon/documents/MeadNeuroMorphElectro.pdf (Stanford Neuromorphic Electron System Paper)
• https://www.nextplatform.com/2017/02/11/intel-gets-serious-neuromorphic-cognitive-computing-future/ (Intel's next-generation platform for neuromorphic and cognitive computing)
• http://news.mit.edu/2011/brain-chip-1115 (MIT chip that mimics the human brain)
• https://www.youtube.com/watch?v=cBJnVW42qL8 (Matt Grob: Brain-Inspired Computing, Presented by Qualcomm)
• https://www.youtube.com/watch?v=_YQTp3jRMIs (Neuromorphic - From Machines to Life, TED Talk)
• https://arxiv.org/abs/1206.3227 (Neuromorphic Hardware Using Spin Devices)
• https://arxiv.org/abs/1703.01909 (Training Deep Networks on the BrainScaleS Wafer-Scale System)
• https://arxiv.org/abs/1701.01791 (classification tasks on neuromorphic systems)
• https://arxiv.org/abs/1701.07715 (Neuromorphic computing on nanoscale spin oscillators)
• https://www.technologyreview.com/s/526506/neuromorphic-chips/ (Neuromorphic chips)
• https://science.energy.gov/~/media/bes/pdf/reports/2016/NCFMtSA_rpt.pdf (Neuromorphic computing - from materials to system architecture)