Efficient Implementation of Spike-by-Spike Neural Networks using Stochastic and Approximative Techniques

• Applicants: Professor Dr. Alberto Garcia-Ortiz; Professor Dr. Klaus Pawelzik
• Subject Area: Electronic Semiconductors, Components and Circuits, Integrated Systems, Sensor Technology, Theoretical Electrical Engineering
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 465087996

The overarching goal of this project is to improve the efficiency of spiking artificial neural networks using hardware and algorithmic approximation techniques. Specifically, the project focuses on the exploitation of the sparseness and robustness of so-called spike-by-spike networks (SbS). Motivated by real nervous systems, spiking neural networks (SNN) offer an alternative to convolutional neural networks (CNNs). The intrinsic advantages of SNNs, such as better parallelization and robustness promise a high potential for improvements. This has led to large research programs in international companies (e.g. Intel and IBM) and to Europe-wide projects. These research efforts try to imitate biological models. The corresponding implementations require hundreds of cores with large and complex circuits. In contrast, SbS offers a compromise between computational requirements and biological realism that preserves essential advantages of biological networks while allowing a much more compact technical implementation. To fully exploit the robustness and efficiency of SbS, dedicated hardware architectures are required. By combining optimized hardware architectures with stochastic and approximate processing approaches, we aim to improve the performance of pulse-based neural networks and their energy consumption by at least one order of magnitude. These developments will lead to special computing units that lie outside the usual design specifications and therefore have to be tested by a realization in an ASIC. We have organized the project in three pillars: A) Design of dedicated hardware architectures for SbS, focusing on robustness aspects and the possibility of using stochastic techniques. B) Analysis and design of hybrid architectures for combining SbS with standard CNN approaches in a symbiotic fashion. C) Evaluation of theoretical approaches with an ASIC implementation and development of a design and evaluation framework accessible to other researchers. This project will have a threefold impact: 1) For electrical engineering, it will provide novel hardware architectures for neural networks with an efficiency approaching that of the human brain. 2) For neuroscience, it will provide a better understanding of the robustness of spiking networks. 3) For machine learning applications, it will add the robustness and sparseness of spiking networks to established deep neural networks in a computationally efficient way. The applicants are specialized in theoretical neuroscience and microelectronics. The rich experience of their groups in multidisciplinary collaboration provides a solid basis for the success of this project.

• DFG Programme: Research Grants