ESSOP: Efficient and scalable stochastic outer product architecture for deep learning

Vinay Joshi; Geethan Karunaratne; Manuel Le Gallo; Irem Boybat; Christophe Piveteau; Abu Sebastian; Bipin Rajendran; Evangelos Eleftheriou

ESSOP: Efficient and scalable stochastic outer product architecture for deep learning

Vinay Joshi^*, Geethan Karunaratne, Manuel Le Gallo, Irem Boybat, Christophe Piveteau, Abu Sebastian, Bipin Rajendran, Evangelos Eleftheriou

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

2 Citations (Scopus)

Abstract

Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14 nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2 % and 93.7 % better in energy and area efficiency respectively for outer product computation.

Original language	English
Title of host publication	2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728133201
Publication status	Published - 2020
Event	52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Virtual, Online Duration: 10 Oct 2020 → 21 Oct 2020

Publication series

Name	Proceedings - IEEE International Symposium on Circuits and Systems
Volume	2020-October
ISSN (Print)	0271-4310

Conference

Conference	52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020
City	Virtual, Online
Period	10/10/2020 → 21/10/2020

Cite this

Joshi, V., Karunaratne, G., Le Gallo, M., Boybat, I., Piveteau, C., Sebastian, A., Rajendran, B., & Eleftheriou, E. (2020). ESSOP: Efficient and scalable stochastic outer product architecture for deep learning. In 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings Article 9180872 (Proceedings - IEEE International Symposium on Circuits and Systems; Vol. 2020-October). Institute of Electrical and Electronics Engineers Inc..

@inbook{1210d99ba8c144f983ed6983d376dfb4,

title = "ESSOP: Efficient and scalable stochastic outer product architecture for deep learning",

abstract = "Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14 nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2 % and 93.7 % better in energy and area efficiency respectively for outer product computation.",

author = "Vinay Joshi and Geethan Karunaratne and {Le Gallo}, Manuel and Irem Boybat and Christophe Piveteau and Abu Sebastian and Bipin Rajendran and Evangelos Eleftheriou",

note = "Funding Information: This project was supported partially by the Semiconductor Research Corporation. Publisher Copyright: {\textcopyright} 2020 IEEE Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020 ; Conference date: 10-10-2020 Through 21-10-2020",

year = "2020",

language = "English",

series = "Proceedings - IEEE International Symposium on Circuits and Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings",

}

Joshi, V, Karunaratne, G, Le Gallo, M, Boybat, I, Piveteau, C, Sebastian, A, Rajendran, B & Eleftheriou, E 2020, ESSOP: Efficient and scalable stochastic outer product architecture for deep learning. in 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings., 9180872, Proceedings - IEEE International Symposium on Circuits and Systems, vol. 2020-October, Institute of Electrical and Electronics Engineers Inc., 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020, Virtual, Online, 10/10/2020.

ESSOP: Efficient and scalable stochastic outer product architecture for deep learning. / Joshi, Vinay; Karunaratne, Geethan; Le Gallo, Manuel et al.
2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. 9180872 (Proceedings - IEEE International Symposium on Circuits and Systems; Vol. 2020-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - ESSOP

T2 - 52nd IEEE International Symposium on Circuits and Systems, ISCAS 2020

AU - Joshi, Vinay

AU - Karunaratne, Geethan

AU - Le Gallo, Manuel

AU - Boybat, Irem

AU - Piveteau, Christophe

AU - Sebastian, Abu

AU - Rajendran, Bipin

AU - Eleftheriou, Evangelos

PY - 2020

Y1 - 2020

N2 - Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14 nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2 % and 93.7 % better in energy and area efficiency respectively for outer product computation.

AB - Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require real-time learning. Matrix-vector multiplications (MVM) and vector-vector outer product (VVOP) are the two most expensive operations associated with training of DNNs. Strategies to improve the efficiency of MVM computation in hardware have been demonstrated with minimal impact on training accuracy. However, the VVOP computation remains a relatively less explored bottleneck even with the aforementioned strategies. Stochastic computing (SC) has been proposed to improve the efficiency of VVOP computation but on relatively shallow networks with bounded activation functions and floating-point (FP) scaling of activation gradients. In this paper, we propose ESSOP, an efficient and scalable stochastic outer product architecture based on the SC paradigm. We introduce efficient techniques to generalize SC for weight update computation in DNNs with the unbounded activation functions (e.g., ReLU), required by many state-of-the-art networks. Our architecture reduces the computational cost by re-using random numbers and replacing certain FP multiplication operations by bit shift scaling. We show that the ResNet-32 network with 33 convolution layers and a fully-connected layer can be trained with ESSOP on the CIFAR-10 dataset to achieve baseline comparable accuracy. Hardware design of ESSOP at 14 nm technology node shows that, compared to a highly pipelined FP16 multiplier design, ESSOP is 82.2 % and 93.7 % better in energy and area efficiency respectively for outer product computation.

UR - http://www.scopus.com/inward/record.url?scp=85109315576&partnerID=8YFLogxK

M3 - Conference paper

AN - SCOPUS:85109315576

T3 - Proceedings - IEEE International Symposium on Circuits and Systems

BT - 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 10 October 2020 through 21 October 2020

ER -

Joshi V, Karunaratne G, Le Gallo M, Boybat I, Piveteau C, Sebastian A et al. ESSOP: Efficient and scalable stochastic outer product architecture for deep learning. In 2020 IEEE International Symposium on Circuits and Systems, ISCAS 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2020. 9180872. (Proceedings - IEEE International Symposium on Circuits and Systems).

ESSOP: Efficient and scalable stochastic outer product architecture for deep learning

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this