Skip to main content

SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training

Publication ,  Conference
Dai, P; Yang, J; Ye, X; Cheng, X; Luo, J; Song, L; Chen, Y; Zhao, W
Published in: Proceedings - Design Automation Conference
July 1, 2020

Training Convolutional Neural Networks (CNNs) usually requires a large number of computational resources. In this paper, SparseTrain is proposed to accelerate CNN training by fully exploiting the sparsity. It mainly involves three levels of innovations: activation gradients pruning algorithm, sparse training dataflow, and accelerator architecture. By applying a stochastic pruning algorithm on each layer, the sparsity of back-propagation gradients can be increased dramatically without degrading training accuracy and convergence rate. Moreover, to utilize both natural sparsity (resulted from ReLU or Pooling layers) and artificial sparsity (brought by pruning algorithm), a sparse-aware architecture is proposed for training acceleration. This architecture supports forward and back-propagation of CNN by adopting 1-Dimensional convolution dataflow. We have built a cycle-accurate architecture simulator to evaluate the performance and efficiency based on the synthesized design with 14nm FinFET technologies. Evaluation results on AlexNet/ResNet show that SparseTrain could achieve about 2.7× speedup and 2.2× energy efficiency improvement on average compared with the original training process.

Duke Scholars

Published In

Proceedings - Design Automation Conference

DOI

ISSN

0738-100X

ISBN

9781450367257

Publication Date

July 1, 2020

Volume

2020-July
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Dai, P., Yang, J., Ye, X., Cheng, X., Luo, J., Song, L., … Zhao, W. (2020). SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training. In Proceedings - Design Automation Conference (Vol. 2020-July). https://doi.org/10.1109/DAC18072.2020.9218710
Dai, P., J. Yang, X. Ye, X. Cheng, J. Luo, L. Song, Y. Chen, and W. Zhao. “SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training.” In Proceedings - Design Automation Conference, Vol. 2020-July, 2020. https://doi.org/10.1109/DAC18072.2020.9218710.
Dai P, Yang J, Ye X, Cheng X, Luo J, Song L, et al. SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training. In: Proceedings - Design Automation Conference. 2020.
Dai, P., et al. “SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training.” Proceedings - Design Automation Conference, vol. 2020-July, 2020. Scopus, doi:10.1109/DAC18072.2020.9218710.
Dai P, Yang J, Ye X, Cheng X, Luo J, Song L, Chen Y, Zhao W. SparseTrain: Exploiting dataflow sparsity for efficient convolutional neural networks training. Proceedings - Design Automation Conference. 2020.

Published In

Proceedings - Design Automation Conference

DOI

ISSN

0738-100X

ISBN

9781450367257

Publication Date

July 1, 2020

Volume

2020-July