DSD: Dense-Sparse-Dense Training for Deep Neural Networks Article Swipe

MAKE IMPACT

Song Han , Jeff Pool , Sharan Narang , Huizi Mao , Enhao Gong , Shijian Tang , Erich Elsen , Péter Vajda , Manohar Paluri , John Tran , Bryan Catanzaro , William J. Dally · YOU? · · 2016 · Open Access ·

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ'93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn't change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at this https URL.

Type

article

Language

en

Landing Page

https://arxiv.org/pdf/1607.04381

OA Status

green

Cited By

29

Related Works

20

OpenAlex ID

https://openalex.org/W2963126723

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W2963126723

Canonical identifier for this work in OpenAlex

Title

DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Work title

Type

article

OpenAlex work type

Language

en

Primary language

Publication year

2016

Year of publication

Publication date

2016-07-15

Full publication date if available

Authors

Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Péter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, William J. Dally

List of authors in order

Landing page

https://arxiv.org/pdf/1607.04381

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/1607.04381

Direct OA link when available

Concepts

Computer science, Pruning, Inference, Residual neural network, Artificial intelligence, Overhead (engineering), Constraint (computer-aided design), Artificial neural network, Deep neural networks, Deep learning, Network architecture, Recurrent neural network, Pattern recognition (psychology), Machine learning, Algorithm, Mathematics, Operating system, Biology, Computer security, Agronomy, Geometry

Top concepts (fields/topics) attached by OpenAlex

Cited by

29

Total citation count in OpenAlex

Citations by year (recent)

2022: 1, 2021: 6, 2020: 11, 2019: 7, 2018: 4

Per-year citation counts (last 5 years)

Related works (count)

20

Other works algorithmically related by OpenAlex

Full payload