Learning both Weights and Connections for Efficient Neural Network

Song Han, Jeff Pool, John Tran, William Dally
Stanford, NVIDIA
(* indicates equal contribution)

News

Waiting for more news.

Awards

No items found.

Competition Awards

No items found.

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9×, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the total number of parameters can be reduced by 13×, from 138 million to 10.3 million, again with no loss of accuracy.

Video

Citation

@inproceedings{NIPS2015_ae0eb3ee,

author = {Han, Song and Pool, Jeff and Tran, John and Dally, William},

booktitle = {Advances in Neural Information Processing Systems},

title = {Learning both Weights and Connections for Efficient Neural Network},

year = {2015}

}

Media

No media articles found.

Acknowledgment

Team Members