Abstract
This paper presents a pruning method, Reconstruction Error Aware Pruning (REAP), to reduce the redundancy of convolutional neural network models for accelerating the inference. REAP is an extension of one of the state-of-the-art channel pruning methods. Our method takes 3 steps, (1) evaluating the importance of each channel based on the reconstruction error of the outputs in each convolutional layer, (2) pruning less important channels, (3) updating the remaining weights by the least squares method so as to reconstruct the outputs. By pruning with REAP, one can produce a fast and accurate model out of a large pretrained model. Besides, REAP saves us lots of time and efforts required for retraining the pruned model. As our method requires a large computational cost, we have developed an algorithm based on biorthogonal system to conduct the computation efficiently. In the experiments, we show that REAP can conduct pruning with smaller sacrifice of the model performances than several existing state-of-the-art methods such as CP [9], ThiNet [17], DCP [25], and so on.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aghasi, A., Abdi, A., Nguyen, N., Romberg, J.: Net-Trim: convex pruning of deep neural networks with performance guarantee. In: Advances in Neural Information Processing Systems, vol. 30, pp. 3177–3186. Curran Associates Inc. (2017)
akamaster. Proper implementation of resnet-s for cifar10/100 in pytorch that matches description of the original paper (2019)
Courbariaux, M., Bengio, Y., David, J.-P.: BinaryConnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, vol. 28, pp. 3123–3131. Curran Associates Inc. (2015)
Deng, J., Dong, W., Socher, R., Li, J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)
Dong, X., Chen, S., Pan, S.: Learning to prune deep neural networks via layer-wise optimal brain surgeon. In: Advances in Neural Information Processing Systems, vol. 30, pp. 4857–4867. Curran Associates Inc. (2017)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778 (2016)
He, T., Fan, Y., Qian, Y., Tan, T., Yu, K.: Reshaping deep neural network for fast decoding by node-pruning, pp. 245–249 (2014)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of International Conference on Computer Vision (2017)
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_48
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks, pp. 2261–2269 (2017)
Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization. In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (Canadian institute for advanced research)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, vol. 2, pp. 598–605. Morgan-Kaufmann (1990)
Liu, B., Wang, M., Foroosh, H., Tappen, M.F., Pensky, M.: Sparse convolutional neural networks, pp. 806–814 (2015)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of International Conference on Computer Vision (2017)
Luo, J.-H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of International Conference on Computer Vision (2017)
Simonyan, K., Zisserman, A.: Very deep convoolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2015)
Wang, H., Zhang, Q., Wang, Y., Hu, H.: Structured probabilistic pruning for convolutional neural network acceleration. In: Proceedings of British Machine Vision Conference (2018)
Xie, G., Wang, J., Zhang, T., Lai, J., Hong, R., Qi, G.-J.: Interleaved structured sparse convolutional neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)
Xue, J., Li, J., Gong, Y.: Restructuring of deep neural network acoustic models with singular value decomposition. In: INTERSPEECH (2013)
Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of Computer Vision and Pattern Recognition (2018)
Yu, X., Liu, T., Wang, X., Tao, D.: On compressing deep models by low rank and sparse decomposition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
Zhou, A., Yao, A., Wang, K., Chen, Y.: Explicit loss-error-aware quantization for low-bit deep neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)
Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Proceedings of Advances in Neural Information Processing Systems (2018)
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cuDNN: efficient Primitives for Deep Learning. Technical report (2011)
Zhao, Q., et al.: M2Det: a single-shot object detector based on multi-level feature pyramid network. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI) (2019)
Acknowledgment
This work was supported by JSPS KAKENHI Grant Number 19K12020 and the Environment Research and Technology Development Fund (3-1905) of the Environmental Restoration and Conservation Agency of Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kamma, K., Wada, T. (2019). Reconstruction Error Aware Pruning for Accelerating Neural Networks. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2019. Lecture Notes in Computer Science(), vol 11844. Springer, Cham. https://doi.org/10.1007/978-3-030-33720-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-33720-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33719-3
Online ISBN: 978-3-030-33720-9
eBook Packages: Computer ScienceComputer Science (R0)