Skip to main content
Springer Nature Link
Log in

Partial convolutional reparameterization network for lightweight image super-resolution

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In recent years, convolutional neural networks (CNNs) have made significant strides in single image super-resolution (SISR). However, redundancy persists in network models concerning both channels and network structures, constituting a challenge in designing lightweight super-resolution (SR) networks. Consequently, finding a balance between efficiency and performance has emerged as the focus in SR research. In response to these challenges, we propose the Partial Convolutional Reparameterization Network (PCRN) for lightweight SR. Specifically, we initially employ partial convolution to reduce channel redundancy. Subsequently, we employ a complex network structure during model training, while in the inference stage, we utilize reparameterization techniques to compress the model, thus reducing redundancy in the network structure. Moreover, we have introduced enhanced spatial attention (ESA) and efficient channel attention (ECA) modules into our approach to enhance the model’s capability to extract key information. In comparative experiments, the proposed PCRN demonstrates superior performance over other efficient SR methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The code and data used in this study are publicly available on GitHub at the following repository: https://github.com/zl11250422/PCDB. The repository contains all the code related to this study and some pretrained models.

References

  1. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  2. Ahn, N., Kang, B., Sohn, K. A.: Fast, accurate, and, lightweight super-resolution with cascading residual network (2018). https://api.semanticscholar.org/CorpusID:4710341. arXiv:1803.08664

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, A.: Low-complexity single image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference (2012)

  4. Chen, J., Kao, S.-H., He, H., Zhuo, W., Wen, S., Lee, C.-H., Gary Chan, S.-H.: Run, don’t walk: chasing higher flops for faster neural networks (2023). arXiv preprint arXiv:2303.03667

  5. Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., Yu, F.: Dual aggregation transformer for image super-resolution. In: ICCV (2023)

  6. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1800–1807 (2017). https://doi.org/10.1109/CVPR.2017.195

  7. Cui, Y., Knoll, A.: Exploring the potential of channel interactions for image restoration. Knowl. Based Syst. 282, 111156 (2023)

    Article  Google Scholar 

  8. Cui, Y., Ren, W., Cao, X., Knoll, A.: Image restoration via frequency selection. IEEE Trans. Pattern Anal. Mach. Intell. (2023)

  9. Cui, Y., Ren, W., Cao, X., Knoll, A.: Focal network for image restoration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011 (2023)

  10. Cui, Y., Ren, W., Yang, S., Cao, X., Knoll, A.: Irnext: rethinking convolutional network design for image restoration. In: Proceedings of the 40th International Conference on Machine Learning (2023)

  11. Cui, Y., Tao, Y., Bing, Z., Ren, W., Gao, X., Cao, X., Huang, K., Knoll, A.: Selective frequency network for image restoration. In: The Eleventh International Conference on Learning Representations (2023)

  12. Cui, Y., Ren, W., Cao, X., Knoll, A.: Revitalizing convolutional network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell., 1–16 (2024). https://doi.org/10.1109/TPAMI.2024.3419007

  13. Cui, Y., Ren, W., Knoll, A.: Omni-kernel network for image restoration. Proc. AAAI Conf. Artif. Intell. 38(2), 1426–1434 (2024). https://doi.org/10.1609/aaai.v38i2.27907. https://ojs.aaai.org/index.php/AAAI/article/view/27907

  14. Ding, X., Guo, Y., Ding, G., Han, J.: Acnet: strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. In: The IEEE International Conference on Computer Vision (ICCV) (2019)

  15. Ding, X., Zhang, X., Han, J., Ding, G.: Diverse branch block: Building a convolution as an inception-like unit. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10886–10895 (2021)

  16. Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: Repvgg: Making vgg-style convnets great again. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733–13742 (2021)

  17. Dong, C., Loy, C. C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision – ECCV 2014. Springer International Publishing, Cham, pp. 184–199 (2014) (ISBN 978-3-319-10593-2)

  18. Dong, C., Change Loy C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision (2016). https://api.semanticscholar.org/CorpusID:13271756

  19. Fang, G., Ma, X., Song, M., Mi, M. B., Wang, X.: Depgraph: towards any structural pruning. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

  20. Gao, G., Li, W., Li, J., Fei, W., Huimin, L., Yi, Y.: Feature distillation interaction weighting network for lightweight image super-resolution. Proc. AAAI Conf. Artif. Intell. 36, 661–669 (2022)

    Google Scholar 

  21. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In: Bengio, Y., LeCun, Y. (eds) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016)

  22. Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv Learning (2016). https://api.semanticscholar.org/CorpusID:125617073

  23. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). arXiv preprint arXiv:1503.02531

  24. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications (2017). https://api.semanticscholar.org/CorpusID:12670695. arXiv: 1704.04861

  25. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  26. Huang, J. B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: IEEE (2015)

  27. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 723–731 (2018). https://doi.org/10.1109/CVPR.2018.00082

  28. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia (ACM MM), pages 2024–2032 (2019)

  29. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds) Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. (2015)

  30. Kim, J., Lee, J. K., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1637–1645 (2016). https://doi.org/10.1109/CVPR.2016.181

  31. Kim, J., Lee, J. K., Mu Lee K.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182

  32. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. Comput. Sci. (2014)

  33. Kong, F., Li, M., Liu, S., Liu, D., He, J., Bai, Y., Chen, F., Fu, L.: Residual local feature network for efficient super-resolution. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 765–775 (2022). https://doi.org/10.1109/CVPRW56347.2022.00092

  34. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5835–5843 (2017). https://doi.org/10.1109/CVPR.2017.618

  35. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105–114 (2017). https://doi.org/10.1109/CVPR.2017.19

  36. Lee, W., Lee, J., Kim, D., Ham, B.: Learning with privileged information for efficient image super-resolution. In: Proceedings of European Conference on Computer Vision (2020)

  37. Li, H., Yang, Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q., Chen, Y.: Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479: 47–59 (2022). (ISSN 0925-2312). https://doi.org/10.1016/j.neucom.2022.01.029

  38. Li, W., Zhou, K., Qi, L., Jiang, N., Lu, J., Jia, J.: Lapar: linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds) Advances in Neural Information Processing Systems, volume 33, pages 20343–20355. Curran Associates Inc. (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/eaae339c4d89fc102edd9dbdb6a28915-Paper.pdf

  39. Li, Z., Liu, Y., Chen, X., Cai, H., Gu, J., Qiao, Y., Dong, C.: Blueprint separable residual network for efficient image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 833–843 (2022)

  40. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1833–1844 (2021). https://doi.org/10.1109/ICCVW54120.2021.00210

  41. Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  42. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  43. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops, pages 41–55. Springer International Publishing, Cham (2020). (ISBN 978-3-030-67070-2)

  44. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2356–2365 (2020). https://doi.org/10.1109/CVPR42600.2020.00243

  45. Liu, Y., Dong, H., Liang, B., Liu, S., Dong, Q., Chen, K., Chen, F., Fu, L., Wang, F.: Unfolding once is enough: A deployment-friendly transformer unit for super-resolution. In: Proceedings of the 31st ACM International Conference on Multimedia, pages 7952–7960 (2023)

  46. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423 (2001). https://doi.org/10.1109/ICCV.2001.937655

  47. Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 76: 21811–21838 (2015). https://api.semanticscholar.org/CorpusID:8887614

  48. Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, Mohammad: Image super-resolution via iterative refinement. IEEE Transa. Pattern Anal. Mach. Intell. 45(4), 4713–4726 (2023). https://doi.org/10.1109/TPAMI.2022.3204461

    Article  Google Scholar 

  49. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874–1883 (2016). https://api.semanticscholar.org/CorpusID:7037846

  50. Sun, L., Dong, J., Tang, J., Pan, J.: Spatially-adaptive feature modulation for efficient image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)

  51. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning (2016). ArXiv:1602.07261. https://api.semanticscholar.org/CorpusID:1023605

  52. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826 (2016). https://doi.org/10.1109/CVPR.2016.308

  53. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2790–2798 (2017). https://doi.org/10.1109/CVPR.2017.298

  54. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: 2017 IEEE International Conference on Computer Vision (ICCV), pages 4809–4817 (2017). https://doi.org/10.1109/ICCV.2017.514

  55. Wang, A., Chen, H., Lin, Z., Han, J., Ding, G.: Revisiting mobile cnn from vit perspective, Repvit (2023)

  56. Wang, A., Chen, H., Lin, Z., Han, J., Ding, G.: Towards real-time segmenting anything, Repvit-sam (2023)

  57. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., Guo, Y.: Exploring sparsity in image super-resolution for efficient inference. In: CVPR (2021)

  58. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: Eca-net: efficient channel attention for deep convolutional neural networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11531–11539 (2020). https://doi.org/10.1109/CVPR42600.2020.01155

  59. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: enhanced super-resolution generative adversarial networks. In: The European Conference on Computer Vision Workshops (ECCVW) (2018)

  60. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  61. Woo, S., Park, J., Lee, J.-Y., Kweon Cbam, I. S.: Convolutional block attention module. In: Computer Vision – ECCV 2018. Springer International Publishing, Cham, pages 3–19 (2018). (ISBN 978-3-030-01234-2)

  62. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces (2010)

  63. Yulun, Z., Kunpeng, L., Kai, L., Lichen, W., Bineng, Z., Yun, F.: Image super-resolution using very deep residual channel attention networks. In: ECCV (2018)

Download references

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Lanzhou University, 222 S. Tianshui Rd., Lanzhou, 730000, China

    Long Zhang & Yi Wan

Authors
  1. Long Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Long Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Wan, Y. Partial convolutional reparameterization network for lightweight image super-resolution. J Real-Time Image Proc 21, 187 (2024). https://doi.org/10.1007/s11554-024-01565-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-024-01565-y

Keywords

Search

Navigation