Skip to main content
Springer Nature Link
Log in

GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15067))

Included in the following conference series:

  • 419 Accesses

Abstract

Implicit neural representations (INRs) recently achieved great success in image representation and compression, offering high visual quality and fast rendering speeds with 10–1000 FPS, assuming sufficient GPU resources are available. However, this requirement often hinders their use on low-end devices with limited memory. In response, we propose a groundbreaking paradigm of image representation and compression by 2D Gaussian Splatting, named GaussianImage. We first introduce 2D Gaussian to represent the image, where each Gaussian has 8 parameters including position, covariance and color. Subsequently, we unveil a novel rendering algorithm based on accumulated summation. Remarkably, our method with a minimum of 3\(\times \) lower GPU memory usage and 5\(\times \) faster fitting time not only rivals INRs (e.g., WIRE, I-NGP) in representation performance, but also delivers a faster rendering speed of 1500–2000 FPS regardless of parameter size. Furthermore, we integrate existing vector quantization technique to build an image codec. Experimental results demonstrate that our codec attains rate-distortion performance comparable to compression-based INRs such as COIN and COIN++, while facilitating decoding speeds of approximately 2000 FPS. Additionally, preliminary proof of concept shows that our codec surpasses COIN and COIN++ in performance when using partial bits-back coding. Code is available at https://github.com/Xinjie-Q/GaussianImage.

X. Zhang and X. Ge—Equal Contribution.

This work was done when Xinjie Zhang and Xingtong Ge interned at SenseTime Research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 70.18
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 86.50
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kodak lossless true color image suite (1999). https://r0k.us/graphics/kodak/

  2. Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

    Google Scholar 

  3. Ahmed, N., Natarajan, T., Rao, K.R.: Discrete cosine transform. IEEE Trans. Comput. 100(1), 90–93 (1974)

    Article  MathSciNet  Google Scholar 

  4. Ballé, J., Laparra, V., Simoncelli, E.P.: Density modeling of images using a generalized normalization transformation (2015). arXiv preprint arXiv:1511.06281

  5. Ballé, J., Laparra, V., Simoncelli, E.P.: End-to-end optimized image compression. In: International Conference on Learning Representations (2017)

    Google Scholar 

  6. Ballé, J., Minnen, D., Singh, S., Hwang, S.J., Johnston, N.: Variational image compression with a scale hyperprior. In: International Conference on Learning Representations (2018)

    Google Scholar 

  7. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855–5864 (2021)

    Google Scholar 

  8. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470–5479 (2022)

    Google Scholar 

  9. Bellard, F.: BPG image format (2014). https://bellard.org/bpg/

  10. Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., Kwak, N.: LSQ+: improving low-bit quantization through learnable offsets and better initialization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 696–697 (2020)

    Google Scholar 

  11. Chen, G., Wang, W.: A survey on 3D gaussian splatting (2024). arXiv preprint arXiv:2401.03890

  12. Chen, H., Gwilliam, M., Lim, S.N., Shrivastava, A.: HNeRV: a hybrid neural representation for videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10270–10279 (2023)

    Google Scholar 

  13. Chen, H., et al.: NeRV: neural representations for videos. Adv. Neural. Inf. Process. Syst. 34, 21557–21568 (2021)

    Google Scholar 

  14. Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8628–8638 (2021)

    Google Scholar 

  15. Chen, Y., et al.: GaussianEditor: swift and controllable 3D editing with gaussian splatting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Google Scholar 

  16. Chen, Z., et al.: https://github.com/oppo-us-research/NeuRBF

  17. Chen, Z., et al.: NeuRBF: a neural fields representation with adaptive radial basis functions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4182–4194 (2023)

    Google Scholar 

  18. Chen, Z., Wang, F., Liu, H.: Text-to-3D using gaussian splatting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Google Scholar 

  19. Cheng, Z., Sun, H., Takeuchi, M., Katto, J.: Learned image compression with discretized gaussian mixture likelihoods and attention modules. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939–7948 (2020)

    Google Scholar 

  20. Dupont, E., Loya, H., Alizadeh, M., Golinski, A., Teh, Y., Doucet, A.: COIN++: neural compression across modalities. Trans. Mach. Learn. Res. 2022(11) (2022)

    Google Scholar 

  21. Dupont, E., Golinski, A., Alizadeh, M., Teh, Y.W., Doucet, A.: COIN: compression with implicit neural representations. In: Neural Compression: From Information Theory to Applications–Workshop@ ICLR 2021 (2021)

    Google Scholar 

  22. Fathony, R., Sahu, A.K., Willmott, D., Kolter, J.Z.: Multiplicative filter networks. In: International Conference on Learning Representations (2020)

    Google Scholar 

  23. Gray, R.: Vector quantization. IEEE ASSP Mag. 1(2), 4–29 (1984)

    Article  Google Scholar 

  24. Guo, Z., Flamich, G., He, J., Chen, Z., Hernández-Lobato, J.M.: Compression with bayesian implicit neural representations. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  25. He, D., Yang, Z., Peng, W., Ma, R., Qin, H., Wang, Y.: ELIC: efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5718–5727 (2022)

    Google Scholar 

  26. Heil, C.E., Walnut, D.F.: Continuous and discrete wavelet transforms. SIAM Rev. 31(4), 628–666 (1989)

    Article  MathSciNet  Google Scholar 

  27. Higham, N.J.: Cholesky factorization. Wiley Interdisc. Rev. Comput. Stat. 1(2), 251–254 (2009)

    Article  Google Scholar 

  28. Huang, H., Li, L., Cheng, H., Yeung, S.K.: Photo-SLAM: real-time simultaneous localization and photorealistic mapping for monocular, stereo, and RGB-D cameras (2023). arXiv preprint arXiv:2311.16728

  29. Keetha, N.,et al.: SplaTAM: splat, track & map 3D gaussians for dense RGB-D SLAM (2023). arXiv preprint arXiv:2312.02126

  30. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4), 139–1 (2023)

    Google Scholar 

  31. Koyuncu, A.B., Gao, H., Boev, A., Gaikov, G., Alshina, E., Steinbach, E.: Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022. ECCV 2022. LNCS, vol. 13679, pp. 447–463. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19800-7_26

  32. Kunze, J., Severo, D., Zani, G., van de Meent, J.W., Townsend, J.: Entropy coding of unordered data structures. In: The Twelfth International Conference on Learning Representations (2024). https://openreview.net/forum?id=afQuNt3Ruh

  33. Ladune, T., Philippe, P., Henry, F., Clare, G., Leguay, T.: COOL-CHIC: coordinate-based low complexity hierarchical image Codec. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13515–13522 (2023)

    Google Scholar 

  34. Leguay, T., Ladune, T., Philippe, P., Clare, G., Henry, F., Déforges, O.: Low-complexity overfitted neural image codec. In: 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6. IEEE (2023)

    Google Scholar 

  35. Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., Liu, Y.: E-NeRV: expedite neural video representation with disentangled spatial-temporal context. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022. ECCV 2022. LNCS, vol. 13695. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19833-5_16

  36. Liu, J., Sun, H., Katto, J.: Learned image compression with mixed transformer-CNN architectures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14388–14397 (2023)

    Google Scholar 

  37. Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3D gaussians: tracking by persistent dynamic view synthesis (2023). arXiv preprint arXiv:2308.09713

  38. Ma, C., Yu, P., Lu, J., Zhou, J.: Recovering realistic details for magnification-arbitrary image super-resolution. IEEE Trans. Image Process. 31, 3669–3683 (2022)

    Article  Google Scholar 

  39. Martel, J.N., Lindell, D.B., Lin, C.Z., Chan, E.R., Monteiro, M., Wetzstein, G.: ACORN: adaptive coordinate networks for neural scene representation. ACM Trans. Graph. (TOG) 40(4), 1–13 (2021)

    Article  Google Scholar 

  40. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pp. 405–421. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  41. Minnen, D., Ballé, J., Toderici, G.D.: Joint autoregressive and hierarchical priors for learned image compression. In: Advances in Neural Information Processing Systems (2018)

    Google Scholar 

  42. Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. (TOG) 41(4), 1–15 (2022)

    Article  Google Scholar 

  43. Nguyen, Q.H., Beksi, W.J.: Single image super-resolution via a dual interactive implicit neural network. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4936–4945 (2023)

    Google Scholar 

  44. Quan, Y., Yao, X., Ji, H.: Single image defocus deblurring via implicit neural inverse kernels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12600–12610 (2023)

    Google Scholar 

  45. Ramasinghe, S., Lucey, S.: Beyond periodicity: towards a unifying framework for activations in coordinate-MLPs. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022. ECCV 2022. LNCS, vol. 13693. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_9

  46. Ryder, T., Zhang, C., Kang, N., Zhang, S.: Split hierarchical variational compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 386–395 (2022)

    Google Scholar 

  47. Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veeraraghavan, A., Baraniuk, R.G.: WIRE: wavelet implicit neural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507–18516 (2023)

    Google Scholar 

  48. Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Adv. Neural. Inf. Process. Syst. 33, 7462–7473 (2020)

    Google Scholar 

  49. Skodras, A., Christopoulos, C., Ebrahimi, T.: The JPEG 2000 still image compression standard. IEEE Signal Process. Mag. 18(5), 36–58 (2001)

    Article  Google Scholar 

  50. Stanley, K.O.: Compositional pattern producing networks: a novel abstraction of development. Genet. Program Evolvable Mach. 8, 131–162 (2007)

    Article  Google Scholar 

  51. Strümpler, Y., Postels, J., Yang, R., Gool, L.V., Tombari, F.: Implicit neural representations for image compression. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022. ECCV 2022. LNCS, vol. 13686, pp. 74–91. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19809-0_5

  52. Takikawa, T., et al.: Neural geometric level of detail: real-time rendering with implicit 3D shapes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358–11367 (2021)

    Google Scholar 

  53. Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. Adv. Neural. Inf. Process. Syst. 33, 7537–7547 (2020)

    Google Scholar 

  54. Townsend, J., Bird, T., Barber, D.: Practical lossless compression with latent variables using bits back coding (2019). arXiv preprint arXiv:1901.04866

  55. Wallace, G.K.: The JPEG still picture compression standard. Commun. ACM 34(4), 30–44 (1991)

    Article  Google Scholar 

  56. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. vol. 2, pp. 1398–1402. IEEE (2003)

    Google Scholar 

  57. Wu, G., et al.: 4D gaussian splatting for real-time dynamic scene rendering (2023). arXiv preprint arXiv:2310.08528

  58. Xie, X., Zhou, P., Li, H., Lin, Z., Shuicheng, Y.: Adan: adaptive nesterov momentum algorithm for faster optimizing deep models. In: Has it Trained Yet? NeurIPS 2022 Workshop (2022)

    Google Scholar 

  59. Xu, D., Wang, P., Jiang, Y., Fan, Z., Wang, Z.: Signal processing for implicit neural representations. Adv. Neural. Inf. Process. Syst. 35, 13404–13418 (2022)

    Google Scholar 

  60. Xu, Q., et al.: Point-NeRF: point-based neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5438–5448 (2022)

    Google Scholar 

  61. Xu, W., Jiao, J.: Revisiting implicit neural representations in low-level vision. In: International Conference on Learning Representations Workshop (2023)

    Google Scholar 

  62. Yan, C., et al.: GS-SLAM: dense visual slam with 3D gaussian splatting (2023). arXiv preprint arXiv:2311.11700

  63. Yan, Y., et al.: Street gaussians for modeling dynamic urban scenes (2024). arXiv preprint arXiv:2401.01339

  64. Yang, Z., Yang, H., Pan, Z., Zhu, X., Zhang, L.: Real-time photorealistic dynamic scene representation and rendering with 4D gaussian splatting (2023). arXiv preprint arXiv:2310.10642

  65. Ye, V., Turkulainen, M., the Nerfstudio team: gsplat. https://github.com/nerfstudio-project/gsplat

  66. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., Tagliasacchi, M.: SoundStream: an end-to-end neural audio codec. IEEE/ACM Trans. Audio Speech Lang. Process. 30, 495–507 (2021)

    Article  Google Scholar 

  67. Zhang, X., et al.: Boosting neural representations for videos with a conditional decoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Google Scholar 

  68. Zhou, X., Lin, Z., Shan, X., Wang, Y., Sun, D., Yang, M.H.: DrivingGaussian: composite gaussian splatting for surrounding dynamic autonomous driving scenes (2023). arXiv preprint arXiv:2312.07920

  69. Zielonka, W., Bagautdinov, T., Saito, S., Zollhöfer, M., Thies, J., Romero, J.: Drivable 3D gaussian avatars (2023). arXiv preprint arXiv:2311.08581

Download references

Acknowledgments

This work was supported by the National Natural Science Fund of China (Project No. 42201461, 62102024, 62331014) and the General Research Fund (Project No. 16209622) from the Hong Kong Research Grants Council.

Author information

Authors and Affiliations

  1. The Hong Kong University of Science and Technology, Hong Kong, China

    Xinjie Zhang & Jun Zhang

  2. Beijing Institute of Technology, Beijing, China

    Xingtong Ge & Jing Geng

  3. SenseTime Research, Hong Kong, China

    Xinjie Zhang, Xingtong Ge & Hongwei Qin

  4. Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China

    Tongda Xu & Yan Wang

  5. The Chinese University of Hong Kong, Hong Kong, China

    Dailan He

  6. Shanghai Jiaotong University, Shanghai, China

    Guo Lu

Authors
  1. Xinjie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingtong Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tongda Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dailan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongwei Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guo Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jing Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jing Geng or Jun Zhang .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 514 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, X. et al. (2025). GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15067. Springer, Cham. https://doi.org/10.1007/978-3-031-72673-6_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72673-6_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72672-9

  • Online ISBN: 978-3-031-72673-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation