Skip to main content
Springer Nature Link
Log in

Freditor: High-Fidelity and Transferable NeRF Editing by Frequency Decomposition

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15099))

Included in the following conference series:

  • 219 Accesses

Abstract

This paper enables high-fidelity, transferable NeRF editing by frequency decomposition. Recent NeRF editing pipelines lift 2D stylization results to 3D scenes while suffering from blurry results, and fail to capture detailed structures caused by the inconsistency between 2D editings. Our critical insight is that low-frequency components of images are more multiview-consistent after editing compared with their high-frequency parts. Moreover, the appearance style is mainly exhibited on the low-frequency components, and the content details especially reside in high-frequency parts. This motivates us to perform editing on low-frequency components, which results in high-fidelity edited scenes. In addition, the editing is performed in the low-frequency feature space, enabling stable intensity control and novel scene transfer. Comprehensive experiments conducted on photorealistic datasets demonstrate the superior performance of high-fidelity and transferable NeRF editing. The project page is at https://aigc3d.github.io/freditor.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 63.34
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 78.06
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/opencv/opencv.

References

  1. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: CVPR (2022)

    Google Scholar 

  2. Boss, M., Jampani, V., Braun, R., Liu, C., Barron, J.T., Lensch, H.P.A.: Neural-PIL: neural pre-integrated lighting for reflectance decomposition. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 6–14 December 2021, Virtual, pp. 10691–10704 (2021). https://proceedings.neurips.cc/paper/2021/hash/58ae749f25eded36f486bc85feb3f0ab-Abstract.html

  3. Brooks, T., Holynski, A., Efros, A.A.: InstructPix2Pix: learning to follow image editing instructions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, 17–24 June 2023, pp. 18392–18402. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.01764

  4. Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), October 2021. https://doi.org/10.1109/iccv48922.2021.00951

  5. Chen, Y., et al.: GaussianEditor: swift and controllable 3D editing with Gaussian splatting. CoRR abs/2311.14521 (2023). https://doi.org/10.48550/ARXIV.2311.14521

  6. Chiang, P., Tsai, M., Tseng, H., Lai, W., Chiu, W.: Stylizing 3D scene via implicit representation and hypernetwork. In: IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, 3–8 January 2022, pp. 215–224. IEEE (2022). https://doi.org/10.1109/WACV51458.2022.00029

  7. Duan, H., Wang, M., Li, Y., Yang, Y.: PaintNeSF: artistic creation of stylized scenes with vectorized 3D strokes. CoRR abs/2311.15637 (2023). https://doi.org/10.48550/ARXIV.2311.15637

  8. Fan, Z., Jiang, Y., Wang, P., Gong, X., Xu, D., Wang, Z.: Unified implicit neural stylization. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XV. LNCS, vol. 13675, pp. 636–654. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_37

  9. Fang, J., Wang, J., Zhang, X., Xie, L., Tian, Q.: GaussianEditor: editing 3D Gaussians delicately with text instructions. CoRR abs/2311.16037 (2023). https://doi.org/10.48550/ARXIV.2311.16037

  10. Feng, Y., Shang, Y., Li, X., Shao, T., Jiang, C., Yang, Y.: PIE-NeRF: physics-based interactive elastodynamics with NeRF. CoRR abs/2311.13099 (2023). https://doi.org/10.48550/ARXIV.2311.13099

  11. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. https://doi.org/10.1109/cvpr.2016.265

  12. Haque, A., Tancik, M., Efros, A., Holynski, A., Kanazawa, A.: Instruct-NeRF2NeRF: editing 3D scenes with instructions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)

    Google Scholar 

  13. Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1998, January 1998. https://doi.org/10.1145/280814.280951

  14. Huang, H.P., Tseng, H.Y., Saini, S., Singh, M., Yang, M.H.: Learning to stylize novel views. In: International Conference on Computer Vision, January 2021

    Google Scholar 

  15. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), October 2017. https://doi.org/10.1109/iccv.2017.167

  16. Huang, Y., He, Y., Yuan, Y., Lai, Y., Gao, L.: StylizedNeRF: consistent 3D scene stylization as stylized nerf via 2D-3D mutual learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, 18–24 June 2022, pp. 18321–18331. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.01780

  17. Hwang, S., Hyung, J., Kim, D., Kim, M., Choo, J.: FaceClipNeRF: text-driven 3D face manipulation using deformable neural radiance fields. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, 1–6 October 2023, pp. 3446–3456. IEEE (2023). https://doi.org/10.1109/ICCV51070.2023.00321

  18. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part II. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

  19. Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing NeRF for editing via feature field distillation. In: NeurIPS (2022). http://papers.nips.cc/paper_files/paper/2022/hash/93f250215e4889119807b6fac3a57aec-Abstract-Conference.html

  20. Koo, J., Park, C., Sung, M.: Posterior distillation sampling. CoRR abs/2311.13831 (2023). https://doi.org/10.48550/ARXIV.2311.13831

  21. Kuang, Z., Luan, F., Bi, S., Shu, Z., Wetzstein, G., Sunkavalli, K.: PaletteNeRF: palette-based appearance editing of neural radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, 17–24 June 2023, pp. 20691–20700. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.01982

  22. Li, B., Weinberger, K.Q., Belongie, S.J., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, 25–29 April 2022. OpenReview.net (2022). https://openreview.net/forum?id=RriDjddCLN

  23. Li, J., et al.: InstructPix2NeRF: instructed 3D portrait editing from a single image. CoRR abs/2311.02826 (2023). https://doi.org/10.48550/ARXIV.2311.02826

  24. Li, Y., Lin, Z.H., Forsyth, D., Huang, J.B., Wang, S.: ClimateNeRF: extreme weather synthesis in neural radiance field. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Google Scholar 

  25. Liang, J., Zeng, H., Zhang, L.: High-resolution photorealistic image translation in real-time: a Laplacian pyramid translation network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9392–9400 (2021)

    Google Scholar 

  26. Liu, K., et al.: StyleRF: zero-shot 3D style transfer of neural radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, 17–24 June 2023, pp. 8338–8348. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.00806

  27. Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.Y., Russell, B.: Editing conditional radiance fields. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), October 2021. https://doi.org/10.1109/iccv48922.2021.00572

  28. Mikaeili, A., Perel, O., Safaee, M., Cohen-Or, D., Mahdavi-Amiri, A.: SKED: sketch-guided text-based 3D editing. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, 1–6 October 2023, pp. 14561–14573. IEEE (2023). https://doi.org/10.1109/ICCV51070.2023.01343

  29. Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P.P., Barron, J.T.: NeRF in the dark: high dynamic range view synthesis from noisy raw images. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022. https://doi.org/10.1109/cvpr52688.2022.01571

  30. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, 23–28 August 2020, Proceedings, Part I. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

  31. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21, 4695–4708 (2012). https://doi.org/10.1109/tip.2012.2214050

    Article  MathSciNet  Google Scholar 

  32. Mu, F., Wang, J., Wu, Y., Li, Y.: 3D photo stylization: learning to generate stylized novel views from a single image. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022. https://doi.org/10.1109/cvpr52688.2022.01579

  33. Munkberg, J., et al.: Extracting triangular 3D models, materials, and lighting from images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8280–8290, June 2022

    Google Scholar 

  34. Nguyen-Phuoc, T., Liu, F., Xiao, L.: SNeRF: stylized neural implicit representations for 3D scenes. ACM Trans. Graph. 41(4), 142:1–142:11 (2022). https://doi.org/10.1145/3528223.3530107

  35. Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 5436–5445. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00548, https://openaccess.thecvf.com/content_CVPR_2020/html/Niklaus_Softmax_Splatting_for_Video_Frame_Interpolation_CVPR_2020_paper.html

  36. Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F.: Neural scene graphs for dynamic scenes. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021. https://doi.org/10.1109/cvpr46437.2021.00288

  37. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18–24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021). http://proceedings.mlr.press/v139/radford21a.html

  38. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.1556

  39. Song, H., Choi, S., Do, H., Lee, C., Kim, T.: Blending-NeRF: text-driven localized editing in neural radiance fields. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, 1–6 October 2023, pp. 14337–14347. IEEE (2023). https://doi.org/10.1109/ICCV51070.2023.01323

  40. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: NeRV: neural reflectance and visibility fields for relighting and view synthesis. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021. https://doi.org/10.1109/cvpr46437.2021.00741

  41. Tancik, M., et al.: NeRFStudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023 (2023)

    Google Scholar 

  42. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow (extended abstract). In: Zhou, Z. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event/Montreal, Canada, 19–27 August 2021, pp. 4839–4843. ijcai.org (2021). https://doi.org/10.24963/IJCAI.2021/662

  43. Tschernezki, V., Laina, I., Larlus, D., Vedaldi, A.: Neural feature fusion fields: 3D distillation of self-supervised 2D image representations. In: International Conference on 3D Vision, 3DV 2022, Prague, Czech Republic, 12–16 September 2022, pp. 443–453. IEEE (2022). https://doi.org/10.1109/3DV57658.2022.00056

  44. Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-NeRF: structured view-dependent appearance for neural radiance fields. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022. https://doi.org/10.1109/cvpr52688.2022.00541

  45. Wang, C., Chai, M., He, M., Chen, D., Liao, J.: CLIP-NeRF: text-and-image driven manipulation of neural radiance fields. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022. https://doi.org/10.1109/cvpr52688.2022.00381

  46. Wang, C., Jiang, R., Chai, M., He, M., Chen, D., Liao, J.: NeRF-Art: text-driven neural radiance fields stylization. IEEE Trans. Vis. Comput. Graph., 1–15 (2023). https://doi.org/10.1109/TVCG.2023.3283400

  47. Xie, T., et al.: PhysGaussian: physics-integrated 3D Gaussians for generative dynamics. CoRR abs/2311.12198 (2023). https://doi.org/10.48550/ARXIV.2311.12198

  48. Yu, H., Guibas, L.J., Wu, J.: Unsupervised discovery of object radiance fields. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, 25–29 April 2022. OpenReview.net (2022). https://openreview.net/forum?id=rwE8SshAlxw

  49. Yuan, Y., Sun, Y., Lai, Y., Ma, Y., Jia, R., Gao, L.: NeRF-editing: geometry editing of neural radiance fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, 18–24 June 2022, pp. 18332–18343. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.01781

  50. Zhang, K., et al.: ARF: artistic radiance fields. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Proceedings, Part XXXI. LNCS, vol. 13691, pp. 717–733. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19821-2_41

  51. Zhang, K., et al.: ARF: artistic radiance fields (2022)

    Google Scholar 

  52. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018. https://doi.org/10.1109/cvpr.2018.00068

Download references

Author information

Authors and Affiliations

  1. Alibaba Group, Hangzhou, China

    Yisheng He, Weihao Yuan, Zilong Dong & Liefeng Bo

  2. Fudan University, Shanghai, China

    Siyu Zhu

  3. The University of Texas at Austin, Austin, USA

    Qixing Huang

Authors
  1. Yisheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weihao Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zilong Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liefeng Bo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qixing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 39236 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

He, Y., Yuan, W., Zhu, S., Dong, Z., Bo, L., Huang, Q. (2025). Freditor: High-Fidelity and Transferable NeRF Editing by Frequency Decomposition. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15099. Springer, Cham. https://doi.org/10.1007/978-3-031-72940-9_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72940-9_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72939-3

  • Online ISBN: 978-3-031-72940-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation