Abstract
In this study, we propose a real-time method for realizing illumination consistency and global illumination in augmented reality (AR). The proposed method uses pix2pix, which is a generative adversarial network (GAN) for image-to-image translation. The network takes an image with k channels as the input, and attempts to generate reflections and shadows of a virtual object corresponding to the illumination condition. We also propose an approach for improving the applicability of the method by combining RGB information with geometric information (normal and depth) as the network input. For evaluating the proposed method, we created a synthetic dataset by using Unreal Engine 4, which can render computer graphics (CG) images with global illumination. The results of an experiment indicated that although generated images were not completely the same as the ground truth, the proposed method reproduced natural-looking reflections and shadows of a virtual object.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Debevec, P.: Image-based lighting. In: Proceedings of the ACM Special Interest Group on Graphics and Interactive Techniques 2006 Courses, p. 4-es (2006)
Pessoa, S.A., et al.: Illumination techniques for photorealistic rendering in augmented reality. In: Proceedings of the Symposium on Virtual and Augmented Reality 2008, pp. 223–232 (2008)
Tuceryan, M.: AR360: dynamic illumination for augmented reality with real-time interaction. In: Proceedings of the IEEE International Conference on Information and Computer Technologies 2019, pp. 170–174 (2019)
Kán, P., Unterguggenberger, J., Kaufmann, H.: High-quality consistent illumination in mobile augmented reality by radiance convolution on the GPU. In: Bebis, G., et al. (eds.) ISVC 2015. LNCS, vol. 9474, pp. 574–585. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-27857-5_52
Gardner, M.-A., et al.: Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36(6), Article no. 176 (2017)
Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.-F.: Deep outdoor illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, pp. 7312–7321 (2017)
Thomas, M.M., Forbes, A.G.: Deep illumination: approximating dynamic global illumination with generative adversarial network. arXiv preprint arXiv:1710.09834 (2017)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, pp. 1125–1134 (2017)
Wang, X., Wang, K., Lian, S.: Deep consistent illumination in augmented reality. In: Adjunct Proceedings of the IEEE International Symposium on Mixed and Augmented Reality 2019, pp. 189–194 (2019)
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018, pp. 8798–8807 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jingtao, H., Komuro, T. (2022). End-to-End Deep Neural Network for Illumination Consistency and Global Illumination. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13598. Springer, Cham. https://doi.org/10.1007/978-3-031-20713-6_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-20713-6_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20712-9
Online ISBN: 978-3-031-20713-6
eBook Packages: Computer ScienceComputer Science (R0)