Abstract
Adding LiDAR for an autonomous driving system will complement the weaknesses of a camera-only solution and enhance its robustness. To fully exploit the multimodal advantage, existing works have proposed various multimodal fusion algorithms to effectively combine LiDAR and camera data for scene and object recognition through 3D semantic segmentation. However, most of these methods leverage softmax-based attention modules for intra-modal feature encoding, and early fusion for inter-modal feature learning, leading to excessive computations and therefore higher latency in semantic segmentation. To mitigate this challenge, we propose the Semantic Segmentation (S2) Agent attention module for 3D semantic segmentation in autonomous driving system using LiDAR and camera. Intra-modal encoding is fully explored instead of early fusion using feature concatenation. We adopt a mid fusion strategy to further reduce computations. Experiments using open benchmark datasets nuScenes and Semantic KITTI show comparable or even better mIoUs than state-of-the-art baseline methods while obtaining better latency performance when compared to the most recent MSeg3D algorithm.
S. Zhang and Y. Guo—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Behley, J., et al.: Semantickitti: a dataset for semantic scene understanding of lidar sequences. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
Caesar, H., et al.: nuscenes: a multimodal dataset for autonomous driving. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: Pointnet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Chen, T.H., Chang, T.S.: Rangeseg: range-aware real time segmentation of 3D lidar point clouds. IEEE Trans. Intell. Veh. 7(1), 93–101 (2021)
Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_49
Cortinhal, T., Tzelepis, G., Aksoy, E.: Salsanext: fast semantic segmentation of lidar point clouds for autonomous driving (2020)
Han, D., Ye, T., Han, Y., Xia, Z., Song, S., Huang, G.: Agent attention: on the integration of softmax and linear attention. arXiv preprint arXiv:2312.08874 (2023)
Hu, Q., et al.: Randla-net: efficient semantic segmentation of large-scale point clouds. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Krispel, G., Opitz, M., Waltner, G., Possegger, H., Bischof, H.: Fuseseg: Lidar point cloud segmentation fusing multi-modal data. Cornell University - arXiv (2019)
Li, J., Dai, H., Han, H., Ding, Y.: MSeg3D: multi-modal 3D semantic segmentation for autonomous driving (2023)
Liu, Y., et al.: Uniseg: a unified multi-modal lidar segmentation network and the openpcseg codebase. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21662–21673 (2023)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: Rangenet++: fast and accurate lidar semantic segmentation. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4213–4220. IEEE (2019)
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., Sun, C.: Attention bottlenecks for multimodal fusion. Adv. Neural. Inf. Process. Syst. 34, 14200–14213 (2021)
Park, J., Kim, C., Kim, S., Jo, K.: Pcscnet: fast 3D semantic segmentation of lidar point cloud for autonomous car using point convolution and sparse convolution network. Expert Syst. Appl. 212, 118815 (2023)
Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
Wang, Y., Shi, T., Yun, P., Tai, L., Liu, M.: Pointseg: real-time semantic segmentation based on 3D lidar point cloud. arXiv preprint arXiv:1807.06288 (2018)
Xia, Z., Pan, X., Song, S., Li, L.E., Huang, G.: Vision transformer with deformable attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4794–4803 (2022)
Xu, C., et al.: SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation, pp. 1–19 (2020)
Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3D object detection and tracking. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Yuan, Y., Chen, X., Wang, J.: Object-Contextual Representations for Semantic Segmentation, pp. 173–190 (2020)
Zermas, D., Izzat, I., Papanikolopoulos, N.: Fast segmentation of 3D point clouds: a paradigm on lidar data for autonomous vehicle applications. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 5067–5073. IEEE (2017)
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Zhou, H., et al.: Cylinder3d: an effective 3D framework for driving-scene lidar semantic segmentation. arXiv, Computer Vision and Pattern Recognition (2020)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, S., Guo, Y., Lu, Y., Zeng, K., He, C., Cai, L. (2025). S2A-Attention for Multimodal 3D Semantic Segmentation Using LiDAR and Cameras in Autonomous Driving. In: Hadfi, R., Anthony, P., Sharma, A., Ito, T., Bai, Q. (eds) PRICAI 2024: Trends in Artificial Intelligence. PRICAI 2024. Lecture Notes in Computer Science(), vol 15284. Springer, Singapore. https://doi.org/10.1007/978-981-96-0125-7_21
Download citation
DOI: https://doi.org/10.1007/978-981-96-0125-7_21
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-0124-0
Online ISBN: 978-981-96-0125-7
eBook Packages: Computer ScienceComputer Science (R0)