Multi-modal fusion can take advantage of the LiDAR and camera to boost the robustness and performance of 3D object detection. However, there are still of great challenges to comprehensively exploit image information and perform accurate diverse feature interaction fusion. In this paper, we proposed a novel multi-modal framework, namely Point-Pixel Fusion for Multi-Modal 3D Object Detection (PPF-Det). The PPF-Det consists of three submodules, Multi Pixel Perception (MPP), Shared Combined Point Feature Encoder (SCPFE), and Point-Voxel-Wise Triple Attention Fusion (PVW-TAF) to address the above problems. Firstly, MPP can make full use of image semantic information to mitigate the problem of resolution mismatch between point cloud and image. In addition, we proposed SCPFE to preliminary extract point cloud features and point-pixel features simultaneously reducing time-consuming on 3D space. Lastly, we proposed a fine alignment fusion strategy PVW-TAF to generate multi-level voxel-fused features based on attention mechanism. Extensive experiments on KITTI benchmarks, conducted on September 24, 2023, demonstrate that our method shows excellent performance.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    PPF-Det: Point-Pixel Fusion for Multi-Modal 3D Object Detection


    Contributors:
    Xie, Guotao (author) / Chen, Zhiyuan (author) / Gao, Ming (author) / Hu, Manjiang (author) / Qin, Xiaohui (author)


    Publication date :

    2024-06-01


    Size :

    17044755 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    RGB pixel-block point-cloud fusion for object detection

    Foster, Timothy / Dalal, Ajaya / Ball, John E. | British Library Conference Proceedings | 2021


    MLF3D: Multi-Level Fusion for Multi-Modal 3D Object Detection

    Jiang, Han / Wang, Jianbin / Xiao, Jianru et al. | IEEE | 2024


    VoPiFNet: Voxel-Pixel Fusion Network for Multi-Class 3D Object Detection

    Wang, Chia-Hung / Chen, Hsueh-Wei / Chen, Yi et al. | IEEE | 2024


    Frustum FusionNet: Amodal 3D Object Detection with Multi-Modal Feature Fusion

    Zuo, Liangyu / Li, Yaochen / Han, Mengtao et al. | IEEE | 2021


    Multi-Modal Sensor Fusion and Object Tracking for Autonomous Racing

    Karle, Phillip / Fent, Felix / Huch, Sebastian et al. | IEEE | 2023