Multi-modal 3D object detection has received growing attention as the information from different sensors like LiDAR and cameras are complementary. Most fusion methods for 3D detection rely on an accurate alignment and calibration between 3D point clouds and RGB images. However, such an assumption is not reliable in a real-world self-driving system, as the alignment between different modalities is easily affected by asynchronous sensors and disturbed sensor placement. We propose a novel Fusion network by Box Matching (FBMNet) for multi-modal 3D detection, which provides an alternative way for cross-modal feature alignment by learning the correspondence at the bounding box level to free up the dependency of calibration during inference. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combining their ROI features. Extensive experiments on the nuScenes dataset demonstrate that our method is much more robust in dealing with challenging cases such as asynchronous sensors, misaligned sensor placement, and degenerated camera images than existing fusion methods. We hope that our FBMNet could provide an available solution to dealing with these challenging cases for safety in real autonomous driving scenarios.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Multi-Modal 3D Object Detection by Box Matching


    Beteiligte:
    Liu, Zhe (Autor:in) / Ye, Xiaoqing (Autor:in) / Zou, Zhikang (Autor:in) / He, Xinwei (Autor:in) / Tan, Xiao (Autor:in) / Ding, Errui (Autor:in) / Wang, Jingdong (Autor:in) / Bai, Xiang (Autor:in)

    Erschienen in:

    Erscheinungsdatum :

    01.12.2024


    Format / Umfang :

    3911014 byte




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Object recognition and categorization using modal matching

    Sclaroff, S. / Pentland, A.P. | IEEE | 1994


    Object Recognition and Categorization Using Modal Matching

    Sclaroff, S. / Pentland, A. P. / IEEE Computer Society; Technical Committee on Pattern Analysis and Machine Intelligence | British Library Conference Proceedings | 1994


    Object velocity detection from multi-modal sensor data

    PRADHAN SABEEK MANI / SLOAN COOPER STOKES | Europäisches Patentamt | 2023

    Freier Zugriff

    MLF3D: Multi-Level Fusion for Multi-Modal 3D Object Detection

    Jiang, Han / Wang, Jianbin / Xiao, Jianru et al. | IEEE | 2024


    Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection

    Zhu, Yaohui / Sun, Xiaoyu / Wang, Miao et al. | IEEE | 2023