In this paper, we introduce a new deep learning architecture for camera and Lidar sensor fusion. The proposed scheme performs 2D object detection using the RGB camera image and the depth, height, and intensity images generated by projecting the 3D Lidar point cloud into camera image plane. The proposed object detector consists of two convolutional neural networks (CNNs) that process the RGB and Lidar images separately as well as the fusion network that combines the feature maps produced at the intermediate layers of the CNNs. We aim to develop a robust object detector that maintains good object detection accuracy even when the quality of the sensor signals is degraded for object detection. Towards this end, we devise the gated fusion unit (GFU) that adjusts the contribution of the feature maps generated by two CNN structures via gating mechanism. Using the GFU, the proposed object detector can fuse the high level feature maps drawn from two modalities with appropriate weights to achieve robust performance. Experiments conducted on the challenging KITTI benchmark show that the proposed camera and Lidar fusion network outperforms the conventional sensor fusion methods even when either of the camera and Lidar sensor signals is corrupted by missing data, occlusion, noise, and illumination change.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Robust Camera Lidar Sensor Fusion Via Deep Gated Information Fusion Network


    Contributors:


    Publication date :

    2018-06-01


    Size :

    3840012 byte




    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    GATED LOAD BALANCING FOR UNCERTAINTY AWARE CAMERA-LIDAR FUSION

    BALACHANDRAN BALAJI SHANKAR / RAVI KUMAR VARUN / YOGAMANI SENTHIL KUMAR | European Patent Office | 2025

    Free access

    Deep fusion of camera and LIDAR

    Schneider, Nick | TIBKAT | 2023

    Free access

    Sensor Fusion of Camera and Lidar Using Kalman Filter

    Kunjumon, Reshma / Sangeetha Gopan, G. S. | Springer Verlag | 2021


    Camera-LIDAR Integration: Probabilistic Sensor Fusion for Semantic Mapping

    Berrio, Julie Stephany / Shan, Mao / Worrall, Stewart et al. | IEEE | 2022


    LIDAR-Camera Fusion Where LIDAR and Camera Validly See Different Things

    BLAU CHAIM / SPRINGER OFER / ROSENBLUM KEVIN et al. | European Patent Office | 2022

    Free access