Data from different modalities, such as infrared and visible images, can offer complementary information, and integrating such information can significantly enhance the capabilities of a system to perceive and recognize its surroundings. Thus, multi-modal object detection has widespread applications, particularly in challenging weather conditions like low-light scenarios. The core of multi-modal fusion lies in developing a reasonable fusion strategy, which can fully exploit the complementary features of different modalities while preventing a significant increase in model complexity. To this end, this paper proposes a novel lightweight cross-fusion module named Channel-Patch Cross Fusion (CPCF), which leverages Channel-wise Cross-Attention (CCA), Patch-wise Cross-Attention (PCA) and Adaptive Gating (AG) to encourage mutual rectification among different modalities. This process simultaneously explores commonalities across modalities while maintaining the uniqueness of each modality. Furthermore, we design a versatile intermediate fusion framework that can leverage CPCF to enhance the performance of multi-modal object detection. The proposed method is extensively evaluated on multiple public multi-modal datasets, namely FLIR, LLVIP, and DroneVehicle. The experiments indicate that our method yields consistent performance gains across various benchmarks and can be extended to different types of detectors, further demonstrating its robustness and generalizability. Our codes are available at https://github.com/Superjie13/CPCF_Multispectral.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Rethinking Self-Attention for Multispectral Object Detection


    Contributors:

    Published in:

    Publication date :

    2024-11-01


    Size :

    4881010 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    OBJECT DETECTION WITH MULTISPECTRAL LIGHT

    BAKER SIMON / HUMM ERIK | European Patent Office | 2023

    Free access


    Multispectral object detection with thermal imaging

    LOW YEW KWANG / TAYLOR RONALD M / EVERLY WARD K | European Patent Office | 2022

    Free access

    Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes

    Wu, Y. / Zhang, D. / Zhang, L. et al. | BASE | 2022

    Free access

    Concrete object recognition by multispectral systems

    Zlotnikov, K.A. | Tema Archive | 1998