Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model’s performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code is publicly available at https://github.com/RuipingL/MISS.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation


    Contributors:


    Publication date :

    2024-06-02


    Size :

    1489473 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English




    Multi-Sensor Scene Segmentation

    Zhang, Xinyu / Li, Jun / Li, Zhiwei et al. | Springer Verlag | 2023


    Focus-aided scene segmentation

    Pertuz, S. / Garcia, M. A. / Puig, D. | British Library Online Contents | 2015


    Method for calculating automatic driving takeover prompt time in car-following scene

    WANG KUN / YU HAIYANG / REN YILONG et al. | European Patent Office | 2022

    Free access

    2018 Robotic Scene Segmentation Challenge

    Allan, M / Kondo, S / Bodenstedt, S et al. | BASE | 2020

    Free access