Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model’s performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code is publicly available at https://github.com/RuipingL/MISS.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation


    Beteiligte:
    Liu, Ruiping (Autor:in) / Zhang, Jiaming (Autor:in) / Peng, Kunyu (Autor:in) / Chen, Yufan (Autor:in) / Cao, Ke (Autor:in) / Zheng, Junwei (Autor:in) / Sarfraz, M. Saquib (Autor:in) / Yang, Kailun (Autor:in) / Stiefelhagen, Rainer (Autor:in)


    Erscheinungsdatum :

    02.06.2024


    Format / Umfang :

    1489473 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Multi-Sensor Scene Segmentation

    Zhang, Xinyu / Li, Jun / Li, Zhiwei et al. | Springer Verlag | 2023


    Focus-aided scene segmentation

    Pertuz, S. / Garcia, M. A. / Puig, D. | British Library Online Contents | 2015


    Method for calculating automatic driving takeover prompt time in car-following scene

    WANG KUN / YU HAIYANG / REN YILONG et al. | Europäisches Patentamt | 2022

    Freier Zugriff

    2018 Robotic Scene Segmentation Challenge

    Allan, M / Kondo, S / Bodenstedt, S et al. | BASE | 2020

    Freier Zugriff