In this study, a pixel-text level multi-stage multi-modality fusion segmentation method is proposed to make the open-world driving scene segmentation more efficient. It can be used for different semantic perceptual needs of autonomous driving scenarios for real-world driving situations. The method can finely segment unseen labels without additional corresponding semantic segmentation labels, only using the existing semantic segmentation data. The proposed method consists of 4 modules. A visual representation embedding module and a segmentation command embedding module are used to extract the driving scene and the segmentation category command. A multi-stage multi-modality fusion module is used to fuse the driving scene visual information and segmentation command text information for different sizes at the pixel-text level. Finally, a cascade segmentation head is used to ground the segmentation command text to the driving scene for encouraging the model to generate corresponding high-quality semantic segmentation results. In the experiment, we first verify the effectiveness of the method for zero-shot segmentation using a popular driving scene segmentation dataset. We also confirm the effectiveness of synonyms unseen label and hierarchy unseen label for the open-world semantic segmentation.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Open-world driving scene segmentation via multi-stage and multi-modality fusion of vision-language embedding


    Beteiligte:
    Niu, Yingjie (Autor:in) / Ding, Ming (Autor:in) / Zhang, Yuxiao (Autor:in) / Ge, Maoning (Autor:in) / Yang, Hanting (Autor:in) / Takeda, Kazuya (Autor:in)


    Erscheinungsdatum :

    04.06.2023


    Format / Umfang :

    1419123 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation

    Salazar-Gomez, Gustavo / Liu, Wenqian / Diaz-Zapata, Manuel et al. | IEEE | 2024


    Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation

    Liu, Ruiping / Zhang, Jiaming / Peng, Kunyu et al. | IEEE | 2024


    Multi-Sensor Scene Segmentation

    Zhang, Xinyu / Li, Jun / Li, Zhiwei et al. | Springer Verlag | 2023


    Fusion of multi-modality volumetric medical imagery

    Aguilar, M. / New, J.R. | IEEE | 2002


    Fusion of Multi-Modality Volumetric Medical Imagery

    Aguilar, M. / International Society of Information Fusion / Institute of Electrical and Electronics Engineers | British Library Conference Proceedings | 2002