In this study, a pixel-text level multi-stage multi-modality fusion segmentation method is proposed to make the open-world driving scene segmentation more efficient. It can be used for different semantic perceptual needs of autonomous driving scenarios for real-world driving situations. The method can finely segment unseen labels without additional corresponding semantic segmentation labels, only using the existing semantic segmentation data. The proposed method consists of 4 modules. A visual representation embedding module and a segmentation command embedding module are used to extract the driving scene and the segmentation category command. A multi-stage multi-modality fusion module is used to fuse the driving scene visual information and segmentation command text information for different sizes at the pixel-text level. Finally, a cascade segmentation head is used to ground the segmentation command text to the driving scene for encouraging the model to generate corresponding high-quality semantic segmentation results. In the experiment, we first verify the effectiveness of the method for zero-shot segmentation using a popular driving scene segmentation dataset. We also confirm the effectiveness of synonyms unseen label and hierarchy unseen label for the open-world semantic segmentation.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Open-world driving scene segmentation via multi-stage and multi-modality fusion of vision-language embedding


    Contributors:
    Niu, Yingjie (author) / Ding, Ming (author) / Zhang, Yuxiao (author) / Ge, Maoning (author) / Yang, Hanting (author) / Takeda, Kazuya (author)


    Publication date :

    2023-06-04


    Size :

    1419123 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation

    Salazar-Gomez, Gustavo / Liu, Wenqian / Diaz-Zapata, Manuel et al. | IEEE | 2024


    Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation

    Liu, Ruiping / Zhang, Jiaming / Peng, Kunyu et al. | IEEE | 2024


    Multi-Sensor Scene Segmentation

    Zhang, Xinyu / Li, Jun / Li, Zhiwei et al. | Springer Verlag | 2023


    Fusion of multi-modality volumetric medical imagery

    Aguilar, M. / New, J.R. | IEEE | 2002


    Fusion of Multi-Modality Volumetric Medical Imagery

    Aguilar, M. / International Society of Information Fusion / Institute of Electrical and Electronics Engineers | British Library Conference Proceedings | 2002