Image segmentation plays a critical role in autonomous driving by providing vehicles with a detailed and accurate understanding of their surroundings. Transformers have recently shown encouraging results in image segmentation. However, transformer-based models are challenging to strike a better balance between performance and efficiency. The computational complexity of the transformer-based models is quadratic with the number of inputs, which severely hinders their application in dense prediction tasks. In this paper, we present the semantic-aware dimension-pooling transformer (SDPT) to mitigate the conflict between accuracy and efficiency. The proposed model comprises an efficient transformer encoder for generating hierarchical features and a semantic-balanced decoder for predicting semantic masks. In the encoder, a dimension-pooling mechanism is used in the multi-head self-attention (MHSA) to reduce the computational cost, and a parallel depth-wise convolution is used to capture local semantics. Simultaneously, we further apply this dimension-pooling attention (DPA) to the decoder as a refinement module to integrate multi-level features. With such a simple yet powerful encoder-decoder framework, we empirically demonstrate that the proposed SDPT achieves excellent performance and efficiency on various popular benchmarks, including ADE20K, Cityscapes, and COCO-Stuff. For example, our SDPT achieves 48.6% mIOU on the ADE20K dataset, which outperforms the current methods with fewer computational costs. The codes can be found at https://github.com/HuCaoFighting/SDPT.


    Access

    Download


    Export, share and cite



    Title :

    SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation


    Contributors:
    Cao, Hu (author) / Chen, Guang (author) / Zhao, Hengshuang (author) / Jiang, Dongsheng (author) / Zhang, Xiaopeng (author) / Tian, Qi (author) / Knoll, Alois (author)

    Published in:

    Publication date :

    2024-11-01


    Size :

    2542242 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English






    Situation-Aware Model Refinement for Semantic Image Segmentation

    Habermayr, Lukas / Hofbauer, Markus / Zacchi, Joao-Vitor et al. | IEEE | 2021


    Hyperbolic Uncertainty Aware Semantic Segmentation

    Chen, Bike / Peng, Wei / Cao, Xiaofeng et al. | IEEE | 2024


    TransRVNet: LiDAR Semantic Segmentation With Transformer

    Cheng, Hui-Xian / Han, Xian-Feng / Xiao, Guo-Qiang | IEEE | 2023