The open-vocabulary understanding of UAV aerial images plays a crucial role in enhancing the intelligence level of remote sensing applications, such as disaster assessment, precision agriculture, and urban planning. In this paper, we propose an innovative open-vocabulary model for UAV images, which combines vision–language methods to achieve efficient recognition and segmentation of unseen categories by generating multi-view image descriptions and feature extraction. To enhance the generalization ability and robustness of the model, we adopted Mixup technology to blend multiple UAV images, generating more diverse and representative training data. To address the limitations of existing open-vocabulary models in UAV image analysis, we leverage the GPT model to generate accurate and professional text descriptions of aerial images, ensuring contextual relevance and precision. The image encoder utilizes a U-Net with Mamba architecture to extract key point information through edge detection and partition pooling, further improving the effectiveness of feature representation. The text encoder employs a fine-tuned BERT model to convert text descriptions of UAV images into feature vectors. Three key loss functions were designed: Generalization Loss to balance old and new category scores, semantic segmentation loss to evaluate model performance on UAV image segmentation tasks, and Triplet Loss to enhance the model’s ability to distinguish features. The Comprehensive Loss Function integrates these terms to ensure robust performance in complex UAV segmentation tasks. Experimental results demonstrate that the proposed method has significant advantages in handling unseen categories and achieving high accuracy in UAV image segmentation tasks, showcasing its potential for practical applications in diverse aerial imagery scenarios.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Expanding Open-Vocabulary Understanding for UAV Aerial Imagery: A Vision–Language Framework to Semantic Segmentation


    Beteiligte:
    Bangju Huang (Autor:in) / Junhui Li (Autor:in) / Wuyang Luan (Autor:in) / Jintao Tan (Autor:in) / Chenglong Li (Autor:in) / Longyang Huang (Autor:in)


    Erscheinungsdatum :

    2025




    Medientyp :

    Aufsatz (Zeitschrift)


    Format :

    Elektronische Ressource


    Sprache :

    Unbekannt




    Aerial-BiSeNet: A real-time semantic segmentation network for high resolution aerial imagery

    WANG, Fang / LUO, Xiaoyan / WANG, Qixiong et al. | Elsevier | 2021

    Freier Zugriff

    Semantic Understanding of Traffic Scenes with Large Vision Language Models

    Jain, Sandesh / Thapa, Surendrabikram / Chen, Kuan-Ting et al. | IEEE | 2024




    Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles

    J. de Curtò / I. de Zarzà / Carlos T. Calafate | DOAJ | 2023

    Freier Zugriff