Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal by grounding natural language instructions to the visual surroundings. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. In this paper, we explore the use of counterfactual thinking as a human-inspired data augmentation method that results in robust models. Counterfactual thinking is a concept that describes the human propensity to create possible alternatives to life events that have already occurred. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data. In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance. APS also serves to do pre-exploration of unseen environments to strengthen the model's ability to generalize. We evaluate the influence of APS on the performance of different VLN baseline models using the room-to-room dataset (R2R). The results show that the adversarial training process with our proposed APS benefits VLN models under both seen and unseen environments. And the pre-exploration process can further gain additional improvements under unseen environments.


    Access

    Download


    Export, share and cite



    Title :

    Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling


    Contributors:

    Publication date :

    2019


    Remarks:

    ECCV'20 (Spotlight)


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models

    Cai, Hengxing / Dong, Jinhan / Tan, Jingjun et al. | ArXiv | 2025

    Free access

    Counterfactual quantum computation

    Hosten, O. / Rakher, M.T. / Barreiro, J.T. et al. | IEEE | 2005


    Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters

    LANDI, FEDERICO / Baraldi, Lorenzo / CORSINI, Massimiliano et al. | BASE | 2019

    Free access


    Multimodal Attention Networks for Low-Level Vision-and-Language Navigation

    Federico Landi / Lorenzo Baraldi / Marcella Cornia et al. | BASE | 2021

    Free access