Vision-and-Language Navigation (VLN) is a challenging task in which an agent needs to follow a language-specified path to reach a target destination. The goal gets even harder as the actions available to the agent get simpler and move towards low-level, atomic interactions with the environment. This setting takes the name of low-level VLN. In this paper, we strive for the creation of an agent able to tackle three key issues: multi-modality, long-term dependencies, and adaptability towards different locomotive settings. To that end, we devise "Perceive, Transform, and Act" (PTA): a fully-attentive VLN architecture that leaves the recurrent approach behind and the first Transformer-like architecture incorporating three different modalities -- natural language, images, and low-level actions for the agent control. In particular, we adopt an early fusion strategy to merge lingual and visual information efficiently in our encoder. We then propose to refine the decoding phase with a late fusion extension between the agent's history of actions and the perceptual modalities. We experimentally validate our model on two datasets: PTA achieves promising results in low-level VLN on R2R and achieves good performance in the recently proposed R4R benchmark. Our code is publicly available at https://github.com/aimagelab/perceive-transform-and-act.


    Access

    Download


    Export, share and cite



    Title :

    Multimodal Attention Networks for Low-Level Vision-and-Language Navigation



    Publication date :

    2021-01-01



    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Classification :

    DDC:    629



    Hierarchical & multimodal video captioning: Discovering and transferring multimodal knowledge for vision to language

    Liu, An-An / Xu, Ning / Wong, Yongkang et al. | British Library Online Contents | 2017


    Multimodal navigation display

    Dostal, Martin / Kolcarek, Pavel | IEEE | 2015


    Multimodal navigation display

    Dostal, Martin | IEEE | 2015

    Free access

    FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models

    Cai, Hengxing / Dong, Jinhan / Tan, Jingjun et al. | ArXiv | 2025

    Free access

    Analysing Driver's Attention Level using Computer Vision

    Bergasa, Luis M. / Buenaposada, Jose M. / Nuevo, Jesus et al. | IEEE | 2008