In recent years, deep learning-based end-to-end autonomous driving has become increasingly popular. However, deep neural networks are like black boxes. Their outputs are generally not explainable, making them not reliable to be used in real-world environments. To provide a solution to this problem, we propose an explainable deep neural network that jointly predicts driving actions and multimodal environment descriptions of traffic scenes, including bird-eye-view (BEV) maps and natural-language environment descriptions. In this network, both the context information from BEV perception and the local information from semantic perception are considered before producing the driving actions and natural-language environment descriptions. To evaluate our network, we build a new dataset with hand-labelled ground truth for driving actions and multimodal environment descriptions. Experimental results show that the combination of context information and local information enhances the prediction performance of driving action and environment description, thereby improving the safety and explainability of our end-to-end autonomous driving network.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Multimodal-XAD: Explainable Autonomous Driving Based on Multimodal Environment Descriptions


    Contributors:
    Feng, Yuchao (author) / Feng, Zhen (author) / Hua, Wei (author) / Sun, Yuxiang (author)

    Published in:

    Publication date :

    2024-12-01


    Size :

    3191271 byte




    Type of media :

    Article (Journal)


    Type of material :

    Electronic Resource


    Language :

    English



    Multimodal End-to-End Autonomous Driving

    Xiao, Yi / Codevilla, Felipe / Gurram, Akhil et al. | IEEE | 2022


    MULTIMODAL MOTION PLANNING FRAMEWORK FOR AUTONOMOUS DRIVING VEHICLES

    ZHANG YAJIA / LI DONG / ZHANG LIANGLIANG et al. | European Patent Office | 2022

    Free access

    Autonomous Multimodal Logistics

    KULKARNI GIRISH / BHASKARAN MICHAEL / GOPALAKRISHNAN VIKRANTH | European Patent Office | 2018

    Free access

    Motion Query-based Multimodal Vehicle Trajectory Prediction for Autonomous Driving

    Jiang, Hao / Niu, Yixun / Hu, Chuan et al. | IEEE | 2023


    Autonomous multimodal logistics

    KULKARNI GIRISH / BHASKARAN MICHAEL / GOPALAKRISHNAN VIKRANTH | European Patent Office | 2019

    Free access