This paper investigates the integration of Large Vision Language Models (LVLMs) with multi-sensor information, including visual and localization data from cameras and LiDAR data to a holistic understanding of traffic videos. Traffic scene understanding is a challenging problem. With complex interaction between the road actors, infrastructure, and traffic rules, it is often difficult to answer questions related to road safety, pedestrian safety, safe maneuvering characteristics, and human factors. Typical processes use a single task-oriented neural network model and combine them through semantic and symbolic reasoning. These processes often suffer from reasoning bias and incompleteness. In recent years, LVLMs have opened new avenues to perceive spatiotemporal information. These models can leverage the large knowledge base from the world and summarize spatiotemporal information effectively. The interactive nature of most of these systems allows humans to directly interact in a visual question-answering mode.In this paper, we have extensively tested the capabilities of such LVLMs to answer key transportation research questions from videos captured through front cameras. We have curated an extensive set of multiple-choice questions to evaluate the performance of these LVLMs. Our results show that LVLMs have abilities to understand various transportation-related aspects to a great extent. Furthermore, we have shown that the addition of supplementary modalities to the VQA settings helps improve the performance of LVLMs. With the addition of 3D trajectories of surrounding objects with the 2D video frames, we observed a significant increase in MCQ performance related to vehicle-to-vehicle interaction tasks. The resources for this paper can be found at https://github.com/sandeshrjain/lvlm-scene


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Semantic Understanding of Traffic Scenes with Large Vision Language Models


    Contributors:


    Publication date :

    2024-06-02


    Size :

    1872472 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    Domain-Incremental Semantic Segmentation for Traffic Scenes

    Liu, Yazhou / Chen, Haoqi / Lasang, Pongsak et al. | IEEE | 2025


    Understanding object relations in traffic scenes

    Hensel, Irina / Bachmann, Alexander / Hummel, Britta et al. | Tema Archive | 2010


    Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles

    J. de Curtò / I. de Zarzà / Carlos T. Calafate | DOAJ | 2023

    Free access

    Semantic Understanding of Foggy Scenes with Purely Synthetic Data

    Hahner, Martin / Dai, Dengxin / Sakaridis, Christos et al. | IEEE | 2019


    A Review of Vision-Based Traffic Semantic Understanding in ITSs

    Chen, Jing / Wang, Qichao / Cheng, Harry H. et al. | IEEE | 2022