Autonomous Vehicles (AVs) have the potential of reducing car accidents and increasing accessibility to transportation. AVs need to be rigorously tested. Scenario-based testing offers a set of approaches to design high-risk tests for AVs at low cost. Since the AVs need to be tested for a large number of scenarios, automated generation approaches are needed. Pre-trained Large Language Models (LLMs) are open-input, general-purpose data generators with good learning and reasoning abilities. However, due to the black-box nature of these systems, it's difficult to get direct evidence of their abilities. In this paper, we address the open question of the reasoning capabilities of pre-trained LLMs specifically in the context of scenario-based testing of AVs. Inspired by QA benchmarks for LLM evaluations for commonsense reasoning, science reasoning, and more, we present our main contribution, ScenarioQA. This benchmark involves an LLM-based QA generation process based on an integration of methods to generate questions and corresponding answers specifically in the context of scenario-based testing. We carry out a comprehensive evaluation of this process and gain valuable insights regarding effective QA generation. In addition, we evaluate several available pre-trained LLMs for these abilities.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    ScenarioQA: Evaluating Test Scenario Reasoning Capabilities of Large Language Models


    Contributors:


    Publication date :

    2024-09-24


    Size :

    369423 byte





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English



    A Large-Scale Traffic Scenario of Berlin for Evaluating Smart Mobility Applications

    Schrab, Karl / Protzmann, Robert / Radusch, Ilja | Springer Verlag | 2023


    Payload test capabilities of a large compensated compact range

    DUDOK, E. / HABERSACK, J. / HARTMANN, F. et al. | AIAA | 1992


    Past and Present Large Solid Rocket Motor Test Capabilities

    Kowalski, Robert R. / Owen, David B., II | NTRS | 2011


    Past and Present Large Solid Rocket Motor Test Capabilities

    R. R. Kowalski / D. B. Owen | NTIS | 2011


    Chat2Scenario: Scenario Extraction From Dataset Through Utilization of Large Language Model

    Zhao, Yongqi / Xiao, Wenbo / Mihalj, Tomislav et al. | IEEE | 2024