The invention discloses an automatic driving method and system based on an end-to-end and multi-modal large model. The method comprises the following steps: determining image information, vehicle historical information, navigation information and text information collected by a plurality of sensors of a target vehicle; based on an encoder corresponding to each type of information, respectively encoding the image information, the vehicle historical information, the navigation information and the text information to obtain a plurality of encoding tokens; respectively inputting the plurality of coding tokens into a trained multi-modal large language model for prediction to obtain a plurality of decoding tokens; decoding the plurality of decoding tokens based on a trajectory decoder and a text decoder, and outputting a planned trajectory and control information of the target vehicle in parallel; and based on the planned trajectory and the control information, controlling the target vehicle to automatically drive according to the planned trajectory, and performing text interpretation on the current automatic driving strategy. Therefore, the scheme is high in integration level and construction efficiency, high in generalization, high in iteration and high in adaptability, and the automatic driving landing and service efficiency is improved.
本申请公开了一种基于端到端与多模态大模型的自动驾驶方法及系统。所述方法包括:确定由目标车辆的多个传感器采集的图像信息、车辆历史信息、导航信息以及文本信息;基于每种信息对应的编码器,分别对图像信息、车辆历史信息、导航信息以及文本信息进行编码,得到多个编码token;将多个编码token分别输入训练好的多模态大语言模型进行预测,得到多个解码token;基于轨迹解码器和文本解码器分别对多个解码token进行解码,并行输出目标车辆的规划轨迹和控制信息;基于规划轨迹和控制信息,控制目标车辆按照规划轨迹自动驾驶,并对当前自动驾驶策略进行文本解释。从而,方案集成度和搭建效率高,泛化性强,迭代性高,适配性强,提高自动驾驶落地与服务效率。
Automatic driving method and system based on end-to-end and multi-modal large model
一种基于端到端与多模态大模型的自动驾驶方法及系统
16.05.2025
Patent
Elektronische Ressource
Chinesisch
IPC: | G06N COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS , Rechnersysteme, basierend auf spezifischen Rechenmodellen / B60W CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION , Gemeinsame Steuerung oder Regelung von Fahrzeug-Unteraggregaten verschiedenen Typs oder verschiedener Funktion / G05D SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES , Systeme zum Steuern oder Regeln nichtelektrischer veränderlicher Größen |
Europäisches Patentamt | 2025
|Automatic driving collaborative perception method and device based on multi-modal large model
Europäisches Patentamt | 2024
|Automatic driving vehicle dynamic pick-up management system based on multi-modal large model
Europäisches Patentamt | 2025
|Cloud intelligent driving control system based on multi-modal large model
Europäisches Patentamt | 2025
|Automatic driving method and system based on multi-view multi-modal fusion
Europäisches Patentamt | 2024
|