Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Experiments, and Challenges

With the broader usage and highly successful development of Large Language Models (LLMs), there has been a growth of interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning ability, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to language interaction and decision-making. In this paper, we first introduce the novel concept of designing LLMs for autonomous driving (LLM4AD). Then, we propose a comprehensive benchmark for evaluating the instruction-following abilities of LLM4AD in simulation. Furthermore, we conduct a series of experiments on real-world vehicle platforms, thoroughly evaluating the performance and potential of our LLM4AD systems. Finally, we envision the main challenges of LLM4AD, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. Our research highlights the significant potential of LLMs to enhance various aspects of autonomous vehicle technology, from perception and scene understanding to language interaction and decision-making.

@article{cui2025_2410.15281,
  title={ Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Experiments, and Challenges },
  author={ Can Cui and Yunsheng Ma and Zichong Yang and Yupeng Zhou and Peiran Liu and Juanwu Lu and Lingxi Li and Yaobin Chen and Jitesh H. Panchal and Amr Abdelraouf and Rohit Gupta and Kyungtae Han and Ziran Wang },
  journal={arXiv preprint arXiv:2410.15281},
  year={ 2025 }
}