Assessing LLM code generation quality through path planning tasks

As LLM-generated code grows in popularity, more evaluation is needed to assess the risks of using such tools, especially for safety-critical applications such as path planning. Existing coding benchmarks are insufficient as they do not reflect the context and complexity of safety-critical applications. To this end, we assessed six LLMs' abilities to generate the code for three different path-planning algorithms and tested them on three maps of various difficulties. Our results suggest that LLM-generated code presents serious hazards for path planning applications and should not be applied in safety-critical contexts without rigorous testing.

@article{chen2025_2504.21276,
  title={ Assessing LLM code generation quality through path planning tasks },
  author={ Wanyi Chen and Meng-Wen Su and Mary L. Cummings },
  journal={arXiv preprint arXiv:2504.21276},
  year={ 2025 }
}