PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs

8 January 2024

Himanshu Beniwal

Papers citing "PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs"

9 / 9 papers shown

Title
LLMs Love Python: A Study of LLMs' Bias for Programming Languages and Libraries Lukas Twist Jie M. Zhang Mark Harman Don Syme Joost Noppen Detlef Nauck 39 0 0 21 Mar 2025
Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Enhancement Protocol Roham Koohestani Philippe de Bekker M. Izadi VLM 45 0 0 07 Mar 2025
How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code Seonghyeon Lee Heejae Chon Joonwon Jang Dongha Lee Hwanjo Yu ALM 39 0 0 02 Mar 2025
Integrating Expert Knowledge into Logical Programs via LLMs Franciszek Górski Oskar Wysocki Marco Valentino André Freitas 34 0 0 17 Feb 2025
LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks Xin Zhou M. Weyssow Ratnadira Widyasari Ting Zhang Junda He Yunbo Lyu Jianming Chang Beiqi Zhang Dan Huang David Lo PILM 146 0 0 10 Feb 2025
How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs Jialun Cao Yuk-Kit Chan Zixuan Ling Wenxuan Wang Shuqing Li ... Pinjia He Shuai Wang Zibin Zheng Michael R. Lyu S. Cheung ALM 69 2 0 18 Jan 2025
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement Tianyu Zheng Ge Zhang Tianhao Shen Xueling Liu Bill Yuchen Lin Jie Fu Wenhu Chen Xiang Yue SyDa 76 102 0 08 Jan 2025
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 194 614 0 20 May 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 190 853 0 09 Feb 2021