Title
YABLoCo: Yet Another Benchmark for Long Context Code Generation Aidar Valeev Roman Garaev Vadim Lomshakov Irina Piontkovskaya Vladimir Ivanov Israel Adewuyi 38 0 0 07 May 2025
SecRepoBench: Benchmarking LLMs for Secure Code Generation in Real-World Repositories Connor Dilgren Purva Chiniya Luke Griffith Yu Ding Yizheng Chen 38 0 0 29 Apr 2025
CrashFixer: A crash resolution agent for the Linux kernel Alex Mathai Chenxi Huang Suwei Ma Jihwan Kim Hailie Mitchell Aleksandr Nogikh Petros Maniatis Franjo Ivančić Junfeng Yang Baishakhi Ray 55 0 0 29 Apr 2025
OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification Shangyu Li Juyong Jiang Tiancheng Zhao Jiasi Shen 41 0 0 29 Apr 2025
Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation Mingwei Liu Juntao Li Ying Wang Xueying Du Zuoyu Ou ... Zhao Wei Y. Xu Fangming Zou Xin Peng Yiling Lou 38 0 0 17 Apr 2025
RTLRepoCoder: Repository-Level RTL Code Completion through the Combination of Fine-Tuning and Retrieval Augmentation Peiyang Wu Nan Guo Junliang Lv Xiao Xiao Xiaochun Ye 29 1 0 11 Apr 2025
Towards an Understanding of Context Utilization in Code Intelligence Yanlin Wang Kefeng Duan Dewu Zheng Ensheng Shi F. Zhang ... Xilin Liu Yuchi Ma Hongyu Zhang Qianxiang Wang Zibin Zheng 29 0 0 11 Apr 2025
OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs Wasi Uddin Ahmad Aleksander Ficek Mehrzad Samadi Jocelyn Huang Vahid Noroozi Somshubra Majumdar Boris Ginsburg ALM 29 0 0 05 Apr 2025
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving Daoguang Zan Zhirong Huang Wei Liu Hanwu Chen L. Zhang ... Jing Su Tianyu Liu Rui Long Kai Shen Liang Xiang 36 1 0 03 Apr 2025
Compute Optimal Scaling of Skills: Knowledge vs Reasoning Nicholas Roberts Niladri S. Chatterji Sharan Narang Mike Lewis Dieuwke Hupkes 46 2 0 13 Mar 2025
DependEval: Benchmarking LLMs for Repository Dependency Understanding Junjia Du Yadi Liu Hongcheng Guo Jiawei Wang Haojian Huang Yunyi Ni Z. Li 46 1 0 09 Mar 2025
Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Enhancement Protocol Roham Koohestani Philippe de Bekker M. Izadi VLM 45 0 0 07 Mar 2025
SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair Zaoyu Chen Haoran Qin Nuo Chen Xiangyu Zhao Lei Xue Xiapu Luo Xiao-Ming Wu 41 0 0 03 Mar 2025
Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs Dayu Yang Tianyang Liu Daoan Zhang Antoine Simoulin Xiaoyi Liu ... Zhaopu Teng Xin Qian Grey Yang Jiebo Luo Julian McAuley ReLM OffRL LRM 81 3 0 26 Feb 2025
CodeSwift: Accelerating LLM Inference for Efficient Code Generation Qianhui Zhao L. Zhang Fang Liu Xiaoli Lian Qiaoyuanhe Meng Ziqian Jiao Zetong Zhou Borui Zhang Runlin Guo Jia Li 41 0 0 24 Feb 2025
SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors Bohan Lyu Siqiao Huang Zichen Liang Qi-An Sun Jiaming Zhang ELM LRM 47 0 0 16 Feb 2025
How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs Jialun Cao Yuk-Kit Chan Zixuan Ling Wenxuan Wang Shuqing Li ... Pinjia He Shuai Wang Zibin Zheng Michael R. Lyu S. Cheung ALM 69 2 0 18 Jan 2025
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing Siyuan Jiang Jia Li He Zong Huanyu Liu Hao Zhu ... Wei Ning G. Wang Yihong Dong Kechi Zhang Ge Li ALM 62 2 0 17 Jan 2025
CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval Y. Liu Rui Meng Shafiq R. Joty Silvio Savarese Caiming Xiong Yingbo Zhou Semih Yavuz 90 3 0 19 Nov 2024
EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations Jia Li Ge Li Xuanming Zhang Yunfei Zhao Yihong Dong Zhi Jin Binhua Li Fei Huang Yongbin Li ALM ELM 39 9 0 30 Oct 2024
Can Language Models Replace Programmers? REPOCOD Says Ñot Yet' Shanchao Liang Yiran Hu Nan Jiang Lin Tan ALM ELM 27 2 0 29 Oct 2024
M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation J. Liu Ken Deng Congnan Liu Jian Yang Shukai Liu ... Zekun Wang Guoan Zhang Bangyu Xiang Wenbo Su Bo Zheng 58 4 0 28 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 39 1 0 09 Oct 2024
Learning Code Preference via Synthetic Evolution Jiawei Liu Thanh Nguyen Mingyue Shang Hantian Ding Xiaopeng Li Yu Yu Varun Kumar Zijian Wang SyDa ALM AAML 23 3 0 04 Oct 2024
Horizon-Length Prediction: Advancing Fill-in-the-Middle Capabilities for Code Generation with Lookahead Planning Yifeng Ding Hantian Ding Shiqi Wang Qing Sun Varun Kumar Zijian Wang 25 2 0 04 Oct 2024
RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph Siru Ouyang W. Yu Kaixin Ma Zilin Xiao Z. Zhang Mengzhao Jia J. Han H. Zhang Dong Yu 47 12 0 03 Oct 2024
Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? Zhenyu Pan Rongyu Cao Yongchang Cao Yingwei Ma Binhua Li Fei Huang Han Liu Yongbin Li 40 4 0 02 Oct 2024
Data Analysis in the Era of Generative AI J. Inala Chenglong Wang Steven Drucker Gonzalo Ramos Victor C. Dibia N. Riche Dave Brown Dan Marshall Jianfeng Gao 20 6 0 27 Sep 2024
RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code Jiachi Chen Qingyuan Zhong Yanlin Wang Kaiwen Ning Yongkun Liu Zenan Xu Zhe Zhao Ting Chen Zibin Zheng AAML 13 7 0 23 Sep 2024
Qwen2.5-Coder Technical Report Binyuan Hui Jian Yang Zeyu Cui Jiaxi Yang Dayiheng Liu ... Fei Huang Xingzhang Ren Xuancheng Ren Jingren Zhou Junyang Lin OSLM 70 195 0 18 Sep 2024
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories Ben Bogin Kejuan Yang Shashank Gupta Kyle Richardson Erin Bransom Peter Clark Ashish Sabharwal Tushar Khot ELM LRM 40 9 0 11 Sep 2024
HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale Huy N. Phan Phong X. Nguyen Nghi D. Q. Bui LLMAG 28 10 0 09 Sep 2024
Statically Contextualizing Large Language Models with Typed Holes Andrew Blinn Xiang Li June Hyung Kim Cyrus Omar 27 1 0 02 Sep 2024
Strategic Optimization and Challenges of Large Language Models in Object-Oriented Programming Zinan Wang 19 0 0 27 Aug 2024
SWE-bench-java: A GitHub Issue Resolving Benchmark for Java Daoguang Zan Zhirong Huang Ailun Yu Shaoxin Lin Yifan Shi ... Bei Guan Pengjie Huang Tao Xie Yongji Wang Qianxiang Wang 23 7 0 26 Aug 2024
Retrieval-augmented code completion for local projects using large language models Marko Hostnik Marko Robnik-Sikonja RALM 17 0 0 09 Aug 2024
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases Xiangyan Liu Bo Lan Zhiyuan Hu Yang Liu Zhicheng Zhang Fei-Yue Wang Michael Shieh Wenmeng Zhou 37 11 0 07 Aug 2024
DDK: Distilling Domain Knowledge for Efficient Large Language Models Jiaheng Liu Chenchen Zhang Jinyang Guo Yuanxing Zhang Haoran Que ... Congnan Liu Wenbo Su Jiamang Wang Lin Qu Bo Zheng 43 3 0 23 Jul 2024
SciCode: A Research Coding Benchmark Curated by Scientists Minyang Tian Luyu Gao Shizhuo Dylan Zhang Xinan Chen Cunwei Fan ... Tianhua Tao Ofir Press Jamie Callan Eliu A. Huerta Hao Peng ELM 32 14 0 18 Jul 2024
On Mitigating Code LLM Hallucinations with API Documentation Nihal Jain Robert Kwiatkowski Baishakhi Ray M. K. Ramanathan Varun Kumar 33 7 0 13 Jul 2024
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates Zeyu Leo Liu Shrey Pandit Xi Ye Eunsol Choi Greg Durrett KELM ALM 53 4 0 08 Jul 2024
Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs Lei Zhang Yunshui Li Jiaming Li Xiaobo Xia Jiaxi Yang Run Luo Minzheng Wang Longze Chen Junhao Liu Min Yang 24 0 0 26 Jun 2024
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu ... David Lo Daniel Fried Xiaoning Du H. D. Vries Leandro von Werra 65 125 0 22 Jun 2024
Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models Sanjay Vishwakarma Francis Harkins Siddharth Golecha Vishal Sharathchandra Bajpe Nicolas Dupuis Luca Buratti David Kremer Ismael Faro Ruchir Puri Juan Cruz-Benito ELM 28 3 0 20 Jun 2024
CodeRAG-Bench: Can Retrieval Augment Code Generation? Zora Zhiruo Wang Akari Asai Xinyan Velocity Yu Frank F. Xu Yiqing Xie Graham Neubig Daniel Fried RALM 67 29 0 20 Jun 2024
DCA-Bench: A Benchmark for Dataset Curation Agents Benhao Huang Yingzhuo Yu Jin Huang Xingjian Zhang Jiaqi Ma 23 1 0 11 Jun 2024
RepoQA: Evaluating Long Context Code Understanding Jiawei Liu Jia Le Tian Vijay Daita Yuxiang Wei Yifeng Ding Yuhan Katherine Wang Jun Yang Lingming Zhang LLMAG 23 17 0 10 Jun 2024
Enhancing Repository-Level Code Generation with Integrated Contextual Information Zhiyuan Pan Xing Hu Xin Xia Xiaohu Yang 23 3 0 05 Jun 2024
R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models Ken Deng Jiaheng Liu He Zhu Congnan Liu Jingxin Li ... Yuanxing Zhang Wenbo Su Bangyu Xiang Tiezheng Ge Bo Zheng 40 2 0 03 Jun 2024
A Survey on Large Language Models for Code Generation Juyong Jiang Fan Wang Jiasi Shen Sungju Kim Sunghun Kim 40 74 0 01 Jun 2024