Title
JumpCoder: Go Beyond Autoregressive Coder via Online Modification Mouxiang Chen Hao Tian Zhongxi Liu Xiaoxue Ren Jianling Sun SyDa KELM 20 2 0 15 Jan 2024
PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs Ankit Yadav Himanshu Beniwal Mayank Singh LRM ALM 14 12 0 08 Jan 2024
Instruction Fusion: Advancing Prompt Evolution through Hybridization Weidong Guo Jiuding Yang Kaitong Yang Xiangyang Li Zhuwei Rao Yu-Syuan Xu Di Niu 8 5 0 25 Dec 2023
ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code Xiangru Tang Yuliang Liu Zefan Cai Yan Shao Junjie Lu ... Yujia Qin Wangchunshu Zhou Yilun Zhao Arman Cohan Mark B. Gerstein ELM LLMAG 25 6 0 16 Nov 2023
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation Weixiang Yan Haitian Liu Yunkun Wang Yunzhe Li Qian Chen ... Tingyu Lin Weishan Zhao Li Zhu Hari Sundaram Shuiguang Deng ELM LRM 13 34 0 14 Nov 2023
Data Augmentation for Code Translation with Comparable Corpora and Multiple References Yiqing Xie Atharva Naik Daniel Fried Carolyn Rose 34 4 0 01 Nov 2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press Karthik Narasimhan ELM 13 130 0 10 Oct 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 37 1,121 0 24 Aug 2023
Benchmarking Causal Study to Interpret Large Language Models for Source Code Daniel Rodríguez-Cárdenas David Nader-Palacio Dipin Khati Henry Burke Denys Poshyvanyk CML ELM 11 15 0 23 Aug 2023
Reflexion: Language Agents with Verbal Reinforcement Learning Noah Shinn Federico Cassano Beck Labash A. Gopinath Karthik Narasimhan Shunyu Yao LLMAG KELM 11 1,070 0 20 Mar 2023
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval Mohammad Abdullah Matin Khan M Saiful Bari Xuan Long Do Weishi Wang Md. Rizwan Parvez Shafiq R. Joty ALM ELM 27 14 0 06 Mar 2023
Large Language Models for Code: Security Hardening and Adversarial Testing Jingxuan He Martin Vechev ELM AAML 8 107 0 10 Feb 2023
Multi-lingual Evaluation of Code Generation Models Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian ... Baishakhi Ray Parminder Bhatia Sudipta Sengupta Dan Roth Bing Xiang ELM 107 117 0 26 Oct 2022
Productivity Assessment of Neural Code Completion Albert Ziegler Eirini Kalliamvakou Shawn Simister Ganesh Sittampalam Alice Li Andrew Rice Devon Rifkin E. Aftandilian 102 176 0 13 May 2022
A Systematic Evaluation of Large Language Models of Code Frank F. Xu Uri Alon Graham Neubig Vincent J. Hellendoorn ELM ALM 193 624 0 26 Feb 2022
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 189 614 0 20 May 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 186 853 0 09 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020