NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and
Natural User Prompts

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

7 May 2024

Xiao Liu

Xiaotao Gu

Yuxiao Dong

Papers citing "NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts"

13 / 13 papers shown

Title
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch Zimu Lu Y. Yang Houxing Ren Haotian Hou Han Xiao Ke Wang Weikang Shi Aojun Zhou Mingjie Zhan H. Li LLMAG 41 0 0 06 May 2025
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? Yancheng He Shilong Li J. Liu Weixun Wang Xingyuan Bu ... Zhongyuan Peng Z. Zhang Zhicheng Zheng Wenbo Su Bo Zheng ELM LRM 74 6 0 26 Feb 2025
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?$ Samuel Miserendino M. Wang Tejal Patwardhan Johannes Heidecke 41 17 0 17 Feb 2025
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 43 1 0 09 Oct 2024
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-AI Xiao Bi : Xiao Bi Deli Chen Guanting Chen ... Yao Zhao Shangyan Zhou Shunfeng Zhou Qihao Zhu Yuheng Zou LRM ALM 139 298 0 05 Jan 2024
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages Erik Nijkamp A. Ghobadzadeh Caiming Xiong Silvio Savarese Yingbo Zhou 141 163 0 03 May 2023
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation Jiawei Liu Chun Xia Yuyao Wang Lingming Zhang ELM ALM 178 780 0 02 May 2023
Multi-lingual Evaluation of Code Generation Models Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian ... Baishakhi Ray Parminder Bhatia Sudipta Sengupta Dan Roth Bing Xiang ELM 109 117 0 26 Oct 2022
Few-shot training LLMs for project-specific code-summarization Toufique Ahmed Prem Devanbu 173 213 0 09 Jul 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
A Systematic Evaluation of Large Language Models of Code Frank F. Xu Uri Alon Graham Neubig Vincent J. Hellendoorn ELM ALM 196 624 0 26 Feb 2022
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 194 614 0 20 May 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 190 853 0 09 Feb 2021