Execution-Based Evaluation for Open-Domain Code Generation

Execution-Based Evaluation for Open-Domain Code Generation

20 December 2022

Daniel Fried

Graham Neubig

Papers citing "Execution-Based Evaluation for Open-Domain Code Generation"

12 / 12 papers shown

Title
Transferable Foundation Models for Geometric Tasks on Point Cloud Representations: Geometric Neural Operators Blaine Quackenbush P. Atzberger 3DPC AI4CE 55 1 0 06 Mar 2025
Game On: Towards Language Models as RL Experimenters Jingwei Zhang Thomas Lampe A. Abdolmaleki Jost Tobias Springenberg Martin Riedmiller LM&Ro 20 0 0 05 Sep 2024
What can Large Language Models Capture about Code Functional Equivalence? Nickil Maveli Antonio Vergari Shay B. Cohen 25 2 0 20 Aug 2024
Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer Mingda Li Abhijit Mishra Utkarsh Mujumdar 24 0 0 19 Aug 2024
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu ... David Lo Daniel Fried Xiaoning Du H. D. Vries Leandro von Werra 65 125 0 22 Jun 2024
CodeRAG-Bench: Can Retrieval Augment Code Generation? Zora Zhiruo Wang Akari Asai Xinyan Velocity Yu Frank F. Xu Yiqing Xie Graham Neubig Daniel Fried RALM 64 29 0 20 Jun 2024
Compositional API Recommendation for Library-Oriented Code Generation Zexiong Ma Shengnan An Bing Xie Zeqi Lin 18 16 0 29 Feb 2024
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code Shuyan Zhou Uri Alon Sumit Agarwal Graham Neubig ELM ALM 13 98 0 10 Feb 2023
Training and Evaluating a Jupyter Notebook Data Science Assistant Shubham Chandel Colin B. Clement Guillermo Serrato Neel Sundaresan 32 43 0 30 Jan 2022
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions Naihao Deng Shuaichen Chang Peng Shi Tao Yu Rui Zhang LMTD 19 4 0 15 Sep 2021
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 189 614 0 20 May 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020