SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

10 October 2023

Alexander Wettig

Ofir Press

Papers citing "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"

29 / 79 papers shown

Title
Human-In-the-Loop Software Development Agents Wannita Takerngsaksiri Jirat Pasuksmit Patanamon Thongtanunam C. Tantithamthavorn Ruixiong Zhang Fan Jiang Jing Li Evan Cook K. Chen Ming Wu LLMAG 95 1 0 19 Nov 2024
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows Fangyu Lei Jixuan Chen Yuxiao Ye Ruisheng Cao Dongchan Shin ... Caiming Xiong Ruoxi Sun Qian Liu Sida I. Wang Tao Yu LMTD 77 20 0 12 Nov 2024
MdEval: Massively Multilingual Code Debugging Shukai Liu Linzheng Chai Jian Yang Jiajun Shi He Zhu ... Yu Hao Liqun Yang Guanglin Niu Ge Zhang Z. Li LRM ELM 67 6 0 04 Nov 2024
A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why? QiHong Chen Jiawei Li Jiecheng Deng Jiachen Yu Justin Tian Jin Chen Iftekhar Ahmed 44 0 0 03 Nov 2024
Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina Yuan Gao Dokyun Lee Gordon Burtch Sina Fazelpour LRM 40 7 0 25 Oct 2024
Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization Guanlin Liu Kaixuan Ji Ning Dai Zheng Wu Chen Dun Q. Gu Lin Yan Quanquan Gu Lin Yan OffRL LRM 46 8 0 11 Oct 2024
FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering Siqiao Xue Tingting Chen Fan Zhou Qingyang Dai Zhixuan Chu Hongyuan Mei 28 4 0 06 Oct 2024
RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph Siru Ouyang W. Yu Kaixin Ma Zilin Xiao Z. Zhang Mengzhao Jia J. Han H. Zhang Dong Yu 47 12 0 03 Oct 2024
VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation Zhijie Wang Zhehua Zhou Jiayang Song Yuheng Huang Zhan Shu Lei Ma LM&Ro 66 5 0 19 Sep 2024
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark Zachary S. Siegel Sayash Kapoor Nitya Nagdir Benedikt Stroebl Arvind Narayanan 27 8 0 17 Sep 2024
SuperCoder2.0: Technical Report on Exploring the feasibility of LLMs as Autonomous Programmer Anmol Gautam Kishore Kumar Adarsh Jha Mukunda NS Ishaan Bhola 48 1 0 17 Sep 2024
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls Kinjal Basu Ibrahim Abdelaziz Kelsey Bradford M. Crouse Kiran Kate ... Yara Rizk Xin Wang Luis A. Lastras Pavan Kapanipathi Pavan Kapanipathi 31 7 0 04 Sep 2024
What can Large Language Models Capture about Code Functional Equivalence? Nickil Maveli Antonio Vergari Shay B. Cohen 25 2 0 20 Aug 2024
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates Zeyu Leo Liu Shrey Pandit Xi Ye Eunsol Choi Greg Durrett KELM ALM 57 4 0 08 Jul 2024
AI Agents That Matter Sayash Kapoor Benedikt Stroebl Zachary S. Siegel Nitya Nadgir Arvind Narayanan 38 32 0 01 Jul 2024
Agentless: Demystifying LLM-based Software Engineering Agents Chunqiu Steven Xia Yinlin Deng Soren Dunn Lingming Zhang LLMAG 32 78 0 01 Jul 2024
On the Transformations across Reward Model, Parameter Update, and In-Context Prompt Deng Cai Huayang Li Tingchen Fu Siheng Li Weiwen Xu ... Leyang Cui Yan Wang Lemao Liu Taro Watanabe Shuming Shi KELM 26 2 0 24 Jun 2024
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu ... David Lo Daniel Fried Xiaoning Du H. D. Vries Leandro von Werra 65 125 0 22 Jun 2024
CodeRAG-Bench: Can Retrieval Augment Code Generation? Zora Zhiruo Wang Akari Asai Xinyan Velocity Yu Frank F. Xu Yiqing Xie Graham Neubig Daniel Fried RALM 67 29 0 20 Jun 2024
Guiding LLM Temporal Logic Generation with Explicit Separation of Data and Control William Murphy Nikolaus Holzer Nathan Koenig Leyi Cui Raven Rothkopf Feitong Qiao Mark Santolucito 11 1 0 11 Jun 2024
Kotlin ML Pack: Technical Report Sergey Titov Mikhail Evtikhiev Anton Shapkin Oleg Smirnov Sergei Boytsov ... Dariia Karaeva Maksim Sheptyakov Mikhail Arkhipov T. Bryksin Egor Bogomolov 24 0 0 29 May 2024
IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities Ziyang Li Saikat Dutta Mayur Naik 40 40 0 27 May 2024
AIOS: LLM Agent Operating System Kai Mei Zelong Li Wujiang Xu Wenyue Hua Mingyu Jin Yongfeng Zhang Shuyuan Xu Ruosong Ye Yingqiang Ge Yongfeng Zhang LLMAG 26 17 0 25 Mar 2024
DebugBench: Evaluating Debugging Capability of Large Language Models Runchu Tian Yining Ye Yujia Qin Xin Cong Yankai Lin ... Yesai Wu Haotian Hui Weichuan Liu Zhiyuan Liu Maosong Sun ELM 21 28 0 09 Jan 2024
Bias Testing and Mitigation in LLM-based Code Generation Dong Huang Qingwen Bu Jie M. Zhang Xiaofei Xie Junjie Chen Heming Cui 33 20 0 03 Sep 2023
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation Jiawei Liu Chun Xia Yuyao Wang Lingming Zhang ELM ALM 178 780 0 02 May 2023
Multi-lingual Evaluation of Code Generation Models Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian ... Baishakhi Ray Parminder Bhatia Sudipta Sengupta Dan Roth Bing Xiang ELM 109 117 0 26 Oct 2022
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 194 614 0 20 May 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 190 853 0 09 Feb 2021