SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

10 October 2023

Alexander Wettig

Ofir Press

Papers citing "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"

50 / 79 papers shown

Title
TRAIL: Trace Reasoning and Agentic Issue Localization Darshan Deshpande Varun Gangal Hersh Mehta Jitin Krishnan Anand Kannappan Rebecca Qian 11 0 0 13 May 2025
MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering Rushi Qiang Yuchen Zhuang Yinghao Li D. Kilman Rongzhi Zhang ... Ian Shu-Hei Wong Sherry Yang Percy Liang Chao Zhang Bo Dai ELM 18 0 0 12 May 2025
Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale Jiale Liu Yifan Zeng Shaokun Zhang Chi Zhang Malte Højmark-Bertelsen Marie Normann Gadeberg H. Wang Qingyun Wu 34 0 0 06 May 2025
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch Zimu Lu Y. Yang Houxing Ren Haotian Hou Han Xiao Ke Wang Weikang Shi Aojun Zhou Mingjie Zhan H. Li LLMAG 41 0 0 06 May 2025
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation Meng-Hao Guo Jiajun Xu Yi Zhang Jiaxi Song Haoyang Peng ... Yongming Rao Houwen Peng Han Hu Gordon Wetzstein Shi-Min Hu ELM LRM 52 0 0 04 May 2025
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks Vishnu Sarukkai Zhiqiang Xie Kayvon Fatahalian LLMAG 68 0 0 01 May 2025
Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation D. Sculley Will Cukierski Phil Culliton Sohier Dane Maggie Demkin ... Addison Howard Paul Mooney Walter Reade Megan Risdal Nate Keating 31 0 0 01 May 2025
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems Shaokun Zhang Ming Yin Jieyu Zhang J. H. Liu Zhiguang Han ... Beibin Li Chi Wang H. Wang Y. Chen Qingyun Wu 47 0 0 30 Apr 2025
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation Sizhe Wang Z. Wang Dongsheng Ma Yongan Yu Rui Ling Z. Li Feiyu Xiong W. Zhang LRM 55 0 0 30 Apr 2025
Turing Machine Evaluation for Large Language Model Haitao Wu Zongbo Han Huaxi Huang Changqing Zhang ELM LRM 59 0 0 29 Apr 2025
CrashFixer: A crash resolution agent for the Linux kernel Alex Mathai Chenxi Huang Suwei Ma Jihwan Kim Hailie Mitchell Aleksandr Nogikh Petros Maniatis Franjo Ivančić Junfeng Yang Baishakhi Ray 55 0 0 29 Apr 2025
OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification Shangyu Li Juyong Jiang Tiancheng Zhao Jiasi Shen 41 0 0 29 Apr 2025
Towards Automated Scoping of AI for Social Good Projects Jacob Emmerson Rayid Ghani Zheyuan Ryan Shi 46 0 0 28 Apr 2025
APE-Bench I: Towards File-level Automated Proof Engineering of Formal Math Libraries Huajian Xin Luming Li Xiaoran Jin Jacques Fleuriot Wenda Li AIMat 48 0 0 27 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning Z. Wang K. Wang Q. Wang Pingyue Zhang Linjie Li ... Jiajun Wu L. Fei-Fei Lijuan Wang Yejin Choi Manling Li 73 1 0 24 Apr 2025
Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark Jasper Götting Pedro Medeiros Jon G Sanders Nathaniel Li Long Phan Karam Elabd Lennart Justen Dan Hendrycks Seth Donoughe ELM 49 2 0 21 Apr 2025
Efficient Pretraining Length Scaling Bohong Wu Shen Yan Sijun Zhang Jianqiao Lu Yutao Zeng Ya Wang Xun Zhou 39 0 0 21 Apr 2025
Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use Anna Goldie Azalia Mirhoseini Hao Zhou Irene Cai Christopher D. Manning SyDa OffRL ReLM LRM 102 3 0 07 Apr 2025
Frontier AI's Impact on the Cybersecurity Landscape Wenbo Guo Yujin Potter Tianneng Shi Zhun Wang Andy Zhang Dawn Song 40 1 0 07 Apr 2025
Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems? Kai Yan Yufei Xu Zhengyin Du Xuesong Yao Z. Wang Xiaowen Guo Jiecao Chen ReLM ELM LRM 87 3 0 01 Apr 2025
L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution Simeng Sun Cheng-Ping Hsieh Faisal Ladhak Erik Arakelyan Santiago Akle Serano Boris Ginsburg ReLM ELM LRM 47 0 0 28 Mar 2025
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities Yuxuan Zhu Antony Kellermann Dylan Bowman Philip Li Akul Gupta ... Avi Dhir Sudhit Rao Kaicheng Yu Twm Stone Daniel Kang LLMAG ELM 70 1 0 21 Mar 2025
Measuring AI Ability to Complete Long Tasks Thomas Kwa Ben West Joel Becker Amy Deng Katharyn Garcia ... Lucas Jun Koba Sato H. Wijk Daniel M. Ziegler Elizabeth Barnes Lawrence Chan ELM 72 6 0 18 Mar 2025
Why Do Multi-Agent LLM Systems Fail? Mert Cemri Melissa Z. Pan Shuyi Yang Lakshya A Agrawal Bhavya Chopra ... Dan Klein Kannan Ramchandran Matei A. Zaharia Joseph E. Gonzalez Ion Stoica LLMAG Presented at ResearchTrend Connect \| LLMAG on 23 Apr 2025 116 5 0 17 Mar 2025
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning Hao Cui Zahra Shamsi Gowoon Cheon Xuejian Ma Shutong Li ... Eun-Ah Kim M. Brenner Viren Jain Sameera Ponda Subhashini Venugopalan ELM LRM 46 0 0 14 Mar 2025
LocAgent: Graph-Guided LLM Agents for Code Localization Zhaoling Chen Xiangru Tang Gangda Deng Fang Wu Jialong Wu Zhiwei Jiang Viktor Prasanna Arman Cohan Xingyao Wang LLMAG 89 2 0 12 Mar 2025
Implicit Reasoning in Transformers is Reasoning through Shortcuts Tianhe Lin Jian Xie Siyu Yuan Deqing Yang ReLM LRM 64 2 0 10 Mar 2025
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation Wei Li Xin Zhang Zhongxin Guo Shaoguang Mao Wen Luo Guangyue Peng Yangyu Huang Houfeng Wang Scarlett Li 55 0 0 09 Mar 2025
Transferable Foundation Models for Geometric Tasks on Point Cloud Representations: Geometric Neural Operators Blaine Quackenbush P. Atzberger 3DPC AI4CE 58 1 0 06 Mar 2025
CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation Peiding Wang L. Zhang Fang Liu Lin Shi Minxiao Li Bo Shen An Fu ELM LRM 48 0 0 05 Mar 2025
The Relationship Between Reasoning and Performance in Large Language Models -- o3 (mini) Thinks Harder, Not Longer Marthe Ballon Andres Algaba Vincent Ginis LRM ReLM 36 4 0 24 Feb 2025
Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents Patrick Tser Jern Kon Jiachen Liu Qiuyi Ding Yiming Qiu Zhenning Yang Yibo Huang Jayanth Srinivasa Myungjin Lee Mosharaf Chowdhury Ang Chen 48 3 0 22 Feb 2025
Measuring AI agent autonomy: Towards a scalable approach with code inspection Peter Cihon Merlin Stein Gagan Bansal Sam Manning Kevin Xu 29 0 0 21 Feb 2025
Forecasting Frontier Language Model Agent Capabilities Govind Pimpale Axel Højmark Jérémy Scheurer Marius Hobbhahn LLMAG ELM 41 1 0 21 Feb 2025
InductionBench: LLMs Fail in the Simplest Complexity Class Wenyue Hua Tyler Wong Sun Fei Liangming Pan Adam Jardine William Yang Wang LRM 63 2 0 20 Feb 2025
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?$ Samuel Miserendino M. Wang Tejal Patwardhan Johannes Heidecke 41 17 0 17 Feb 2025
Building A Proof-Oriented Programmer That Is 64% Better Than GPT-4o Under Data Scarcity Dylan Zhang Justin Wang Tianran Sun 36 0 0 17 Feb 2025
SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors Bohan Lyu Siqiao Huang Zichen Liang Qi-An Sun Jiaming Zhang ELM LRM 47 0 0 16 Feb 2025
Cardiverse: Harnessing LLMs for Novel Card Game Prototyping Danrui Li Sen Zhang Sam S. Sohn Kaidong Hu Muhammad Usman Mubbasir Kapadia 35 0 0 10 Feb 2025
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories Yijia Xiao Runhui Wang Luyang Kong Davor Golac Wei Wang LLMAG 61 0 0 10 Feb 2025
Otter: Generating Tests from Issues to Validate SWE Patches Toufique Ahmed Jatin Ganhotra Rangeet Pan Avraham Shinnar Saurabh Sinha Martin Hirzel 70 0 0 07 Feb 2025
Learning to Generate Unit Tests for Automated Debugging Archiki Prasad Elias Stengel-Eskin Justin Chih-Yao Chen Zaid Khan Mohit Bansal ELM 76 1 0 03 Feb 2025
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning Zehan Qi Xiao-Chang Liu Iat Long Iong Hanyu Lai X. Sun ... Shuntian Yao Tianjie Zhang Wei Xu J. Tang Yuxiao Dong 93 14 0 28 Jan 2025
AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds Yinfang Chen Manish Shetty Gagan Somashekar Minghua Ma Yogesh L. Simmhan Jonathan Mace Chetan Bansal Rujia Wang Saravan Rajmohan 46 0 0 12 Jan 2025
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Bradley Brown Jordan Juravsky Ryan Ehrlich Ronald Clark Quoc V. Le Christopher Ré Azalia Mirhoseini ALM LRM 76 207 0 03 Jan 2025
Cocoa: Co-Planning and Co-Execution with AI Agents K. J. Kevin Feng Kevin Pu Matt Latzke Tal August Pao Siangliulue Jonathan Bragg Daniel S. Weld Amy X. Zhang Joseph Chee Chang LM&Ro LLMAG 87 4 0 14 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 87 4 0 28 Nov 2024
Are Large Language Models Memorizing Bug Benchmarks? Daniel Ramos Claudia Mamede Kush Jain Paulo Canelas Catarina Gamboa Claire Le Goues PILM ELM 92 6 0 20 Nov 2024
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games Davide Paglieri Bartłomiej Cupiał Samuel Coward Ulyana Piterbarg Maciej Wolczyk ... Lerrel Pinto Rob Fergus Jakob Foerster Jack Parker-Holder Tim Rocktaschel LLMAG LRM 101 10 0 20 Nov 2024