LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

IEEE Working Conference on Mining Software Repositories (MSR), 2023

16 March 2023

Catherine Tony

Markus Mutas

Nicolás E. Díaz Ferreyra

Riccardo Scandariato

ELM

ArXiv (abs)PDF HTML

Papers citing "LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations"

34 / 34 papers shown

Title
Secure Code Generation at Scale with Reflexion Arup Datta Ahmed Aljohani Hyunsook Do ELM 68 0 0 05 Nov 2025
QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback Taku Mikuriya Tatsuya Ishigaki Masayuki Kawarada Shunya Minami Tadashi Kadowaki ... Shunya Takata Takumi Kato Tamotsu Basseda Reo Yamada Hiroya Takamura ALM ELM 201 1 0 30 Oct 2025
Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies Bin Wang Y. Zhong MiDi Wan W. Yu YuanBing Ouyang Y. Huang Hui Li SILM AAML 145 0 0 27 Oct 2025
SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios Junkai Chen Huihui Huang Yunbo Lyu Junwen An Jieke Shi ... Yikun Li Zhenhao Li Xin Zhou Xing Hu David Lo ELM 112 2 0 26 Sep 2025
Detecting Stealthy Data Poisoning Attacks in AI Code Generators Cristina Improta AAML SILM 76 0 0 29 Aug 2025
From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users Sadia Sultana Chowa Riasad Alvi Subhey Sadi Rahman M. R M. R M. Islam Mukhtar Hussain Sami Azam LLMAG LM&Ro ELM 195 5 0 24 Aug 2025
Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development Sattvik Sahai Prasoon Goyal Michael Johnston Anna Gottardi Yao Lu ... Lavina Vaz Leslie Ball Maureen Murray Rahul Gupta Shankar Ananthakrishna 77 1 0 13 Aug 2025
Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective Yutao Mou Xiao Deng Yuxiao Luo Shikun Zhang Wei Ye ELM 175 3 0 15 May 2025
Frontier AI's Impact on the Cybersecurity Landscape Wenbo Guo Wenbo Guo Tianneng Shi Yu Yang Andy Zhang Patrick Gage Kelley Kurt Thomas Dawn Song Dawn Song 379 12 0 07 Apr 2025
SandboxEval: Towards Securing Test Environment for Untrusted Code Rafiqul Rabin Jesse Hostetler Sean McGregor Brett Weir Nick Judd ELM 294 3 0 27 Mar 2025
Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets Hamed Jelodar Mohammad Meymani Roozbeh Razavi-Far 202 15 0 21 Mar 2025
Rethinking the Evaluation of Secure Code Generation Shih-Chieh Dai Jun Xu Guanhong Tao ELM 280 7 0 18 Mar 2025
Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality Roham Koohestani Philippe de Bekker Begüm Koç Maliheh Izadi VLM 462 1 0 07 Mar 2025
Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models Marc Bruni Fabio Gabrielli Mohammad Ghafari Martin Kropp SILM 162 12 0 09 Feb 2025
PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback Yun Peng Akhilesh Deepak Gotmare Michael R. Lyu Caiming Xiong Silvio Savarese Doyen Sahoo 204 0 0 18 Nov 2024
From Solitary Directives to Interactive Encouragement! LLM Secure Code Generation by Natural Language Prompting Shigang Liu Bushra Sabir Seung Ick Jang Yuval Kansal Yansong Gao Kristen Moore A. Abuadbba Surya Nepal 215 4 0 18 Oct 2024
Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub ScenariosIEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2024 Zhi Chen Lingxiao Jiang LLMAG 152 5 0 16 Oct 2024
APILOT: Navigating Large Language Models to Generate Secure Code by Sidestepping Outdated API Pitfalls Weiheng Bai Keyang Xuan Pengxiang Huang Qiushi Wu Jianing Wen Jingjing Wu Kangjie Lu LLMAG KELM 134 5 0 25 Sep 2024
Dynamic Code Orchestration: Harnessing the Power of Large Language Models for Adaptive Script Execution J. D. Vecchio Andrew Perreault Eliana Furmanek 58 0 0 07 Aug 2024
Prompting Techniques for Secure Code Generation: A Systematic Investigation Catherine Tony Nicolás E. Díaz Ferreyra Markus Mutas Salem Dhiff Riccardo Scandariato SILM 334 32 0 09 Jul 2024
INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness Hung Le Yingbo Zhou Caiming Xiong Silvio Savarese Doyen Sahoo 202 6 0 23 Jun 2024
When LLMs Meet Cybersecurity: A Systematic Literature Review Jie Zhang Haoyu Bu Hui Wen Yu Chen Lun Li Hongsong Zhu 304 128 0 06 May 2024
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeInternational Conference on Learning Representations (ICLR), 2024 Naman Jain King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida I. Wang Armando Solar-Lezama Koushik Sen Ion Stoica ELM 369 867 0 12 Mar 2024
Exploring Advanced Methodologies in Security Evaluation for LLMs Junming Huang Jiawei Zhang Qi Wang Weihong Han Yanchun Zhang 249 0 0 28 Feb 2024
Ocassionally Secure: A Comparative Analysis of Code Generation Assistants Ran Elgedawy John Sadik Senjuti Dutta Senjuti Dutta Konstantinos Georgiou ... Fujiao Ji Fujiao Ji Kyungchan Lim Qian Liu Scott Ruoti 216 9 0 01 Feb 2024
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness Manav Singhal Tushar Aggarwal Abhijeet Awasthi Nagarajan Natarajan Aditya Kanade 208 23 0 29 Jan 2024
Towards Trustworthy AI Software Development Assistance Daniel Maninger Krishna Narasimhan Mira Mezini 178 5 0 14 Dec 2023
Can LLMs Patch Security Issues? Kamel Alrashedy Abdullah Aljasser Pradyumna Tambwekar Matthew Gombolay AAML 380 12 0 13 Nov 2023
Automating the Correctness Assessment of AI-generated Code for Security ContextsJournal of Systems and Software (JSS), 2023 Domenico Cotroneo Alessio Foggia Cristina Improta Pietro Liguori R. Natella 160 17 0 28 Oct 2023
LLM for SoC Security: A Paradigm ShiftIEEE Access (IEEE Access), 2023 Dipayan Saha Shams Tarek Katayoon Yahyaei S. Saha Jingbo Zhou M. Tehranipoor Farimah Farahmandi 296 78 0 09 Oct 2023
CompVPD: Iteratively Identifying Vulnerability Patches Based on Human Validation Results with a Precise Context Tianyu Chen Lin Li Taotao Qian Jingyi Liu Wei Yang Ding Li Guangtai Liang Qianxiang Wang Tao Xie 159 0 0 04 Oct 2023
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical StudyACM Transactions on Software Engineering and Methodology (TOSEM), 2023 Yujia Fu Peng Liang Amjed Tahir Zengyang Li Mojtaba Shahin Jiaxin Yu Jinfu Chen SILM 355 24 0 03 Oct 2023
Using ChatGPT as a Static Application Security Testing Tool Atieh Bakhshandeh Abdalsamad Keramatfar Amir Norouzi Mohammad Mahdi Chekidehkhoun 157 23 0 28 Aug 2023
Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning AttacksIEEE International Conference on Program Comprehension (ICPC), 2023 Domenico Cotroneo Cristina Improta Pietro Liguori R. Natella SILM 291 48 0 04 Aug 2023