Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

23 May 2023

Yi Liu

Lida Zhao

Kailong Wang

Yang Liu

ArXiv PDF HTML

Papers citing "Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study"

42 / 42 papers shown

Title
LM-Scout: Analyzing the Security of Language Model Integration in Android Apps Muhammad Ibrahim Gűliz Seray Tuncay Z. Berkay Celik Aravind Machiry Antonio Bianchi 16 0 0 13 May 2025
Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs Chetan Pathade AAML SILM 46 0 0 07 May 2025
Attack and defense techniques in large language models: A survey and new perspectives Zhiyu Liao Kang Chen Yuanguo Lin Kangkang Li Yunxuan Liu Hefeng Chen Xingwang Huang Yuanhui Yu AAML 54 0 0 02 May 2025
RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models Bang An Shiyue Zhang Mark Dredze 54 0 0 25 Apr 2025
StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation Shenyang Liu Yang Gao Shaoyan Zhai Liqiang Wang 27 0 0 06 Apr 2025
sudo rm -rf agentic_security Sejin Lee Jian Kim Haon Park Ashkan Yousefpour Sangyoon Yu Min Song AAML 85 0 0 26 Mar 2025
Single-pass Detection of Jailbreaking Input in Large Language Models Leyla Naz Candogan Yongtao Wu Elias Abad Rocamora Grigorios G. Chrysos V. Cevher AAML 45 0 0 24 Feb 2025
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks Ang Li Yin Zhou Vethavikashini Chithrra Raghuram Tom Goldstein Micah Goldblum AAML 59 7 0 12 Feb 2025
When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search Xuan Chen Yuzhou Nie Wenbo Guo Xiangyu Zhang 103 9 0 28 Jan 2025
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models Jingwei Yi Yueqi Xie Bin Zhu Emre Kiciman Guangzhong Sun Xing Xie Fangzhao Wu AAML 47 61 0 28 Jan 2025
ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models H. Zhang Hongfu Gao Qiang Hu Guanhua Chen L. Yang Bingyi Jing Hongxin Wei Bing Wang Haifeng Bai Lei Yang AILaw ELM 47 1 0 24 Oct 2024
Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks Samuele Poppi Zheng-Xin Yong Yifei He Bobbie Chern Han Zhao Aobo Yang Jianfeng Chi AAML 43 11 0 23 Oct 2024
SPIN: Self-Supervised Prompt INjection Leon Zhou Junfeng Yang Chengzhi Mao AAML SILM 20 0 0 17 Oct 2024
RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process Peiran Wang Xiaogeng Liu Chaowei Xiao AAML 21 3 0 11 Oct 2024
Do Unlearning Methods Remove Information from Language Model Weights? Aghyad Deeb Fabien Roger AAML MU 40 11 0 11 Oct 2024
Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond Shanshan Han 64 1 0 09 Oct 2024
PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs Jiahao Yu Yangguang Shao Hanwen Miao Junzheng Shi SILM AAML 60 3 0 23 Sep 2024
Prompt Obfuscation for Large Language Models David Pape Thorsten Eisenhofer Thorsten Eisenhofer Lea Schönherr AAML 31 2 0 17 Sep 2024
Differentially Private Kernel Density Estimation Erzhi Liu Jerry Yao-Chieh Hu Alex Reneau Zhao Song Han Liu 47 3 0 03 Sep 2024
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification Boyang Zhang Yicong Tan Yun Shen Ahmed Salem Michael Backes Savvas Zannettou Yang Zhang LLMAG AAML 38 12 0 30 Jul 2024
Merging Improves Self-Critique Against Jailbreak Attacks Victor Gallego AAML MoMe 34 3 0 11 Jun 2024
Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents Avital Shafran R. Schuster Vitaly Shmatikov 37 27 0 09 Jun 2024
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner Xunguang Wang Daoyuan Wu Zhenlan Ji Zongjie Li Pingchuan Ma Shuai Wang Yingjiu Li Yang Liu Ning Liu Juergen Rahmel AAML 64 6 0 08 Jun 2024
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models Chia-Yi Hsu Yu-Lin Tsai Chih-Hsun Lin Pin-Yu Chen Chia-Mu Yu Chun-ying Huang 38 30 0 27 May 2024
Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward Xuan Xie Jiayang Song Zhehua Zhou Yuheng Huang Da Song Lei Ma OffRL 35 6 0 12 Apr 2024
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack M. Russinovich Ahmed Salem Ronen Eldan 26 75 0 02 Apr 2024
ImgTrojan: Jailbreaking Vision-Language Models with ONE Image Xijia Tao Shuai Zhong Lei Li Qi Liu Lingpeng Kong 30 25 0 05 Mar 2024
Can Large Language Models Detect Misinformation in Scientific News Reporting? Yupeng Cao Aishwarya Muralidharan Nair Elyon Eyimife Nastaran Jamalipour Soofi K. P. Subbalakshmi J. Wullert Chumki Basu David Shallcross 19 8 0 22 Feb 2024
Comprehensive Assessment of Jailbreak Attacks Against LLMs Junjie Chu Yugeng Liu Ziqing Yang Xinyue Shen Michael Backes Yang Zhang AAML 22 65 0 08 Feb 2024
Black-Box Access is Insufficient for Rigorous AI Audits Stephen Casper Carson Ezell Charlotte Siegmann Noam Kolt Taylor Lynn Curtis ... Michael Gerovitch David Bau Max Tegmark David M. Krueger Dylan Hadfield-Menell AAML 13 75 0 25 Jan 2024
All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks Kazuhiro Takemoto 32 21 0 18 Jan 2024
AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models Dong Shu Mingyu Jin Suiyuan Zhu Beichen Wang Zihao Zhou Chong Zhang Yongfeng Zhang ELM 37 12 0 17 Jan 2024
Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations Wenjie Mo Jiashu Xu Qin Liu Jiong Wang Jun Yan Chaowei Xiao Muhao Chen Muhao Chen AAML 42 17 0 16 Nov 2023
Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization Zhexin Zhang Junxiao Yang Pei Ke Fei Mi Hongning Wang Minlie Huang AAML 16 113 0 15 Nov 2023
Fake Alignment: Are LLMs Really Aligned Well? Yixu Wang Yan Teng Kexin Huang Chengqi Lyu Songyang Zhang Wenwei Zhang Xingjun Ma Yu-Gang Jiang Yu Qiao Yingchun Wang 17 14 0 10 Nov 2023
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts Yichen Gong Delong Ran Jinyuan Liu Conglei Wang Tianshuo Cong Anyu Wang Sisi Duan Xiaoyun Wang MLLM 129 116 0 09 Nov 2023
Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection Beizhe Hu Qiang Sheng Juan Cao Yuhui Shi Yang Li Danding Wang Peng Qi 24 80 0 21 Sep 2023
Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis He Zhang Chuhao Wu Jingyi Xie Yao Lyu Jie Cai John M. Carroll 19 48 0 19 Sep 2023
PentestGPT: An LLM-empowered Automatic Penetration Testing Tool Gelei Deng Yi Liu Víctor Mayoral-Vilches Peng Liu Yuekang Li Yuan Xu Tianwei Zhang Yang Liu M. Pinzger Stefan Rass LLMAG 20 78 0 13 Aug 2023
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models Erfan Shayegani Yue Dong Nael B. Abu-Ghazaleh 13 126 0 26 Jul 2023
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots Gelei Deng Yi Liu Yuekang Li Kailong Wang Ying Zhang Zefeng Li Haoyu Wang Tianwei Zhang Yang Liu SILM 22 118 0 16 Jul 2023
A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need? Chaoning Zhang Chenshuang Zhang Sheng Zheng Yu Qiao Chenghao Li ... Lik-Hang Lee Yang Yang Heng Tao Shen In So Kweon Choong Seon Hong 72 152 0 21 Mar 2023