Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

23 February 2023

Sahar Abdelnabi

Mario Fritz

Papers citing "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection"

50 / 289 papers shown

Title
Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications Junlin Wang Tianyi Yang Roy Xie Bhuwan Dhingra SILM AAML 29 3 0 10 Jun 2024
SecureNet: A Comparative Study of DeBERTa and Large Language Models for Phishing Detection Sakshi Mahendru Tejul Pandit 28 1 0 10 Jun 2024
Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents Avital Shafran R. Schuster Vitaly Shmatikov 37 27 0 09 Jun 2024
A Survey of Language-Based Communication in Robotics William Hunt Sarvapali D. Ramchurn Mohammad D. Soorati LM&Ro 47 12 0 06 Jun 2024
Ranking Manipulation for Conversational Search Engines Samuel Pfrommer Yatong Bai Tanmay Gautam Somayeh Sojoudi SILM 39 4 0 05 Jun 2024
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways Zehang Deng Yongjian Guo Changzhou Han Wanlun Ma Junwu Xiong Sheng Wen Yang Xiang 42 22 0 04 Jun 2024
HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model Ziyang Wang Jianzhou You Haining Wang Tianwei Yuan Shichao Lv Yang Wang Limin Sun 29 2 0 04 Jun 2024
Safeguarding Large Language Models: A Survey Yi Dong Ronghui Mu Yanghao Zhang Siqi Sun Tianle Zhang ... Yi Qi Jinwei Hu Jie Meng Saddek Bensalem Xiaowei Huang OffRL KELM AILaw 35 17 0 03 Jun 2024
BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguards Diego Dorn Alexandre Variengien Charbel-Raphaël Ségerie Vincent Corruble 24 7 0 03 Jun 2024
Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Jing Jiang Min-Bin Lin AAML 60 29 0 03 Jun 2024
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization Yu-Min Tseng Yu-Chao Huang Teng-Yun Hsiao Yu-Ching Hsu Chao-Wei Huang Jia-Yin Foo Yun-Nung Chen LLMAG 252 67 0 03 Jun 2024
BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models Jiaqi Xue Meng Zheng Yebowen Hu Fei Liu Xun Chen Qian Lou AAML SILM 16 25 0 03 Jun 2024
Teams of LLM Agents can Exploit Zero-Day Vulnerabilities Richard Fang Antony Kellermann Akul Gupta Qiusi Zhan Richard Fang R. Bindu Daniel Kang LLMAG 33 27 0 02 Jun 2024
Exfiltration of personal information from ChatGPT via prompt injection Gregory Schwartzman SILM 19 1 0 31 May 2024
Preemptive Answer "Attacks" on Chain-of-Thought Reasoning Rongwu Xu Zehan Qi Wei Xu LRM SILM 48 6 0 31 May 2024
Phantom: General Trigger Attacks on Retrieval Augmented Language Generation Harsh Chaudhari Giorgio Severi John Abascal Matthew Jagielski Christopher A. Choquette-Choo Milad Nasr Cristina Nita-Rotaru Alina Oprea SILM AAML 67 28 0 30 May 2024
Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation Maya Anderson Guy Amit Abigail Goldsteen AAML 37 13 0 30 May 2024
AI Risk Management Should Incorporate Both Safety and Security Xiangyu Qi Yangsibo Huang Yi Zeng Edoardo Debenedetti Jonas Geiping ... Chaowei Xiao Bo-wen Li Dawn Song Peter Henderson Prateek Mittal AAML 43 10 0 29 May 2024
Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges Jonas Becker Jan Philip Wahle Bela Gipp Terry Ruas 23 9 0 24 May 2024
Certifiably Robust RAG against Retrieval Corruption Chong Xiang Tong Wu Zexuan Zhong David Wagner Danqi Chen Prateek Mittal SILM 25 41 0 24 May 2024
Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data Haoran Li Xinyuan Zhao Dadi Guo Hanlin Gu Ziqian Zeng Yuxing Han Yangqiu Song Lixin Fan Qiang Yang 21 1 0 23 May 2024
Semantic-guided Prompt Organization for Universal Goal Hijacking against LLMs Yihao Huang Chong Wang Xiaojun Jia Qing-Wu Guo Felix Juefei Xu Jian Zhang G. Pu Yang Liu 30 8 0 23 May 2024
Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming Jiaxu Liu Xiangyu Yin Sihao Wu Jianhong Wang Meng Fang Xinping Yi Xiaowei Huang 29 4 0 21 May 2024
Sociotechnical Implications of Generative Artificial Intelligence for Information Access Bhaskar Mitra Henriette Cramer Olya Gurevich 40 2 0 19 May 2024
Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors Jiachen Sun Changsheng Wang Jiong Wang Yiwei Zhang Chaowei Xiao AAML VLM 26 2 0 17 May 2024
Large Language Models for Cyber Security: A Systematic Literature Review HanXiang Xu Shenao Wang Ningke Li K. Wang Yanjie Zhao Kai Chen Ting Yu Yang Janet Liu H. Wang 29 23 0 08 May 2024
Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent Shang Shang Xinqiang Zhao Zhongjiang Yao Yepeng Yao Liya Su Zijing Fan Xiaodan Zhang Zhengwei Jiang 55 3 0 06 May 2024
Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications Quan Zhang Binqi Zeng Chijin Zhou Gwihwan Go Heyuan Shi Yu Jiang SILM AAML 24 19 0 26 Apr 2024
Watch Out for Your Guidance on Generation! Exploring Conditional Backdoor Attacks against Large Language Models Jiaming He Wenbo Jiang Guanyu Hou Wenshu Fan Rui Zhang Hongwei Li AAML 48 0 0 23 Apr 2024
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions Eric Wallace Kai Y. Xiao R. Leike Lilian Weng Johannes Heidecke Alex Beutel SILM 47 114 0 19 Apr 2024
LLMs for Cyber Security: New Opportunities D. Divakaran Sai Teja Peddinti 22 11 0 17 Apr 2024
LLM Agents can Autonomously Exploit One-day Vulnerabilities Richard Fang R. Bindu Akul Gupta Daniel Kang SILM LLMAG 73 53 0 11 Apr 2024
GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications Shishir G. Patil Tianjun Zhang Vivian Fang Noppapon C Roy Huang Uc Berkeley Aaron Hao Martin Casado Joseph E. Gonzalez Raluca Ada Popa Ion Stoica ALM 24 9 0 10 Apr 2024
Goal-guided Generative Prompt Injection Attack on Large Language Models Chong Zhang Mingyu Jin Qinkai Yu Chengzhi Liu Haochen Xue Xiaobo Jin AAML SILM 34 9 0 06 Apr 2024
Fairness in Large Language Models: A Taxonomic Survey Zhibo Chu Zichong Wang Wenbin Zhang AILaw 41 32 0 31 Mar 2024
Optimization-based Prompt Injection Attack to LLM-as-a-Judge Jiawen Shi Zenghui Yuan Yinuo Liu Yue Huang Pan Zhou Lichao Sun Neil Zhenqiang Gong AAML 41 39 0 26 Mar 2024
Large Language Models for Blockchain Security: A Systematic Literature Review Zheyuan He Zihao Li Sen Yang Ao Qiao Xiaosong Zhang Xiapu Luo Ting Chen Ting Chen PILM 42 14 0 21 Mar 2024
Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices Sara Abdali Richard Anarfi C. Barberan Jia He PILM 65 24 0 19 Mar 2024
Large language models in 6G security: challenges and opportunities Tri Nguyen Huong Nguyen Ahmad Ijaz Saeid Sheikhi Athanasios V. Vasilakos Panos Kostakos ELM 22 7 0 18 Mar 2024
Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models Yi Luo Zheng-Wen Lin Yuhao Zhang Jiashuo Sun Chen Lin Chengjin Xu Xiangdong Su Yelong Shen Jian Guo Yeyun Gong LM&MA ELM ALM AI4TS 18 1 0 18 Mar 2024
Logits of API-Protected LLMs Leak Proprietary Information Matthew Finlayson Xiang Ren Swabha Swayamdipta PILM 21 21 0 14 Mar 2024
Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? Egor Zverev Sahar Abdelnabi Soroush Tabesh Mario Fritz Christoph H. Lampert 43 19 0 11 Mar 2024
Automatic and Universal Prompt Injection Attacks against Large Language Models Xiaogeng Liu Zhiyuan Yu Yizhe Zhang Ning Zhang Chaowei Xiao SILM AAML 38 33 0 07 Mar 2024
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks Dario Pasquini Martin Strohmeier Carmela Troncoso AAML 18 21 0 06 Mar 2024
Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications Stav Cohen Ron Bitton Ben Nassi 30 17 0 05 Mar 2024
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents Qiusi Zhan Zhixiang Liang Zifan Ying Daniel Kang LLMAG 44 72 0 05 Mar 2024
Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models Arijit Ghosh Chowdhury Md. Mofijul Islam Vaibhav Kumar F. H. Shezan Vaibhav Kumar Vinija Jain Aman Chadha AAML PILM 34 28 0 03 Mar 2024
A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems Fangzhou Wu Ning Zhang Somesh Jha P. McDaniel Chaowei Xiao 32 68 0 28 Feb 2024
LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History Akash Gupta Ivaxi Sheth Vyas Raina Mark J. F. Gales Mario Fritz 30 4 0 28 Feb 2024
Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems Zhenting Qi Hanlin Zhang Eric Xing Sham Kakade Hima Lakkaraju SILM 40 17 0 27 Feb 2024