Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation

14 March 2024

Yunhao Gou

Kai Chen

Zhili Liu

Lanqing Hong

Hang Xu

Papers citing "Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation"

21 / 21 papers shown

Title
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models J. Liu Hangyu Guo Ranjie Duan Xingyuan Bu Yancheng He ... Yingshui Tan Yanan Wu Jihao Gu Y. Li J. Zhu MLLM 58 0 0 25 Apr 2025
CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models Xiangyu Yin Jiaxu Liu Zhen Chen Jinwei Hu Yi Dong Xiaowei Huang Wenjie Ruan AAML 41 0 0 08 Mar 2025
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment Zhili Liu Yunhao Gou Kai Chen Lanqing Hong Jiahui Gao ... Yu Zhang Zhenguo Li Xin Jiang Q. Liu James T. Kwok MoE 93 9 0 20 Feb 2025
Understanding and Rectifying Safety Perception Distortion in VLMs Xiaohan Zou Jian Kang George Kesidis Lu Lin 87 0 0 18 Feb 2025
UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models Sejoon Oh Yiqiao Jin Megha Sharma Donghyun Kim Eric Ma Gaurav Verma Srijan Kumar 43 5 0 03 Nov 2024
Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models Shicheng Xu Liang Pang Yunchang Zhu Huawei Shen Xueqi Cheng MLLM 33 1 0 16 Oct 2024
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time Yi Ding Bolian Li Ruqi Zhang MLLM 49 4 0 09 Oct 2024
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models Muhammad Jehanzeb Mirza Mengjie Zhao Zhuoyuan Mao Sivan Doveh Wei Lin ... Yuki Mitsufuji Horst Possegger Rogerio Feris Leonid Karlinsky James Glass VLM 69 1 0 08 Oct 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 48 21 0 26 Sep 2024
BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger Yulin Chen Haoran Li Zihao Zheng Zihao Zheng Yangqiu Song Bryan Hooi 27 6 0 17 Aug 2024
DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception Yibo Wang Ruiyuan Gao Kai Chen Kaiqiang Zhou Yingjie Cai ... Zhenguo Li Lihui Jiang Dit-Yan Yeung Qiang Xu Kai Zhang DiffM 100 6 0 20 Mar 2024
Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts Zhili Liu Kai Chen Jianhua Han Lanqing Hong Hang Xu Zhenguo Li James T. Kwok MoE 101 19 0 08 Feb 2024
Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models Yongshuo Zong Ondrej Bohdal Tingyang Yu Yongxin Yang Timothy M. Hospedales VLM MLLM 49 56 0 03 Feb 2024
Red Teaming Visual Language Models Mukai Li Lei Li Yuwei Yin Masood Ahmed Zhenguang Liu Qi Liu VLM 25 11 0 23 Jan 2024
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs Penghao Wu Saining Xie LRM 42 120 0 21 Dec 2023
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts Yichen Gong Delong Ran Jinyuan Liu Conglei Wang Tianshuo Cong Anyu Wang Sisi Duan Xiaoyun Wang MLLM 127 116 0 09 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 114 367 0 07 Nov 2023
On the Adversarial Robustness of Multi-Modal Foundation Models Christian Schlarmann Matthias Hein AAML 90 84 0 21 Aug 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 198 1,089 0 20 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 380 4,010 0 28 Jan 2022