Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

28 November 2023

Conghui He

Papers citing "Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization"

50 / 94 papers shown

Title
Critique Before Thinking: Mitigating Hallucination through Rationale-Augmented Instruction Tuning Zexian Yang Dian Li Dayan Wu Gang Liu Weiping Wang MLLM LRM 36 0 0 12 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 60 0 0 05 May 2025
Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models Sangmin Woo Kang Zhou Yun Zhou Shuai Wang Sheng Guan Haibo Ding Lin Lee Cheong VPVLM 81 0 0 30 Apr 2025
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception Yuanchen Wu Lu Zhang Hang Yao Junlong Du Ke Yan Shouhong Ding Yunsheng Wu X. Li MLLM 68 0 0 29 Apr 2025
AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization Jinda Lu Jinghan Li Yuan Gao Junkang Wu Jiancan Wu X. Wang Xiangnan He 34 0 0 22 Apr 2025
VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models Haojian Huang Haodong Chen Shengqiong Wu Meng Luo Jinlan Fu Xinya Du H. Zhang Hao Fei AI4TS 63 0 0 17 Apr 2025
TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection C. Xie Tongxuan Liu Lei Jiang Yuting Zeng J. Guo Yunheng Shen Weizhe Huang Jing Li Xiaohua Xu VLM 56 0 0 05 Apr 2025
Q-Insight: Understanding Image Quality via Visual Reinforcement Learning Weiqi Li X. Zhang Shijie Zhao Y. Zhang Junlin Li Li Zhang Jian Andrew Zhang 46 3 0 28 Mar 2025
Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMs Zitian Wang Yue Liao Kang Rong Fengyun Rao Yibo Yang Si Liu 70 0 0 26 Mar 2025
Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategy Yinan Sun Xiongkuo Min Zicheng Zhang Yixuan Gao Y. Cao Guangtao Zhai VLM 59 0 0 26 Mar 2025
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization Zefeng Zhang Hengzhu Tang Jiawei Sheng Zhenyu Zhang Yiming Ren Zhenyang Li Dawei Yin Duohe Ma Tingwen Liu 43 0 0 23 Mar 2025
Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations Shuo Li Jiajun Sun Guodong Zheng Xiaoran Fan Yujiong Shen ... Wenming Tan Tao Ji Tao Gui Qi Zhang Xuanjing Huang AAML VLM 83 0 0 19 Mar 2025
DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning R. Zhao Junliang Ye Z. Wang Guangce Liu Yiwen Chen Yikai Wang J. Zhu AI4CE 45 0 0 19 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Y. Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang T. Tan 73 2 0 18 Mar 2025
Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song H. Shen 63 1 0 11 Mar 2025
Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning Qinghao Ye Xianhan Zeng Fu Li C. Li Haoqi Fan CoGe 80 0 0 10 Mar 2025
SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner Kejia Chen Jiawen Zhang Jiacong Hu Jiazhen Yang Jian Lou Zunlei Feng Mingli Song 53 0 0 06 Mar 2025
Visual-RFT: Visual Reinforcement Fine-Tuning Ziyu Liu Zeyi Sun Yuhang Zang Xiaoyi Dong Y. Cao Haodong Duan D. Lin Jiaqi Wang ObjD VLM LRM 70 40 0 03 Mar 2025
Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding Wei Suo Lijun Zhang Mengyang Sun Lin Yuanbo Wu Peng Wang Y. Zhang MLLM VLM 47 1 0 01 Mar 2025
HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioning Maria Lymperaiou Giorgos Filandrianos Angeliki Dimitriou Athanasios Voulodimos Giorgos Stamou MLLM 35 0 0 01 Mar 2025
Towards Statistical Factuality Guarantee for Large Vision-Language Models Z. Li Chao Yan Nicholas J. Jackson Wendi Cui B. Li Jiaxin Zhang Bradley Malin 67 0 0 27 Feb 2025
Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models Yaqi Sun Kyohei Atarashi Koh Takeuchi Hisashi Kashima MLLM 46 0 0 24 Feb 2025
Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding Kyungmin Min Minbeom Kim Kang-il Lee Dongryeol Lee Kyomin Jung MLLM 67 3 0 20 Feb 2025
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Ziyu Liu ... Haodong Duan W. Zhang Kai Chen D. Lin Jiaqi Wang VLM 68 17 0 21 Jan 2025
Supervision-free Vision-Language Alignment Giorgio Giannone Ruoteng Li Qianli Feng Evgeny Perevodchikov Rui Chen Aleix M. Martinez VLM 58 0 0 08 Jan 2025
Multimodal Preference Data Synthetic Alignment with Reward Model Robert Wijaya Ngoc-Bao Nguyen Ngai-man Cheung MLLM SyDa 47 0 0 23 Dec 2024
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence Jinghan He Kuan Zhu Haiyun Guo Junfeng Fang Zhenglin Hua Yuheng Jia Ming Tang Tat-Seng Chua J. T. Wang VLM 72 0 0 18 Dec 2024
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization Kangyu Zhu Peng Xia Yun-Qing Li Hongtu Zhu Sheng Wang Huaxiu Yao 95 1 0 09 Dec 2024
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan S. Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 96 14 0 03 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 95 1 0 03 Dec 2024
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach Shijian Deng Wentian Zhao Yu-Jhe Li Kun Wan Daniel Miranda Ajinkya Kale Yapeng Tian LRM 67 0 0 26 Nov 2024
Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models Hao Yi Qingyang Li Y. Hu Fuzheng Zhang Di Zhang Yong Liu VGen 67 0 0 25 Nov 2024
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models Junzhe Chen Tianshu Zhang S. Huang Yuwei Niu Linfeng Zhang Lijie Wen Xuming Hu MLLM VLM 102 1 0 22 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie X. Sun Zhanhui Kang Xirong Li MLLM 33 3 0 15 Nov 2024
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 28 10 0 05 Nov 2024
Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models J. Wu Tsz Ting Chung Kai Chen Dit-Yan Yeung VLM LRM 47 3 0 30 Oct 2024
Vision-Language Models Can Self-Improve Reasoning via Reflection Kanzhi Cheng Yantao Li Fangzhi Xu Jianbing Zhang Hao Zhou Yang Liu ReLM LRM 44 16 0 30 Oct 2024
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models Ziyu Liu Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Haodong Duan Conghui He Yuanjun Xiong Dahua Lin Jiaqi Wang 27 7 0 23 Oct 2024
Reducing Hallucinations in Vision-Language Models via Latent Space Steering Sheng Liu Haotian Ye Lei Xing James Zou VLM LLMSV 31 5 0 21 Oct 2024
A Survey of Hallucination in Large Visual Language Models Wei Lan Wenyi Chen Qingfeng Chen Shirui Pan Huiyu Zhou Yi-Lun Pan LRM 28 4 0 20 Oct 2024
Modality-Fair Preference Optimization for Trustworthy MLLM Alignment Songtao Jiang Yan Zhang Ruizhe Chen Yeying Jin Zuozhu Liu MLLM MoE 19 6 0 20 Oct 2024
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment Chenhang Cui An Zhang Yiyang Zhou Zhaorun Chen Gelei Deng Huaxiu Yao Tat-Seng Chua 49 4 0 18 Oct 2024
Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs Shuo Li Tao Ji Xiaoran Fan Linsheng Lu L. Yang ... Y. Wang Xiaohui Zhao Tao Gui Qi Zhang Xuanjing Huang 40 0 0 15 Oct 2024
VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang L. Chen Yazheng Yang Benyou Wang Lingpeng Kong Q. Liu VLM ALM 34 16 0 12 Oct 2024
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models Qin Liu Chao Shang Ling Liu Nikolaos Pappas Jie Ma Neha Anna John Srikanth Doss Kadarundalagi Raghuram Doss Lluís Marquez Miguel Ballesteros Yassine Benajiba 34 3 0 11 Oct 2024
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models Yuying Shang Xinyi Zeng Yutao Zhu Xiao Yang Zhengwei Fang Jingyuan Zhang Jiawei Chen Zinan Liu Yu Tian VLM MLLM 25 1 0 09 Oct 2024
Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models Yufang Liu Tao Ji Changzhi Sun Yuanbin Wu Aimin Zhou VLM MLLM 28 1 0 04 Oct 2024
HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding Fan Yuan Chi Qin Xiaogang Xu Piji Li VLM MLLM 17 4 0 30 Sep 2024
Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding Xiaoyu Liang Jiayuan Yu Lianrui Mu Jiedong Zhuang Jiaqi Hu Yuchen Yang Jiangnan Ye Lu Lu Jian Chen Haoji Hu VLM 35 0 0 10 Sep 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 27 8 0 28 Aug 2024