Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

28 November 2023

Conghui He

Papers citing "Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization"

44 / 94 papers shown

Title
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data Chenglong Wang Yang Gan Yifu Huo Yongyu Mu Murun Yang ... Chunliang Zhang Tongran Liu Quan Du Di Yang Jingbo Zhu VLM 61 4 0 22 Aug 2024
CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs Yassine Ouali Adrian Bulat Brais Martínez Georgios Tzimiropoulos VLM MLLM 22 18 0 19 Aug 2024
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models Fushuo Huo Wenchao Xu Zhong Zhang Haozhao Wang Zhicheng Chen Peilin Zhao VLM MLLM 61 18 0 04 Aug 2024
Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation Cephas Mpungu Qiyuan Chen Wei Wei Jiashuo Sun G. Mapp VLM RALM LRM 22 15 0 01 Aug 2024
Mitigating Multilingual Hallucination in Large Vision-Language Models Xiaoye Qu Mingyang Song Wei Wei Jianfeng Dong Yu Cheng VLM LRM 26 2 0 01 Aug 2024
UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models Liu Qi Yongyi He Lian Defu Zhi Zheng Tong Bill Xu Liu Che Chen Enhong MLLM 29 1 0 23 Jul 2024
DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer Jinfeng Wei Xiaofeng Zhang 21 12 0 21 Jul 2024
Clinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization Md Sultan al Nahian R. Kavuluru MedIm AI4CE 31 0 0 19 Jul 2024
Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models Yuji Zhang Sha Li Jiateng Liu Pengfei Yu Yi Ren Fung Jing Li Manling Li Heng Ji 29 10 0 10 Jul 2024
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output Pan Zhang Xiaoyi Dong Yuhang Zang Yuhang Cao Rui Qian ... Kai Chen Jifeng Dai Yu Qiao Dahua Lin Jiaqi Wang 43 98 0 03 Jul 2024
STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Guohao Sun Can Qin Huazhu Fu Linwei Wang Zhiqiang Tao LM&MA 32 3 0 28 Jun 2024
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models Mingrui Wu Jiayi Ji Oucheng Huang Jiale Li Yuhang Wu Xiaoshuai Sun Rongrong Ji 48 8 0 24 Jun 2024
Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? Gregor Geigle Radu Timofte Goran Glavas 35 0 0 20 Jun 2024
mDPO: Conditional Preference Optimization for Multimodal Large Language Models Fei Wang Wenxuan Zhou James Y. Huang Nan Xu Sheng Zhang Hoifung Poon Muhao Chen 59 15 0 17 Jun 2024
Detecting and Evaluating Medical Hallucinations in Large Vision Language Models Jiawei Chen Dingkang Yang Tong Wu Yue Jiang Xiaolu Hou Mingcheng Li Shunli Wang Dongling Xiao Ke Li Lihua Zhang LM&MA VLM 29 18 0 14 Jun 2024
On Softmax Direct Preference Optimization for Recommendation Yuxin Chen Junfei Tan An Zhang Zhengyi Yang Leheng Sheng Enzhi Zhang Xiang Wang Tat-Seng Chua 29 23 0 13 Jun 2024
CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models Junho Kim Hyunjun Kim Yeonju Kim Yong Man Ro MLLM 34 10 0 04 Jun 2024
NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models Kai Wu Boyuan Jiang Zhengkai Jiang Qingdong He Donghao Luo Shengzhi Wang Qingwen Liu Chengjie Wang VLM MLLM 25 3 0 30 May 2024
Offline Regularised Reinforcement Learning for Large Language Models Alignment Pierre Harvey Richemond Yunhao Tang Daniel Guo Daniele Calandriello M. G. Azar ... Gil Shamir Rishabh Joshi Tianqi Liu Rémi Munos Bilal Piot OffRL 40 21 0 29 May 2024
RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs Sangmin Woo Jaehyuk Jang Donguk Kim Yubin Choi Changick Kim 29 0 0 28 May 2024
Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models Sangmin Woo Donguk Kim Jaehyuk Jang Yubin Choi Changick Kim 33 12 0 28 May 2024
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song Hengtao Shen MLLM 49 10 0 24 May 2024
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement Xiyao Wang Jiuhai Chen Zhaoyang Wang Yuhang Zhou Yiyang Zhou ... Tianyi Zhou Tom Goldstein Parminder Bhatia Furong Huang Cao Xiao 58 33 0 24 May 2024
CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario Zhizhao Duan Hao Cheng Duo Xu Xi Wu Xiangxie Zhang Xi Ye Zhen Xie 24 6 0 06 May 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 80 139 0 29 Apr 2024
Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunities Xiaomin Yu Yezhaohui Wang Yanfang Chen Zhen Tao Dinghao Xi Shichao Song Simin Niu Zhiyu Li 62 7 0 25 Apr 2024
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback Wenyi Xiao Ziwei Huang Leilei Gan Wanggui He Haoyuan Li Zhelun Yu Hao Jiang Fei Wu Linchao Zhu MLLM 37 22 0 22 Apr 2024
Self-Supervised Visual Preference Alignment Ke Zhu Liang Zhao Zheng Ge Xiangyu Zhang 27 12 0 16 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 43 57 0 01 Apr 2024
Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art Neeloy Chakraborty Melkior Ornik Katherine Driggs-Campbell LRM 54 9 0 25 Mar 2024
IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding Lanyun Zhu Deyi Ji Tianrun Chen Peng Xu Jieping Ye Jun Liu MLLM 39 41 0 28 Feb 2024
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning Yiyang Zhou Chenhang Cui Rafael Rafailov Chelsea Finn Huaxiu Yao VLM MLLM 27 89 0 18 Feb 2024
EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models Shangyu Xing Fei Zhao Zhen Wu Tuo An Weihao Chen Chunhui Li Jianbing Zhang Xinyu Dai MLLM MU 42 5 0 15 Feb 2024
InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment Jianing Wang Junda Wu Yupeng Hou Yao Liu Ming Gao Julian McAuley 17 32 0 13 Feb 2024
Factuality of Large Language Models in the Year 2024 Yuxia Wang Minghan Wang Muhammad Arslan Manzoor Fei Liu Georgi Georgiev Rocktim Jyoti Das Preslav Nakov LRM HILM 30 7 0 04 Feb 2024
$Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models$ Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models Zongbo Han Zechen Bai Haiyang Mei Qianli Xu Changqing Zhang Mike Zheng Shou VLM 21 7 0 02 Feb 2024
A Survey on Hallucination in Large Vision-Language Models Hanchao Liu Wenyuan Xue Yifei Chen Dapeng Chen Xiutian Zhao Ke Wang Liping Hou Rong-Zhi Li Wei Peng LRM MLLM 14 110 0 01 Feb 2024
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model Xiao-wen Dong Pan Zhang Yuhang Zang Yuhang Cao Bin Wang ... Conghui He Xingcheng Zhang Yu Qiao Dahua Lin Jiaqi Wang VLM MLLM 76 242 0 29 Jan 2024
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences Xiyao Wang Yuhang Zhou Xiaoyu Liu Hongjin Lu Yuancheng Xu ... Taixi Lu Gedas Bertasius Mohit Bansal Huaxiu Yao Furong Huang LRM VLM 81 65 0 19 Jan 2024
LightHouse: A Survey of AGI Hallucination Feng Wang LRM HILM VLM 24 3 0 08 Jan 2024
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation Qidong Huang Xiao-wen Dong Pan Zhang Bin Wang Conghui He Jiaqi Wang Dahua Lin Weiming Zhang Neng H. Yu MLLM 26 165 0 29 Nov 2023
VIGC: Visual Instruction Generation and Correction Bin Wang Fan Wu Xiao Han Jiahui Peng Huaping Zhong ... Xiao-wen Dong Weijia Li Wei Li Jiaqi Wang Conghui He MLLM 19 61 0 24 Aug 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019