RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models

6 July 2024

Hongtu Zhu

Linjun Zhang

Huaxiu Yao

MedIm

ArXiv (abs)PDF HTML HuggingFace (28 upvotes)Github (83★)

Papers citing "RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models"

41 / 41 papers shown

Title
MuISQA: Multi-Intent Retrieval-Augmented Generation for Scientific Question Answering Zhiyuan Li Haisheng Yu Guangchuan Guo Nan Zhou Jiajun Zhang RALM 176 0 0 20 Nov 2025
SkinGPT-R1: Adapter-Only Dual Distillation for Efficient Dermatology Reasoning Yuhao Shen Jiahe Qian Zhangtianyi Chen Yuanhao He Juexiao Zhou LRM 132 0 0 19 Nov 2025
BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel Hao Li Jiajun He Guangshuo Wang Dengguo Feng Zheng Li Min Zhang AAML 193 0 0 15 Nov 2025
ToM: Leveraging Tree-oriented MapReduce for Long-Context Reasoning in Large Language Models Jiani Guo Zuchao Li Jie Wu Qianren Wang Yun Li Lefei Zhang Hai Zhao Yujiu Yang VLM 113 0 0 01 Nov 2025
Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGs Lee Qi Zun Mohamad Zulhilmi Bin Abdul Halim Goh Man Fye 93 0 0 17 Oct 2025
TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models Junyi Zhang Jia-Chen Gu Wenbo Hu Yu Zhou Robinson Piramuthu Nanyun Peng VLM 222 0 0 29 Sep 2025
MedFact-R1: Towards Factual Medical Reasoning via Pseudo-Label Augmentation Gengliang Li Rongyu Chen Bin Li Linlin Yang Guodong Ding HILM MedIm LRM 84 0 0 18 Sep 2025
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey Wenbin An Jiahao Nie Yaqiang Wu Feng Tian Shijian Lu Q. Zheng MLLM 126 1 0 14 Aug 2025
MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling Jifan Gao Mahmudur Rahman John R. Caskey Madeline Oguss Ann O'Rourke ... Anne Stey Anoop Mayampurath M. Churpek Guanhua Chen Majid Afshar 116 2 0 07 Aug 2025
A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering Ziruo Yi Jinyu Liu Ting Xiao Mark V. Albert 127 0 0 04 Aug 2025
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI David Restrepo Ira Ktena Maria Vakalopoulou Stergios Christodoulidis Enzo Ferrante 80 0 0 31 Jul 2025
Transform Before You Query: A Privacy-Preserving Approach for Vector Retrieval with Embedding Space Alignment Ruiqi He Zekun Fei Jiaqi Li Xinyuan Zhu Biao Yi Siyi Lv Weijie Liu Zheli Liu LLMSV 125 0 0 24 Jul 2025
Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG Rakesh Raj Madavan Akshat Kaimal Hashim Faisal Chandrakala Shanmuganathan MedIm 57 1 0 20 Jul 2025
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment Tianyu Chen Jian Lou Wenjie Wang 99 0 0 10 Jun 2025
MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systems Peiru Yang Jinhua Yin Haoran Zheng Xueying Bai Huili Wang Yufei Sun Xintian Li Shangguang Wang Yongfeng Huang Tao Qi AAML 130 0 0 09 Jun 2025
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks Sanjoy Chowdhury Mohamed Elmoghany Yohan Abeysinghe Mahmoud Ahmed Sayan Nag Salman Khan Mohamed Elhoseiny Dinesh Manocha 253 3 0 08 Jun 2025
VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning Qiuchen Wang Ruixue Ding Y. Zeng Zehui Chen Lin Yen-Chen Shihang Wang Pengjun Xie Fei Huang Feng Zhao VLM LRM 240 23 0 28 May 2025
Learning to Route Queries Across Knowledge Bases for Step-wise Retrieval-Augmented Reasoning Chunyi Peng Zhipeng Xu Zhenghao Liu Yishan Li Shi Yu ... Zhiyuan Liu Yu Gu Minghe Yu Ge Yu Maosong Sun LRM 222 3 0 28 May 2025
A Multimodal Multi-Agent Framework for Radiology Report Generation Ziruo Yi Ting Xiao Mark V. Albert MedIm 173 5 0 14 May 2025
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal RetrievalAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Wei Yang Jingjing Fu Rongpin Wang Jinyu Wang Lei Song Jiang Bian 268 4 0 10 May 2025
FinSage: A Multi-aspect RAG System for Financial Filings Question Answering Xinyu Wang Jijun Chi Zhenghan Tai Tung Sum Thomas Kwok Muzhi Li ... Jerry Huang Jingrui Tian Fengran Mo Yufei Cui Ling Zhou 406 13 0 20 Apr 2025
M $^2$ IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering Yanshu Li Yi Cao Hongyang He Qisen Cheng Xiang Fu Xi Xiao Tianyang Wang Ruixiang Tang VLM 246 1 0 06 Apr 2025
One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image Ezzeldin Shereen Dan Ristea Shae McFadden Shae McFadden V. Mavroudis Chris Hicks 424 1 0 02 Apr 2025
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond Xiaoye Qu Yafu Li Zhaochen Su Weigao Sun Jianhao Yan ... Chaochao Lu Yue Zhang Xian-Sheng Hua Bowen Zhou Yu Cheng ReLM OffRL LRM 414 95 0 27 Mar 2025
Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook Xu Zheng Ziqiao Weng Yuanhuiyi Lyu Lutao Jiang Haiwei Xue Bin Ren Danda Pani Paudel Andrii Zadaianchuk Luc Van Gool Xuming Hu 3DV 324 23 0 23 Mar 2025
MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding S. Han Peng Xia Ruiyi Zhang Tong Sun Yun Li Hongtu Zhu Huaxiu Yao VLM 376 25 0 18 Mar 2025
Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image AnalysisInternational Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2025 Chenjun Li Laurin Lux Alexander H. Berger Fernando Navarro Mert R. Sabuncu Johannes C. Paetzold 263 3 0 12 Mar 2025
Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation Yinuo Liu Zenghui Yuan Guiyao Tie Jiawen Shi Lichao Sun Lichao Sun Neil Zhenqiang Gong 312 4 0 08 Mar 2025
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks Hyeonjeong Ha Qiusi Zhan Jeonghwan Kim Dimitrios Bralios Saikrishna Sanniboina Nanyun Peng Kai-Wei Chang Daniel Kang Heng Ji KELM AAML 311 9 0 25 Feb 2025
Adversarial Hubness in Multi-Modal Retrieval Tingwei Zhang Fnu Suya Rishi Jha Collin Zhang Vitaly Shmatikov AAML 484 2 0 18 Dec 2024
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization Kangyu Zhu Peng Xia Yun Li Hongtu Zhu Sheng Wang Huaxiu Yao 450 10 0 09 Dec 2024
Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry Wenjun Hou Yi Cheng Kaishuai Xu Yan Hu Wenjie Li Jiang-Dong Liu 161 4 0 17 Nov 2024
Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications Monica Riedler Stefan Langer VLM 213 27 0 29 Oct 2024
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Peng Xia Siwei Han Shi Qiu Yiyang Zhou Zhaoyang Wang ... Chenhang Cui Mingyu Ding Linjie Li Lijuan Wang Huaxiu Yao 251 28 0 14 Oct 2024
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases Zhaorun Chen Zhen Xiang Chaowei Xiao Dawn Song Bo Li LLMAG AAML 221 166 0 17 Jul 2024
GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM Keshav Bimbraw Ye Wang Jing Liu T. Koike-Akino VLM MedIm LM&MA 188 4 0 15 Jul 2024
Timo: Towards Better Temporal Reasoning for Language Models Zhaochen Su Jun Zhang Tong Zhu Xiaoye Qu Juntao Li Min Zhang Yu Cheng LRM 186 28 0 20 Jun 2024
CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language ModelsNeural Information Processing Systems (NeurIPS), 2024 Peng Xia Ze Chen Juanxi Tian Yangrui Gong Ruibo Hou ... Jimeng Sun Zongyuan Ge Gang Li James Zou Huaxiu Yao MU VLM 212 63 0 10 Jun 2024
Calibrated Self-Rewarding Vision Language ModelsNeural Information Processing Systems (NeurIPS), 2024 Yiyang Zhou Zhiyuan Fan Dongjie Cheng Sihan Yang Zhaorun Chen Chenhang Cui Xiyao Wang Yun Li Linjun Zhang Huaxiu Yao VLM 242 63 0 23 May 2024
MoVA: Adapting Mixture of Vision Experts to Multimodal Context Zhuofan Zong Bingqi Ma Dazhong Shen Guanglu Song Hao Shao Dongzhi Jiang Jiaming Song Yu Liu MoE 222 81 0 19 Apr 2024
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning Hao Shao Shengju Qian Han Xiao Guanglu Song Zhuofan Zong Letian Wang Yu Liu Jiaming Song VGen LRM MLLM 249 194 0 25 Mar 2024