Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 586 papers shown

Title
Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models Weiyi Wu Xinwen Xu Chongyang Gao Xingjian Diao Siting Li Lucas A. Salas Jiang Gui 9 0 0 12 May 2025
DanceGRPO: Unleashing GRPO on Visual Generation Zeyue Xue Jie Wu Yu Gao Fangyuan Kong Lingting Zhu ... Zhiheng Liu Wei Liu Qiushan Guo Weilin Huang Ping Luo EGVM VGen 37 0 0 12 May 2025
Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation Stefan Vasilev Christian Herold Baohao Liao Seyyed Hadi Hashemi Shahram Khadivi Christof Monz MU 49 0 0 09 May 2025
G-FOCUS: Towards a Robust Method for Assessing UI Design Persuasiveness Jaehyun Jeon Janghan Yoon Minsoo Kim Sumin Shim Yejin Choi Hanbin Kim Youngjae Yu AAML 33 0 0 08 May 2025
Flow-GRPO: Training Flow Matching Models via Online RL Jie Liu Gongye Liu Jiajun Liang Y. Li Jiaheng Liu X. Wang Pengfei Wan Di Zhang Wanli Ouyang AI4CE 66 0 0 08 May 2025
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment Xueyao Zhang Y. Wang Chaoren Wang Z. Li Zhuo Chen Zhizheng Wu 41 0 0 07 May 2025
LLAMAPIE: Proactive In-Ear Conversation Assistants Tuochao Chen Nicholas Batchelder Alisa Liu Noah A. Smith Shyamnath Gollakota 36 0 0 07 May 2025
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Z. Wang Shi Wang Qianqian Xu Q. Huang 37 0 0 07 May 2025
GASCADE: Grouped Summarization of Adverse Drug Event for Enhanced Cancer Pharmacovigilance Sofia Jamil Aryan Dabad Bollampalli Areen Reddy S. Saha Rajiv Misra Adil A. Shakur 46 0 0 07 May 2025
Guide your favorite protein sequence generative model Junhao Xiong Hunter Nisonoff Ishan Gaur Jennifer Listgarten DiffM 48 0 0 07 May 2025
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models Xiaoyu Xu Minxin Du Qingqing Ye Haibo Hu MU 50 0 0 07 May 2025
EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning Zhenghao Xing Xiaowei Hu Chi-Wing Fu W. Wang Jifeng Dai Pheng-Ann Heng MLLM OffRL VLM LRM 38 0 0 07 May 2025
On-Device LLM for Context-Aware Wi-Fi Roaming Ju-Hyung Lee Yanqing Lu 16 0 0 07 May 2025
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch Zimu Lu Y. Yang Houxing Ren Haotian Hou Han Xiao Ke Wang Weikang Shi Aojun Zhou Mingjie Zhan H. Li LLMAG 41 0 0 06 May 2025
Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning Yibin Wang Zhimin Li Yuhang Zang Chunyu Wang Qinglin Lu Cheng Jin J. T. Wang LRM 42 0 0 06 May 2025
Soft Best-of-n Sampling for Model Alignment C. M. Verdun Alex Oesterling Himabindu Lakkaraju Flavio du Pin Calmon BDL 53 0 0 06 May 2025
RIFT: Closed-Loop RL Fine-Tuning for Realistic and Controllable Traffic Simulation Keyu Chen Wenchao Sun Hao Cheng Sifa Zheng 35 0 0 06 May 2025
VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making Jake Grigsby Yuke Zhu Michael S Ryoo Juan Carlos Niebles OffRL VLM 31 0 0 06 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 22 0 0 05 May 2025
SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing Ming Li Xin Gu Fan Chen X. Xing Longyin Wen C. L. P. Chen Sijie Zhu DiffM 71 1 0 05 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 50 0 0 05 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 60 0 0 05 May 2025
RM-R1: Reward Modeling as Reasoning X. Chen Gaotang Li Z. Wang Bowen Jin Cheng Qian ... Y. Zhang D. Zhang Tong Zhang Hanghang Tong Heng Ji ReLM OffRL LRM 49 0 0 05 May 2025
What Is AI Safety? What Do We Want It to Be? Jacqueline Harding Cameron Domenico Kirk-Giannini 53 0 0 05 May 2025
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL Jiarui Yao Yifan Hao Hanning Zhang Hanze Dong Wei Xiong Nan Jiang Tong Zhang LRM 47 0 0 05 May 2025
R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning Yi-Fan Zhang Xingyu Lu X. Hu Chaoyou Fu Bin Wen ... J. Chen Fan Yang Z. Zhang Tingting Gao Liang Wang OffRL LRM 34 0 0 05 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 31 0 0 05 May 2025
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra S. Song Ce Zhang James Y. Zou ALM 24 0 0 05 May 2025
FairPO: Robust Preference Optimization for Fair Multi-Label Learning Soumen Kumar Mondal Akshit Varmora Prateek Chanda Ganesh Ramakrishnan 25 0 0 05 May 2025
Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study Xiaoyu Tian Sitong Zhao Haotian Wang Shuaiting Chen Yiping Peng Yunjie Ji Han Zhao Xiangang Li OffRL LRM 27 0 0 04 May 2025
Semantic Probabilistic Control of Language Models Kareem Ahmed Catarina G Belém Padhraic Smyth Sameer Singh 35 0 0 04 May 2025
Demystifying optimized prompts in language models Rimon Melamed Lucas H. McCabe H. H. Huang 34 0 0 04 May 2025
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Jiancong Xiao Bojian Hou Zhanliang Wang Ruochen Jin Q. Long Weijie Su Li Shen 28 0 0 04 May 2025
R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation Meng-Hao Guo Jiajun Xu Yi Zhang Jiaxi Song Haoyang Peng ... Yongming Rao Houwen Peng Han Hu Gordon Wetzstein Shi-Min Hu ELM LRM 52 0 0 04 May 2025
LookAlike: Consistent Distractor Generation in Math MCQs Nisarg Parikh Nigel Fernandez Alexander Scarlatos Simon Woodhead Andrew S. Lan 37 0 0 03 May 2025
On the Limitations of Steering in Language Model Alignment Chebrolu Niranjan Kokil Jaidka G. Yeo LLMSV 36 0 0 02 May 2025
SmallPlan: Leverage Small Language Models for Sequential Path Planning with Simulation-Powered, LLM-Guided Distillation Quang P.M. Pham Khoi T.N. Nguyen Nhi H. Doan Cuong Pham Kentaro Inui Dezhen Song 54 0 0 01 May 2025
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks Vishnu Sarukkai Zhiqiang Xie Kayvon Fatahalian LLMAG 68 0 0 01 May 2025
Optimal Interactive Learning on the Job via Facility Location Planning Shivam Vats Michelle Zhao Patrick Callaghan Mingxi Jia Maxim Likhachev Oliver Kroemer George Konidaris 29 0 0 01 May 2025
Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning Lang Feng Weihao Tan Zhiyi Lyu Longtao Zheng Haiyang Xu M. Yan Fei Huang Bo An 20 0 0 01 May 2025
AdaR1: From Long-CoT to Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization H. Luo Haiying He Y. Wang Jinluan Yang Rui Liu Naiqiang Tan Xiaochun Cao Dacheng Tao Li Shen LRM 26 0 0 30 Apr 2025
WebThinker: Empowering Large Reasoning Models with Deep Research Capability X. Li Jiajie Jin Guanting Dong Hongjin Qian Yutao Zhu Yongkang Wu Ji-Rong Wen Zhicheng Dou LLMAG LRM 85 1 0 30 Apr 2025
BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models Zhiting Fan Ruizhe Chen Zuozhu Liu 44 0 0 30 Apr 2025
A Domain-Agnostic Scalable AI Safety Ensuring Framework Beomjun Kim Kangyeon Kim Sunwoo Kim Heejin Ahn 38 0 0 29 Apr 2025
HyPerAlign: Hypotheses-driven Personalized Alignment Cristina Garbacea Chenhao Tan 44 0 0 29 Apr 2025
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception Yuanchen Wu Lu Zhang Hang Yao Junlong Du Ke Yan Shouhong Ding Yunsheng Wu X. Li MLLM 68 0 0 29 Apr 2025
CachePrune: Neural-Based Attribution Defense Against Indirect Prompt Injection Attacks Rui Wang Junda Wu Yu Xia Tong Yu R. Zhang Ryan Rossi Lina Yao Julian McAuley AAML SILM 46 0 0 29 Apr 2025
Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think Hasan Hammoud Hani Itani Bernard Ghanem ReLM LRM 69 0 0 29 Apr 2025
m-KAILIN: Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training Meng Xiao Xunxin Cai Chengrui Wang Yuanchun Zhou 48 0 0 28 Apr 2025
GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets Mingqian He Fei Zhao Chonggang Lu Z. Liu Y. Wang Haofu Qian OffRL AI4TS VLM 64 0 0 28 Apr 2025