Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 627 papers shown

Title
Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator Kaiwen Zheng Yongxin Chen Huayu Chen Guande He Ming-Yu Liu J. Zhu Qinsheng Zhang DiffM 47 0 0 03 Mar 2025
DPR: Diffusion Preference-based Reward for Offline Reinforcement Learning Teng Pang Bingzheng Wang Guoqiang Wu Yilong Yin OffRL 65 0 0 03 Mar 2025
VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation Wenhao Wang Y. Yang DiffM VGen 84 0 0 03 Mar 2025
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 35 0 0 03 Mar 2025
A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning Shashank Gupta Chaitanya Ahuja Tsung-Yu Lin Sreya Dutta Roy Harrie Oosterhuis Maarten de Rijke Satya Narayan Shukla 44 1 0 02 Mar 2025
Personalize Your LLM: Fake it then Align it Yijing Zhang Dyah Adila Changho Shin Frederic Sala 86 0 0 02 Mar 2025
SFO: Piloting VLM Feedback for Offline RL Jacob Beck OffRL 31 0 0 02 Mar 2025
Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Zachary Yahn Yichang Xu Ling Liu 53 8 0 01 Mar 2025
Foot-In-The-Door: A Multi-turn Jailbreak for LLMs Zixuan Weng Xiaolong Jin Jinyuan Jia X. Zhang AAML 46 0 0 27 Feb 2025
Societal Alignment Frameworks Can Improve LLM Alignment Karolina Stañczak Nicholas Meade Mehar Bhatia Hattie Zhou Konstantin Böttinger ... Timothy P. Lillicrap Ana Marasović Sylvie Delacroix Gillian K. Hadfield Siva Reddy 59 0 0 27 Feb 2025
LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces Rashid Mushkani Shravan Nayak Hugo Berard Allison Cohen Shin Koseki Hadrien Bertrand 54 2 0 27 Feb 2025
Shh, don't say that! Domain Certification in LLMs Cornelius Emde Alasdair Paren Preetham Arvind Maxime Kayser Tom Rainforth Thomas Lukasiewicz Bernard Ghanem Philip H. S. Torr Adel Bibi 45 1 0 26 Feb 2025
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework Kaishuai Xu Tiezheng YU Wenjun Hou Yi Cheng Liangyou Li Xin Jiang Lifeng Shang Q. Liu Wenjie Li ELM 66 0 0 26 Feb 2025
Self-Memory Alignment: Mitigating Factual Hallucinations with Generalized Improvement Siyuan Zhang Y. Zhang Yinpeng Dong Hang Su HILM KELM 93 0 0 26 Feb 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 74 0 0 26 Feb 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 82 1 0 26 Feb 2025
ANPMI: Assessing the True Comprehension Capabilities of LLMs for Multiple Choice Questions Gyeongje Cho Yeonkyoung So Jaejin Lee ELM 59 0 0 26 Feb 2025
M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance Qingpei Guo Kaiyou Song Zipeng Feng Ziping Ma Qinglong Zhang ... Yunxiao Sun Tai-WeiChang Jingdong Chen Ming Yang Jun Zhou MLLM VLM 77 3 0 26 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 72 0 0 25 Feb 2025
Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems Matthew Barker Andrew Bell Evan Thomas James Carr Thomas Andrews Umang Bhatt 80 1 0 25 Feb 2025
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference Xiangyu Zhao Shengyuan Ding Zicheng Zhang Haian Huang Maosong Cao ... Wenhai Wang Guangtao Zhai Haodong Duan Hua Yang Kai Chen 97 6 0 25 Feb 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms Yashan Wang Shangda Wu Jianhuai Hu Xingjian Du Yueqi Peng Yongxin Huang Shuai Fan Xiaobing Li Feng Yu Maosong Sun 99 2 0 25 Feb 2025
A General Framework to Enhance Fine-tuning-based LLM Unlearning J. Ren Zhenwei Dai X. Tang Hui Liu Jingying Zeng ... R. Goutam Suhang Wang Yue Xing Qi He Hui Liu MU 107 1 0 25 Feb 2025
Post-edits Are Preferences Too Nathaniel Berger Stefan Riezler M. Exel Matthias Huck 32 0 0 24 Feb 2025
RLTHF: Targeted Human Feedback for LLM Alignment Yifei Xu Tusher Chakraborty Emre Kıcıman Bibek Aryal Eduardo Rodrigues ... Rafael Padilha Leonardo Nunes Shobana Balakrishnan Songwu Lu Ranveer Chandra 98 1 0 24 Feb 2025
Unveiling Reasoning Thresholds in Language Models: Scaling, Fine-Tuning, and Interpretability through Attention Maps Yen-Che Hsiao Abhishek Dutta LRM ReLM ELM 54 0 0 24 Feb 2025
Spontaneous Giving and Calculated Greed in Language Models Yuxuan Li Hirokazu Shirado ReLM LRM AI4CE 38 0 0 24 Feb 2025
Is Free Self-Alignment Possible? Dyah Adila Changho Shin Yijing Zhang Frederic Sala MoMe 105 2 0 24 Feb 2025
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint Qianli Ma Dongrui Liu Qian Chen Linfeng Zhang Jing Shao MoMe 61 0 0 24 Feb 2025
NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions Weizhe Yuan Jane Dwivedi-Yu Song Jiang Karthik Padthe Yang Li ... Ilia Kulikov Kyunghyun Cho Yuandong Tian Jason Weston Xian Li ReLM LRM 43 10 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 52 2 0 24 Feb 2025
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent Yuheng Zhang Dian Yu Tao Ge Linfeng Song Zhichen Zeng Haitao Mi Nan Jiang Dong Yu 56 1 0 24 Feb 2025
Sequence-level Large Language Model Training with Contrastive Preference Optimization Zhili Feng Dhananjay Ram Cole Hawkins Aditya Rawal Jinman Zhao Sheng Zha 57 0 0 23 Feb 2025
DISC: Dynamic Decomposition Improves LLM Inference Scaling Jonathan Light Wei Cheng Wu Yue Masafumi Oyamada Mengdi Wang Santiago Paternain Haifeng Chen ReLM LRM 56 1 0 23 Feb 2025
Fine-Tuning Qwen 2.5 3B for Realistic Movie Dialogue Generation Kartik Gupta VGen 41 0 0 22 Feb 2025
A generative approach to LLM harmfulness detection with special red flag tokens Sophie Xhonneux David Dobre Mehrnaz Mohfakhami Leo Schwinn Gauthier Gidel 45 1 0 22 Feb 2025
IPO: Your Language Model is Secretly a Preference Classifier Shivank Garg Ayush Singh Shweta Singh Paras Chopra 54 1 0 22 Feb 2025
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation Shuo Tang Xianghe Pang Zexi Liu Bohan Tang Rui Ye Xiaowen Dong Y. Wang Yanfeng Wang S. Chen SyDa LLMAG 124 3 0 21 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 93 5 0 21 Feb 2025
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment Sizhe Wang Yongqi Tong Hengyuan Zhang Dawei Li Xin Zhang Tianlong Chen 85 5 0 21 Feb 2025
A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics Ting-Ruen Wei Haowei Liu Xuyang Wu Yi Fang LRM AI4CE ReLM KELM 113 1 0 21 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 60 23 0 20 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Z. Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 35 1 0 20 Feb 2025
UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning Vaidehi Patil Elias Stengel-Eskin Mohit Bansal MU CLL 73 2 0 20 Feb 2025
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment Zhili Liu Yunhao Gou Kai Chen Lanqing Hong Jiahui Gao ... Yu Zhang Zhenguo Li Xin Jiang Q. Liu James T. Kwok MoE 96 9 0 20 Feb 2025
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images Shengguang Wu Fan-Yun Sun Kaiyue Wen Nick Haber VLM 72 0 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 36 4 0 20 Feb 2025
Diffusion Models in Recommendation Systems: A Survey Ting-Ruen Wei Yi Fang 77 2 0 20 Feb 2025
Drift: Decoding-time Personalized Alignments with Implicit User Preferences Minbeom Kim Kang-il Lee Seongho Joo Hwaran Lee Thibaut Thonet Kyomin Jung AI4TS 107 1 0 20 Feb 2025