Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

41 / 641 papers shown

Title
Institutional Platform for Secure Self-Service Large Language Model Exploration V. Bumgardner Mitchell A. Klusty W. V. Logan Samuel E. Armstrong Caylin D. Hickey Jeff Talbert Caylin Hickey Jeff Talbert 44 1 0 01 Feb 2024
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF Banghua Zhu Michael I. Jordan Jiantao Jiao 21 22 0 29 Jan 2024
Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation Yuxin Liang Zhuoyang Song Hao Wang Jiaxing Zhang HILM 31 28 0 27 Jan 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 230 294 0 18 Jan 2024
Aligning Large Language Models with Counterfactual DPO Bradley Butcher ALM 21 1 0 17 Jan 2024
Canvil: Designerly Adaptation for LLM-Powered User Experiences K. J. Kevin Feng Q. V. Liao Ziang Xiao Jennifer Wortman Vaughan Amy X. Zhang David W. McDonald 26 16 0 17 Jan 2024
Crowd-PrefRL: Preference-Based Reward Learning from Crowds David Chhan Ellen R. Novoseller Vernon J. Lawhern 27 5 0 17 Jan 2024
TOFU: A Task of Fictitious Unlearning for LLMs Pratyush Maini Zhili Feng Avi Schwarzschild Zachary Chase Lipton J. Zico Kolter MU CLL 38 141 0 11 Jan 2024
A Minimaximalist Approach to Reinforcement Learning from Human Feedback Gokul Swamy Christoph Dann Rahul Kidambi Zhiwei Steven Wu Alekh Agarwal OffRL 28 94 0 08 Jan 2024
Theoretical guarantees on the best-of-n alignment policy Ahmad Beirami Alekh Agarwal Jonathan Berant Alex DÁmour Jacob Eisenstein Chirag Nagpal A. Suresh 42 42 0 03 Jan 2024
An Invitation to Deep Reinforcement Learning Bernhard Jaeger Andreas Geiger OffRL OOD 62 5 0 13 Dec 2023
LLM360: Towards Fully Transparent Open-Source LLMs Zhengzhong Liu Aurick Qiao W. Neiswanger Hongyi Wang Bowen Tan ... Zhiting Hu Mark Schulze Preslav Nakov Timothy Baldwin Eric P. Xing 38 68 0 11 Dec 2023
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization Zhiyuan Zhao Bin Wang Linke Ouyang Xiao-wen Dong Jiaqi Wang Conghui He MLLM VLM 29 105 0 28 Nov 2023
A density estimation perspective on learning from pairwise human preferences Vincent Dumoulin Daniel D. Johnson Pablo Samuel Castro Hugo Larochelle Yann Dauphin 23 12 0 23 Nov 2023
Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandit Tian Huang Ke Li Ke Li 18 1 0 23 Nov 2023
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model Kai Yang Jian Tao Jiafei Lyu Chunjiang Ge Jiaxin Chen Qimai Li Weihan Shen Xiaolong Zhu Xiu Li EGVM 16 87 0 22 Nov 2023
Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectification Haoqiang Kang Juntong Ni Huaxiu Yao HILM LRM 14 33 0 15 Nov 2023
Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization Zhexin Zhang Junxiao Yang Pei Ke Fei Mi Hongning Wang Minlie Huang AAML 16 113 0 15 Nov 2023
Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models Keming Lu Hongyi Yuan Runji Lin Junyang Lin Zheng Yuan Chang Zhou Jingren Zhou MoE LRM 32 52 0 15 Nov 2023
An Emulator for Fine-Tuning Large Language Models using Small Language Models Eric Mitchell Rafael Rafailov Archit Sharma Chelsea Finn Christopher D. Manning ALM 27 51 0 19 Oct 2023
Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model Qichen Ye Junling Liu Dading Chong Peilin Zhou Yining Hua ... Meng Cao Ziming Wang Xuxin Cheng Andrew Liu Zhenhua Guo AI4MH LM&MA ELM 25 19 0 13 Oct 2023
Constructive Large Language Models Alignment with Diverse Feedback Tianshu Yu Ting-En Lin Yuchuan Wu Min Yang Fei Huang Yongbin Li ALM 30 8 0 10 Oct 2023
Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond Hao Sun OffRL 18 20 0 09 Oct 2023
Improving Summarization with Human Edits Zonghai Yao Benjamin J Schloss Sai P. Selvaraj 24 2 0 09 Oct 2023
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints Chaoqi Wang Yibo Jiang Yuguang Yang Han Liu Yuxin Chen 19 81 0 28 Sep 2023
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model Avamarie Brueggeman Andrea Madotto Zhaojiang Lin Tushar Nagarajan Matt Smith ... Peyman Heidari Yue Liu Kavya Srinet Babak Damavandi Anuj Kumar MLLM 24 92 0 27 Sep 2023
Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding Jiacheng Liu Andrew Cohen Ramakanth Pasunuru Yejin Choi Hannaneh Hajishirzi Asli Celikyilmaz 6 22 0 26 Sep 2023
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data Guan-Bo Wang Sijie Cheng Xianyuan Zhan Xiangang Li Sen Song Yang Liu ALM 8 227 0 20 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 49 699 0 19 Sep 2023
OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs Patrick Haller Ansar Aynetdinov A. Akbik 20 24 0 07 Sep 2023
FLM-101B: An Open LLM and How to Train It with $100K Budget$ Xiang Li Yiqun Yao Xin Jiang Xuezhi Fang Xuying Meng ... LI DU Bowen Qin Zheng-Wei Zhang Aixin Sun Yequan Wang 52 21 0 07 Sep 2023
How to Protect Copyright Data in Optimization of Large Language Models? T. Chu Zhao-quan Song Chiwun Yang 28 29 0 23 Aug 2023
Instruction Position Matters in Sequence Generation with Large Language Models Yanjun Liu Xianfeng Zeng Fandong Meng Jie Zhou LRM 35 8 0 23 Aug 2023
Scaling Sentence Embeddings with Large Language Models Ting Jiang Shaohan Huang Zhongzhi Luan Deqing Wang Fuzhen Zhuang LRM 34 40 0 31 Jul 2023
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment Kevin Kaichuang Yang Dan Klein Asli Celikyilmaz Nanyun Peng Yuandong Tian ALM 25 31 0 24 Jul 2023
Let Me Teach You: Pedagogical Foundations of Feedback for Language Models Beatriz Borges Niket Tandon Tanja Kaser Antoine Bosselut 17 3 0 01 Jul 2023
On the Creativity of Large Language Models Giorgio Franceschelli Mirco Musolesi 61 49 0 27 Mar 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 212 2,232 0 22 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 203 1,651 0 15 Oct 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019