Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

5 June 2024

Chelsea Finn

Papers citing "Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms"

42 / 42 papers shown

Title
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang James Thorne 29 0 0 12 May 2025
Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations Ran Tian Kratarth Goel 46 0 0 25 Mar 2025
Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators Seungone Kim Ian Wu Jinu Lee Xiang Yue Seongyun Lee ... Kiril Gashteovski Carolin (Haas) Lawrence J. Hockenmaier Graham Neubig Sean Welleck LRM 50 2 0 25 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuyao Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang Tieniu Tan 164 2 0 18 Mar 2025
Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation Bowen Baker Joost Huizinga Leo Gao Zehao Dou M. Guan Aleksander Mądry Wojciech Zaremba J. Pachocki David Farhi LRM 74 11 0 14 Mar 2025
RePO: ReLU-based Preference Optimization Junkang Wu Kexin Huang Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xuben Wang 76 0 0 10 Mar 2025
Robust Multi-Objective Preference Alignment with Online DPO Raghav Gupta Ryan Sullivan Yunxuan Li Samrat Phatale Abhinav Rastogi 37 0 0 01 Mar 2025
Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time Jiazheng Li Yuxiang Zhou Junru Lu Gladys Tyen Lin Gui Cesare Aloisi Yulan He LRM 39 2 0 26 Feb 2025
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 85 0 0 14 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 56 0 0 08 Feb 2025
The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking Yuchun Miao Sen Zhang Liang Ding Yuqi Zhang L. Zhang Dacheng Tao 81 3 0 31 Jan 2025
Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Zhenyu Hou Xin Lv Rui Lu J. Zhang Yongqian Li Zijun Yao Juanzi Li J. Tang Yuxiao Dong OffRL LRM ReLM 61 20 0 20 Jan 2025
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization Chia-Yu Hung Navonil Majumder Zhifeng Kong Ambuj Mehrish Rafael Valle Bryan Catanzaro Soujanya Poria Bryan Catanzaro Soujanya Poria 52 5 0 30 Dec 2024
When Can Proxies Improve the Sample Complexity of Preference Learning? Yuchen Zhu Daniel Augusto de Souza Zhengyan Shi Mengyue Yang Pasquale Minervini Alexander DÁmour Matt J. Kusner 80 0 0 21 Dec 2024
Drowning in Documents: Consequences of Scaling Reranker Inference Mathew Jacob Erik Lindgren Matei A. Zaharia Michael Carbin Omar Khattab Andrew Drozdov OffRL 74 4 0 18 Nov 2024
Efficient Alignment of Large Language Models via Data Sampling Amrit Khera Rajat Ghosh Debojyoti Dutta 36 1 0 15 Nov 2024
Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval Sheryl Hsu Omar Khattab Chelsea Finn Archit Sharma KELM RALM 41 6 0 30 Oct 2024
Process Supervision-Guided Policy Optimization for Code Generation Ning Dai Zheng Wu Renjie Zheng Ziyun Wei Wenlei Shi Xing Jin Guanlin Liu Chen Dun Liang Huang Lin Yan 54 8 0 23 Oct 2024
Understanding Likelihood Over-optimisation in Direct Alignment Algorithms Zhengyan Shi Sander Land Acyr F. Locatelli Matthieu Geist Max Bartolo 46 4 0 15 Oct 2024
How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective Teng Xiao Mingxiao Li Yige Yuan Huaisheng Zhu Chao Cui V. Honavar ALM 28 7 0 14 Oct 2024
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 91 14 0 11 Oct 2024
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 47 8 0 11 Oct 2024
Generative Reward Models Dakota Mahan Duy Phung Rafael Rafailov Chase Blagden Nathan Lile Louis Castricato Jan-Philipp Fränken Chelsea Finn Alon Albalak VLM SyDa OffRL 27 26 0 02 Oct 2024
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits Duy Nguyen Archiki Prasad Elias Stengel-Eskin Joey Tianyi Zhou 23 3 0 02 Oct 2024
Evaluating Robustness of Reward Models for Mathematical Reasoning Sunghwan Kim Dongjin Kang Taeyoon Kwon Hyungjoo Chae Jungsoo Won Dongha Lee Jinyoung Yeo 30 4 0 02 Oct 2024
The Perfect Blend: Redefining RLHF with Mixture of Judges Tengyu Xu Eryk Helenowski Karthik Abinav Sankararaman Di Jin Kaiyan Peng ... Gabriel Cohen Yuandong Tian Hao Ma Sinong Wang Han Fang 38 9 0 30 Sep 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 37 2 0 25 Sep 2024
Selective Preference Optimization via Token-Level Reward Function Estimation Kailai Yang Zhiwei Liu Qianqian Xie Jimin Huang Erxue Min Sophia Ananiadou 33 10 0 24 Aug 2024
Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment Karel DÓosterlinck Winnie Xu Chris Develder Thomas Demeester A. Singh Christopher Potts Douwe Kiela Shikib Mehri 33 10 0 12 Aug 2024
LIONs: An Empirically Optimized Approach to Align Language Models Xiao Yu Qingyang Wu Yu Li Zhou Yu ALM 32 3 0 09 Jul 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 40 8 0 25 Jun 2024
Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding Kaiyan Zhang Jianyu Wang Ning Ding Biqing Qi Ermo Hua Xingtai Lv Bowen Zhou 43 8 0 18 Jun 2024
WPO: Enhancing RLHF with Weighted Preference Optimization Wenxuan Zhou Ravi Agrawal Shujian Zhang Sathish Indurthi Sanqiang Zhao Kaiqiang Song Silei Xu Chenguang Zhu 35 18 0 17 Jun 2024
Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence Junru Lu Jiazheng Li Siyu An Meng Zhao Yulan He Di Yin Xing Sun 39 14 0 16 Jun 2024
Preference Alignment with Flow Matching Minu Kim Yongsik Lee Sehyeok Kang Jihwan Oh Song Chong Seyoung Yun 32 1 0 30 May 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 81 21 0 29 May 2024
Learn Your Reference Model for Real Good Alignment Alexey Gorbatovski Boris Shaposhnikov Alexey Malakhov Nikita Surnachev Yaroslav Aksenov Ian Maksimov Nikita Balagansky Daniil Gavrilov OffRL 54 26 0 15 Apr 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 173 449 0 02 Feb 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 72 32 0 30 Dec 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 243 4,469 0 23 Jan 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019