Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

29 January 2024

Papers citing "Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF"

29 / 29 papers shown

Title
In-context Ranking Preference Optimization Junda Wu Rohan Surana Zhouhang Xie Yiran Shen Yu Xia Tong Yu Ryan Rossi Prithviraj Ammanabrolu Julian McAuley 38 0 0 21 Apr 2025
Energy-Based Reward Models for Robust Language Model Alignment Anamika Lochab Ruqi Zhang 36 0 0 17 Apr 2025
Collab: Controlled Decoding using Mixture of Agents for LLM Alignment Souradip Chakraborty Sujay Bhatt Udari Madhushani Sehwag Soumya Suvra Ghosal Jiahao Qiu Mengdi Wang Dinesh Manocha Furong Huang Alec Koppel Sumitra Ganesh 39 2 0 27 Mar 2025
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 74 0 0 14 Feb 2025
The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking Yuchun Miao Sen Zhang Liang Ding Yuqi Zhang L. Zhang Dacheng Tao 81 3 0 31 Jan 2025
Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques Natalia Zhang X. Wang Qiwen Cui Runlong Zhou Sham Kakade Simon S. Du OffRL 35 0 0 10 Jan 2025
When Can Proxies Improve the Sample Complexity of Preference Learning? Yuchen Zhu Daniel Augusto de Souza Zhengyan Shi Mengyue Yang Pasquale Minervini Alexander DÁmour Matt J. Kusner 64 0 0 21 Dec 2024
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Avinandan Bose Zhihan Xiong Aadirupa Saha S. Du Maryam Fazel 64 0 0 13 Dec 2024
Self-Generated Critiques Boost Reward Modeling for Language Models Yue Yu Zhengxing Chen Aston Zhang L Tan Chenguang Zhu ... Suchin Gururangan Chao-Yue Zhang Melanie Kambadur Dhruv Mahajan Rui Hou LRM ALM 78 14 0 25 Nov 2024
Optimal Design for Reward Modeling in RLHF Antoine Scheid Etienne Boursier Alain Durmus Michael I. Jordan Pierre Ménard Eric Moulines Michal Valko OffRL 24 5 0 22 Oct 2024
Understanding Likelihood Over-optimisation in Direct Alignment Algorithms Zhengyan Shi Sander Land Acyr F. Locatelli Matthieu Geist Max Bartolo 43 3 0 15 Oct 2024
Reward Learning From Preference With Ties Jinsong Liu Dongdong Ge Ruihao Zhu 19 0 0 05 Oct 2024
Post-hoc Reward Calibration: A Case Study on Length Bias Zeyu Huang Zihan Qiu Zili Wang Edoardo M. Ponti Ivan Titov 34 5 0 25 Sep 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 28 1 0 25 Sep 2024
Understanding the Gains from Repeated Self-Distillation Divyansh Pareek Simon S. Du Sewoong Oh 27 3 0 05 Jul 2024
Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs Rui Yang Ruomeng Ding Yong Lin Huan Zhang Tong Zhang 19 42 0 14 Jun 2024
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms Rafael Rafailov Yaswanth Chittepu Ryan Park Harshit S. Sikchi Joey Hejna Bradley Knox Chelsea Finn S. Niekum 39 13 0 05 Jun 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 68 20 0 29 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 25 41 0 26 May 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 20 54 0 01 Mar 2024
InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling Yuchun Miao Sen Zhang Liang Ding Rong Bao Lefei Zhang Dacheng Tao 14 12 0 14 Feb 2024
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 197 2,953 0 22 Mar 2023
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 220 495 0 28 Sep 2022
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 213 327 0 23 Aug 2022
Offline RL for Natural Language Generation with Implicit Language Q Learning Charles Burton Snell Ilya Kostrikov Yi Su Mengjiao Yang Sergey Levine OffRL 119 101 0 05 Jun 2022
Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation Xiaoyu Chen Han Zhong Zhuoran Yang Zhaoran Wang Liwei Wang 116 59 0 23 May 2022
Teaching language models to support answers with verified quotes Jacob Menick Maja Trebacz Vladimir Mikulik John Aslanides Francis Song ... Mia Glaese Susannah Young Lucy Campbell-Gillingham G. Irving Nat McAleese ELM RALM 213 204 0 21 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 273 1,561 0 18 Sep 2019