Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

18 December 2023

Wei Xiong

Tong Zhang

Papers citing "Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint"

25 / 125 papers shown

Title
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 19 21 0 30 May 2024
Preference Alignment with Flow Matching Minu Kim Yongsik Lee Sehyeok Kang Jihwan Oh Song Chong Seyoung Yun 19 1 0 30 May 2024
Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models Masatoshi Uehara Yulai Zhao Ehsan Hajiramezanali Gabriele Scalia Gökçen Eraslan Avantika Lal Sergey Levine Tommaso Biancalani 45 13 0 30 May 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 73 21 0 29 May 2024
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization Jiancong Xiao Ziniu Li Xingyu Xie E. Getzen Cong Fang Qi Long Weijie J. Su 41 10 0 26 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 57 335 0 23 May 2024
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process Ermo Hua Biqing Qi Kaiyan Zhang Yue Yu Ning Ding Xingtai Lv Kai Tian Bowen Zhou 32 3 0 20 May 2024
The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback Ruitao Chen Liwei Wang 64 1 0 18 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 21 92 0 13 May 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Guhao Feng Guhao Feng Li Zhao Di He Jiang Bian Liwei Wang Jiang Bian Liwei Wang 47 56 0 29 Apr 2024
REBEL: Reinforcement Learning via Regressing Relative Rewards Zhaolin Gao Jonathan D. Chang Wenhao Zhan Owen Oertell Gokul Swamy Kianté Brantley Thorsten Joachims J. Andrew Bagnell Jason D. Lee Wen Sun OffRL 30 31 0 25 Apr 2024
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data Fahim Tajwar Anika Singh Archit Sharma Rafael Rafailov Jeff Schneider Tengyang Xie Stefano Ermon Chelsea Finn Aviral Kumar 28 103 0 22 Apr 2024
Stepwise Alignment for Constrained Language Model Policy Optimization Akifumi Wachi Thien Q. Tran Rei Sato Takumi Tanabe Yohei Akimoto 34 5 0 17 Apr 2024
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study Shusheng Xu Wei Fu Jiaxuan Gao Wenjie Ye Weiling Liu Zhiyu Mei Guangju Wang Chao Yu Yi Wu 33 131 0 16 Apr 2024
ROPO: Robust Preference Optimization for Large Language Models Xize Liang Chao Chen Shuang Qiu Jie Wang Yue-bo Wu Zhihang Fu Zhihao Shi Feng Wu Jieping Ye 43 1 0 05 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 144 113 0 04 Apr 2024
Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation Xiaoying Zhang Jean-François Ton Wei Shen Hongning Wang Yang Liu 24 13 0 08 Mar 2024
Reward Generalization in RLHF: A Topological Perspective Tianyi Qiu Fanzhi Zeng Jiaming Ji Dong Yan Kaile Wang Jiayi Zhou Yang Han Josef Dai Xuehai Pan Yaodong Yang AI4CE 19 3 0 15 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 36 9 0 11 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 230 294 0 18 Jan 2024
Adversarial Model for Offline Reinforcement Learning M. Bhardwaj Tengyang Xie Byron Boots Nan Jiang Ching-An Cheng AAML OffRL 27 25 0 21 Feb 2023
A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Games Wei Xiong Han Zhong Chengshuai Shi Cong Shen Tong Zhang 53 19 0 04 Oct 2022
Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation Xiaoyu Chen Han Zhong Zhuoran Yang Zhaoran Wang Liwei Wang 118 59 0 23 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019