Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

18 December 2023

Wei Xiong

Tong Zhang

Papers citing "Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint"

50 / 125 papers shown

Title
Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization Guanlin Liu Kaixuan Ji Ning Dai Zheng Wu Chen Dun Q. Gu Lin Yan Quanquan Gu Lin Yan OffRL LRM 48 8 0 11 Oct 2024
MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization Yougang Lyu Lingyong Yan Zihan Wang Dawei Yin Pengjie Ren Maarten de Rijke Z. Z. Ren 55 6 0 10 Oct 2024
Automatic Curriculum Expert Iteration for Reliable LLM Reasoning Zirui Zhao Hanze Dong Amrita Saha Caiming Xiong Doyen Sahoo LRM 27 3 0 10 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 87 2 0 10 Oct 2024
DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback Guojun Xiong Ujwal Dinesha Debajoy Mukherjee Jian Li Srinivas Shakkottai 28 2 0 07 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 56 3 0 06 Oct 2024
RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization Hanyang Zhao Genta Indra Winata Anirban Das Shi-Xiong Zhang D. Yao Wenpin Tang Sambit Sahu 54 4 0 05 Oct 2024
Learning Code Preference via Synthetic Evolution Jiawei Liu Thanh Nguyen Mingyue Shang Hantian Ding Xiaopeng Li Yu Yu Varun Kumar Zijian Wang SyDa ALM AAML 26 3 0 04 Oct 2024
Aligning LLMs with Individual Preferences via Interaction Shujin Wu May Fung Cheng Qian Jeonghwan Kim Dilek Z. Hakkani-Tür Heng Ji 26 9 0 04 Oct 2024
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits Duy Nguyen Archiki Prasad Elias Stengel-Eskin Mohit Bansal 23 2 0 02 Oct 2024
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data Ziyi Ye Xiangsheng Li Qiuchi Li Qingyao Ai Yujia Zhou Wei Shen Dong Yan Yiqun Liu 48 10 0 01 Oct 2024
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown Xingzhou Lou Dong Yan Wei Shen Yuzi Yan Jian Xie Junge Zhang 45 21 0 01 Oct 2024
The Crucial Role of Samplers in Online Direct Preference Optimization Ruizhe Shi Runlong Zhou Simon S. Du 53 7 0 29 Sep 2024
Inference-Time Language Model Alignment via Integrated Value Guidance Zhixuan Liu Zhanhui Zhou Yuanfu Wang Chao Yang Yu Qiao 24 7 0 26 Sep 2024
Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization Ruijie Xu Zhihan Liu Yongfei Liu Shipeng Yan Zhaoran Wang Zhi-Li Zhang Xuming He ALM 28 1 0 26 Sep 2024
Training Language Models to Win Debates with Self-Play Improves Judge Accuracy Samuel Arnesen David Rein Julian Michael ELM 28 3 0 25 Sep 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 35 1 0 25 Sep 2024
Orthogonal Finetuning for Direct Preference Optimization Chenxu Yang Ruipeng Jia Naibin Gu Zheng-Shen Lin Siyuan Chen Chao Pang Weichong Yin Yu Sun Hua-Hong Wu Weiping Wang 27 0 0 23 Sep 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 10 0 18 Sep 2024
Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement Simon Yu Liangyu Chen Sara Ahmadian Marzieh Fadaee 29 6 0 17 Sep 2024
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do Guijin Son Hyunwoo Ko Hoyoung Lee Yewon Kim Seunghyeok Hong ALM ELM 46 5 0 17 Sep 2024
Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison Judy Hanwen Shen Archit Sharma Jun Qin 37 4 0 15 Sep 2024
Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences Z. Liu Junjie Xu Xingjiao Wu J. Yang Liang He 23 0 0 11 Sep 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 37 11 0 11 Sep 2024
On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization Yong Lin Skyler Seto Maartje ter Hoeve Katherine Metcalf B. Theobald Xuan Wang Yizhe Zhang Chen Huang Tong Zhang 29 12 0 05 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 51 0 0 28 Aug 2024
Selective Preference Optimization via Token-Level Reward Function Estimation Kailai Yang Zhiwei Liu Qianqian Xie Jimin Huang Erxue Min Sophia Ananiadou 28 10 0 24 Aug 2024
ProFuser: Progressive Fusion of Large Language Models Tianyuan Shi Fanqi Wan Canbin Huang Xiaojun Quan Chenliang Li Ming Yan Ji Zhang MoMe 21 2 0 09 Aug 2024
On the Generalization of Preference Learning with DPO Shawn Im Yixuan Li 33 1 0 06 Aug 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 27 4 0 26 Jul 2024
A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More Zhichao Wang Bin Bi Shiva K. Pentyala Kiran Ramnath Sougata Chaudhuri ... Z. Zhu Xiang-Bo Mao S. Asur Na Na Cheng OffRL 34 38 0 23 Jul 2024
Improving Context-Aware Preference Modeling for Language Models Silviu Pitis Ziang Xiao Nicolas Le Roux Alessandro Sordoni 24 8 0 20 Jul 2024
PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods Yiying Wang Xiaojing Li Binzhu Wang Yueyang Zhou Yingru Lin ... Fei Yu Zewei Zhao Song Jin Renji Gong Wanqing Xu 19 1 0 09 Jul 2024
OffsetBias: Leveraging Debiased Data for Tuning Evaluators Junsoo Park Seungyeon Jwa Meiying Ren Daeyoung Kim Sanghyuk Choi ALM 29 29 0 09 Jul 2024
Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning Yifang Chen Shuohang Wang Ziyi Yang Hiteshi Sharma Nikos Karampatziakis Donghan Yu Kevin G. Jamieson Simon Shaolei Du Yelong Shen OffRL 41 4 0 02 Jul 2024
Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning Yuheng Zhang Dian Yu Baolin Peng Linfeng Song Ye Tian Mingyue Huo Nan Jiang Haitao Mi Dong Yu 30 14 0 30 Jun 2024
Decoding-Time Language Model Alignment with Multiple Objectives Ruizhe Shi Yifang Chen Yushi Hu Alisa Liu Hannaneh Hajishirzi Noah A. Smith Simon Du 44 30 0 27 Jun 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 38 8 0 25 Jun 2024
Cascade Reward Sampling for Efficient Decoding-Time Alignment Bolian Li Yifan Wang A. Grama Ruqi Zhang Ruqi Zhang AI4TS 47 9 0 24 Jun 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 44 132 0 18 Jun 2024
mDPO: Conditional Preference Optimization for Multimodal Large Language Models Fei Wang Wenxuan Zhou James Y. Huang Nan Xu Sheng Zhang Hoifung Poon Muhao Chen 62 15 0 17 Jun 2024
Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level Jie Liu Zhanhui Zhou Jiaheng Liu Xingyuan Bu Chao Yang Han-Sen Zhong Wanli Ouyang 23 16 0 17 Jun 2024
Style Transfer with Multi-iteration Preference Optimization Shuai Liu Jonathan May 32 3 0 17 Jun 2024
Bootstrapping Language Models with DPO Implicit Rewards Changyu Chen Zichen Liu Chao Du Tianyu Pang Qian Liu Arunesh Sinha Pradeep Varakantham Min-Bin Lin SyDa ALM 62 23 0 14 Jun 2024
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing Zhangchen Xu Fengqing Jiang Luyao Niu Yuntian Deng Radha Poovendran Yejin Choi Bill Yuchen Lin SyDa 27 110 0 12 Jun 2024
OPTune: Efficient Online Preference Tuning Lichang Chen Jiuhai Chen Chenxi Liu John Kirchenbauer Davit Soselia Chen Zhu Tom Goldstein Tianyi Zhou Heng Huang 34 4 0 11 Jun 2024
3D-Properties: Identifying Challenges in DPO and Charting a Path Forward Yuzi Yan Yibo Miao J. Li Yipin Zhang Jian Xie Zhijie Deng Dong Yan 39 11 0 11 Jun 2024
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling Lin Gui Cristina Garbacea Victor Veitch BDL LM&MA 36 35 0 02 Jun 2024
Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment Yueqin Yin Zhendong Wang Yujia Xie Weizhu Chen Mingyuan Zhou 33 4 0 31 May 2024