Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

18 December 2023

Wei Xiong

Tong Zhang

Papers citing "Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint"

50 / 125 papers shown

Title
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment Xueyao Zhang Y. Wang Chaoren Wang Z. Li Zhuo Chen Zhizheng Wu 46 0 0 07 May 2025
Policy-labeled Preference Learning: Is Preference Enough for RLHF? Taehyun Cho Seokhun Ju Seungyub Han Dohyeong Kim Kyungjae Lee Jungwoo Lee OffRL 24 0 0 06 May 2025
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL Jiarui Yao Yifan Hao Hanning Zhang Hanze Dong Wei Xiong Nan Jiang Tong Zhang LRM 47 0 0 05 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 53 0 0 05 May 2025
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback Nan Lu Ethan X. Fang Junwei Lu 60 0 0 27 Apr 2025
Anyprefer: An Agentic Framework for Preference Data Synthesis Yiyang Zhou Z. Wang Tianle Wang Shangyu Xing Peng Xia ... Chetan Bansal Weitong Zhang Ying Wei Mohit Bansal Huaxiu Yao 54 0 0 27 Apr 2025
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models J. Liu Hangyu Guo Ranjie Duan Xingyuan Bu Yancheng He ... Yingshui Tan Yanan Wu Jihao Gu Y. Li J. Zhu MLLM 70 0 0 25 Apr 2025
Compass-V2 Technical Report Sophia Maria MoE LRM 29 0 0 22 Apr 2025
Direct Advantage Regression: Aligning LLMs with Online AI Reward Li He He Zhao Stephen Wan Dadong Wang Lina Yao Tongliang Liu 27 0 0 19 Apr 2025
Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo João Loula Benjamin LeBrun Li Du Ben Lipkin Clemente Pasti ... Ryan Cotterel Vikash K. Mansinghka Alexander K. Lew Tim Vieira Timothy J. O'Donnell 32 1 0 17 Apr 2025
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce Wei Xiong Jiarui Yao Yuhui Xu Bo Pang Lei Wang ... Junnan Li Nan Jiang Tong Zhang Caiming Xiong Hanze Dong OffRL LRM 33 2 0 15 Apr 2025
CHARM: Calibrating Reward Models With Chatbot Arena Scores Xiao Zhu Chenmien Tan Pinzhen Chen Rico Sennrich Yanlin Zhang Hanxu Hu ALM 24 0 0 14 Apr 2025
How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients Ming Li Y. Li Ziyue Li Tianyi Zhou LRM 19 1 0 14 Apr 2025
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Kai Ye Hongyi Zhou Jin Zhu Francesco Quinzan C. Shi 20 0 0 03 Apr 2025
Entropy-Based Adaptive Weighting for Self-Training Xiaoxuan Wang Yihe Deng Mingyu Derek Ma Wei Wang LRM 45 0 0 31 Mar 2025
Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF Syrine Belakaria Joshua Kazdan Charles Marx Chris Cundy W. Neiswanger Sanmi Koyejo Barbara Engelhardt Stefano Ermon 29 0 0 28 Mar 2025
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization Yunan Wang Jijie Li Bo Zhang Liangdong Wang Guang Liu 58 0 0 20 Mar 2025
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation Songjun Tu Jiahao Lin Xiangyu Tian Qichao Zhang Linjing Li ... Nan Xu Wei He Xiangyuan Lan D. Jiang Dongbin Zhao LRM 44 2 0 17 Mar 2025
Representation-based Reward Modeling for Efficient Safety Alignment of Large Language Model Qiyuan Deng X. Bai Kehai Chen Yaowei Wang Liqiang Nie Min Zhang OffRL 57 0 0 13 Mar 2025
RePO: ReLU-based Preference Optimization Junkang Wu Kexin Huang Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He X. Wang 71 0 0 10 Mar 2025
UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality Zelei Cheng Xin-Qiang Cai Yuting Tang Pushi Zhang Boming Yang Xinyu Xing 39 0 0 10 Mar 2025
Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models Kefan Song Jin Yao Runnan Jiang Rohan Chandra Shangtong Zhang ALM 46 0 0 10 Mar 2025
SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner Kejia Chen Jiawen Zhang Jiacong Hu Jiazhen Yang Jian Lou Zunlei Feng Mingli Song 57 0 0 06 Mar 2025
Self-Memory Alignment: Mitigating Factual Hallucinations with Generalized Improvement Siyuan Zhang Y. Zhang Yinpeng Dong Hang Su HILM KELM 93 0 0 26 Feb 2025
Self-rewarding correction for mathematical reasoning Wei Xiong Hanning Zhang Chenlu Ye Lichang Chen Nan Jiang Tong Zhang ReLM KELM LRM 64 9 0 26 Feb 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 74 0 0 26 Feb 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 82 1 0 26 Feb 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He X. Wang 42 0 0 25 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 72 0 0 25 Feb 2025
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent Yuheng Zhang Dian Yu Tao Ge Linfeng Song Zhichen Zeng Haitao Mi Nan Jiang Dong Yu 56 1 0 24 Feb 2025
Preference Optimization for Reasoning with Pseudo Feedback Fangkai Jiao Geyang Guo Xingxing Zhang Nancy F. Chen Shafiq R. Joty Furu Wei LRM 99 8 0 17 Feb 2025
PIPA: Preference Alignment as Prior-Informed Statistical Estimation Junbo Li Zhangyang Wang Qiang Liu OffRL 97 0 0 09 Feb 2025
Nearly Optimal Sample Complexity of Offline KL-Regularized Contextual Bandits under Single-Policy Concentrability Qingyue Zhao Kaixuan Ji Heyang Zhao Tong Zhang Q. Gu OffRL 35 0 0 09 Feb 2025
ARIES: Stimulating Self-Refinement of Large Language Models by Iterative Preference Optimization Yongcheng Zeng Xinyu Cui Xuanfa Jin Guoqing Liu Zexu Sun ... Dong Li Ning Yang Jianye Hao H. Zhang J. Wang LRM LLMAG 76 1 0 08 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 113 3 0 06 Feb 2025
Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques Natalia Zhang X. Wang Qiwen Cui Runlong Zhou Sham Kakade Simon S. Du OffRL 43 1 0 10 Jan 2025
Online Learning from Strategic Human Feedback in LLM Fine-Tuning Shugang Hao Lingjie Duan 79 3 0 22 Dec 2024
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Avinandan Bose Zhihan Xiong Aadirupa Saha S. Du Maryam Fazel 66 1 0 13 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang J. Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 121 6 0 05 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 97 3 0 02 Dec 2024
Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization Zhuotong Chen Fang Liu Jennifer Zhu Wanyu Du Yanjun Qi 33 0 0 07 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 54 3 0 07 Nov 2024
SEE-DPO: Self Entropy Enhanced Direct Preference Optimization Shivanshu Shekhar Shreyas Singh Tong Zhang 30 4 0 06 Nov 2024
Sample-Efficient Alignment for LLMs Zichen Liu Changyu Chen Chao Du Wee Sun Lee Min-Bin Lin 31 3 0 03 Nov 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 58 3 0 30 Oct 2024
Uncertainty-Penalized Direct Preference Optimization Sam Houliston Alizée Pace Alexander Immer Gunnar Rätsch 20 0 0 26 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 50 2 0 20 Oct 2024
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling Jiahao Qiu Yifu Lu Yifan Zeng Jiacheng Guo Jiayi Geng Huazheng Wang Kaixuan Huang Yue Wu Mengdi Wang 34 22 0 18 Oct 2024
A Theoretical Survey on Foundation Models Shi Fu Yuzhu Chen Yingjie Wang Dacheng Tao 21 0 0 15 Oct 2024
Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization Guanlin Liu Kaixuan Ji Ning Dai Zheng Wu Chen Dun Q. Gu Lin Yan Quanquan Gu Lin Yan OffRL LRM 46 8 0 11 Oct 2024