A General Theoretical Paradigm to Understand Learning from Human Preferences

18 October 2023

Bilal Piot

Daniele Calandriello

Papers citing "A General Theoretical Paradigm to Understand Learning from Human Preferences"

50 / 415 papers shown

Title
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 71 0 0 10 Mar 2025
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs Jongwoo Ko Tianyi Chen Sungnyun Kim Tianyu Ding Luming Liang Ilya Zharkov Se-Young Yun VLM 148 0 0 10 Mar 2025
ACAI for SBOs: AI Co-creation for Advertising and Inspiration for Small Business Owners Nimisha Karnatak Adrien Baranes Rob Marchant Triona Butler Kristen Olson 73 0 0 09 Mar 2025
Research on Superalignment Should Advance Now with Parallel Optimization of Competence and Conformity HyunJin Kim Xiaoyuan Yi Jing Yao Muhua Huang Jinyeong Bak James Evans Xing Xie 39 0 0 08 Mar 2025
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment Wen Yang Junhong Wu Chen Wang Chengqing Zong J. Zhang 73 1 0 06 Mar 2025
Adding Alignment Control to Language Models Wenhong Zhu Weinan Zhang Rui Wang 55 0 0 06 Mar 2025
Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models Alessio Galatolo Zhenbang Dai Katie Winkle Meriem Beloucif 53 0 0 05 Mar 2025
Preserving Cultural Identity with Context-Aware Translation Through Multi-Agent AI Systems Mahfuz Ahmed Anik Abdur Rahman Azmine Toushik Wasi Md Manjurul Ahsan 47 0 0 05 Mar 2025
FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4 Jiarui Yao Ruida Wang Tong Zhang LRM 60 0 0 05 Mar 2025
Adversarial Tokenization Renato Lui Geh Zilei Shao Guy Van den Broeck SILM AAML 87 0 0 04 Mar 2025
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation Songming Zhang Xue Zhang Tong Zhang Bojie Hu Yufeng Chen Jinan Xu 50 1 0 04 Mar 2025
DPR: Diffusion Preference-based Reward for Offline Reinforcement Learning Teng Pang Bingzheng Wang Guoqiang Wu Yilong Yin OffRL 68 0 0 03 Mar 2025
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning Gokul Swamy Sanjiban Choudhury Wen Sun Zhiwei Steven Wu J. Andrew Bagnell OffRL 42 7 0 03 Mar 2025
Robust Multi-Objective Preference Alignment with Online DPO Raghav Gupta Ryan Sullivan Yunxuan Li Samrat Phatale Abhinav Rastogi 37 0 0 01 Mar 2025
Self-rewarding correction for mathematical reasoning Wei Xiong Hanning Zhang Chenlu Ye Lichang Chen Nan Jiang Tong Zhang ReLM KELM LRM 67 9 0 26 Feb 2025
FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users Anikait Singh Sheryl Hsu Kyle Hsu E. Mitchell Stefano Ermon Tatsunori Hashimoto Archit Sharma Chelsea Finn SyDa OffRL 57 1 0 26 Feb 2025
OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment Jiaxin Deng Shiyao Wang Kuo Cai Lejian Ren Qigen Hu Weifeng Ding Qiang Luo Guorui Zhou 74 2 0 26 Feb 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 77 0 0 26 Feb 2025
CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization Shuming Shi Ruobing Zuo Gaolei He Jianlin Wang Chenyang Xu Zhengfeng Yang 60 0 0 25 Feb 2025
Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data Siqi Guo Ilgee Hong Vicente Balmaseda Changlong Yu Liang Qiu Xin Liu Haoming Jiang Tuo Zhao Tianbao Yang 43 0 0 25 Feb 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He X. Wang 48 0 0 25 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 72 0 0 25 Feb 2025
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent Yuheng Zhang Dian Yu Tao Ge Linfeng Song Zhichen Zeng Haitao Mi Nan Jiang Dong Yu 56 1 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 52 2 0 24 Feb 2025
Post-edits Are Preferences Too Nathaniel Berger Stefan Riezler M. Exel Matthias Huck 37 0 0 24 Feb 2025
Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance Chenghua Huang Lu Wang Fangkai Yang Pu Zhao Z. Li Qingwei Lin Dongmei Zhang Saravan Rajmohan Qi Zhang OffRL 52 1 0 24 Feb 2025
C-3DPO: Constrained Controlled Classification for Direct Preference Optimization Kavosh Asadi Julien Han Xingzi Xu Dominique Perrault-Joncas Shoham Sabach Karim Bouyarmane Mohammad Ghavamzadeh 34 0 0 22 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 93 5 0 21 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Z. Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 35 1 0 20 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 60 23 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 43 4 0 20 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 92 1 0 18 Feb 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Y. Wang Chengxuan Qian Huaxiu Yao Zhengzhong Tu 87 6 0 18 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu V. Cevher 49 0 0 18 Feb 2025
Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models Yingshui Tan Yilei Jiang Y. Li J. Liu Xingyuan Bu Wenbo Su Xiangyu Yue Xiaoyong Zhu Bo Zheng ALM 80 1 0 17 Feb 2025
Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training Fenghua Weng Jian Lou Jun Feng Minlie Huang Wenjie Wang AAML 64 1 0 17 Feb 2025
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 80 0 0 14 Feb 2025
PIPA: Preference Alignment as Prior-Informed Statistical Estimation Junbo Li Zhangyang Wang Qiang Liu OffRL 102 0 0 09 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 51 0 0 08 Feb 2025
ARIES: Stimulating Self-Refinement of Large Language Models by Iterative Preference Optimization Yongcheng Zeng Xinyu Cui Xuanfa Jin Guoqing Liu Zexu Sun ... Dong Li Ning Yang Jianye Hao H. Zhang J. Wang LRM LLMAG 84 1 0 08 Feb 2025
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance Shehzeen Samarah Hussain Paarth Neekhara Xuesong Yang Edresson Casanova Subhankar Ghosh Mikyas T. Desta Roy Fejgin Rafael Valle Jason Chun Lok Li 59 2 0 07 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 115 3 0 06 Feb 2025
Process Reinforcement through Implicit Rewards Ganqu Cui Lifan Yuan Z. Wang Hanbin Wang Wendi Li ... Yu Cheng Zhiyuan Liu Maosong Sun Bowen Zhou Ning Ding OffRL LRM 73 52 0 03 Feb 2025
The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking Yuchun Miao Sen Zhang Liang Ding Yuqi Zhang L. Zhang Dacheng Tao 81 3 0 31 Jan 2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking Benjamin Feuer Micah Goldblum Teresa Datta Sanjana Nambiar Raz Besaleli Samuel Dooley Max Cembalest John P. Dickerson ALM 37 7 0 28 Jan 2025
LiPO: Listwise Preference Optimization through Learning-to-Rank Tianqi Liu Zhen Qin Junru Wu Jiaming Shen Misha Khalman ... Mohammad Saleh Simon Baumgartner Jialu Liu Peter J. Liu Xuanhui Wang 133 48 0 28 Jan 2025
Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning Hao Sun M. Schaar 92 14 0 28 Jan 2025
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback Yafu Li Xuyang Hu Xiaoye Qu Linjie Li Yu-Xi Cheng 51 3 0 22 Jan 2025
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion Yannis Flet-Berliac Nathan Grinsztajn Florian Strub Bill Wu Eugene Choi ... Arash Ahmadian Yash Chandak M. G. Azar Olivier Pietquin Matthieu Geist OffRL 62 4 0 17 Jan 2025
Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques Natalia Zhang X. Wang Qiwen Cui Runlong Zhou Sham Kakade Simon S. Du OffRL 48 0 0 10 Jan 2025