Dueling RL: Reinforcement Learning with Trajectory Preferences

8 November 2021

Papers citing "Dueling RL: Reinforcement Learning with Trajectory Preferences"

50 / 61 papers shown

Title
Learning Guarantee of Reward Modeling Using Deep Neural Networks Yuanhang Luo Yeheng Ge Ruijian Han Guohao Shen 31 0 0 10 May 2025
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback Nan Lu Ethan X. Fang Junwei Lu 132 0 0 27 Apr 2025
Reinforcement Learning from Multi-level and Episodic Human Feedback Muhammad Qasim Elahi Somtochukwu Oguchienti Maheed H. Ahmed Mahsa Ghasemi OffRL 47 0 0 20 Apr 2025
Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning Hyungkyu Kang Min-hwan Oh OffRL 45 0 0 07 Mar 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 80 0 0 26 Feb 2025
Online Learning from Strategic Human Feedback in LLM Fine-Tuning Shugang Hao Lingjie Duan 84 3 0 22 Dec 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 42 28 0 26 Oct 2024
Optimal Design for Reward Modeling in RLHF Antoine Scheid Etienne Boursier Alain Durmus Michael I. Jordan Pierre Ménard Eric Moulines Michal Valko OffRL 45 6 0 22 Oct 2024
A Theoretical Survey on Foundation Models Shi Fu Yuzhu Chen Yingjie Wang Dacheng Tao 28 0 0 15 Oct 2024
DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback Guojun Xiong Ujwal Dinesha Debajoy Mukherjee Jian Li Srinivas Shakkottai 42 2 0 07 Oct 2024
Almost Free: Self-concordance in Natural Exponential Families and an Application to Bandits Shuai Liu Alex Ayoub Flore Sentenac Xiaoqi Tan Csaba Szepesvári 26 1 0 01 Oct 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 37 2 0 25 Sep 2024
Advances in Preference-based Reinforcement Learning: A Review Youssef Abdelkareem Shady Shehata Fakhri Karray OffRL 51 9 0 21 Aug 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 34 4 0 26 Jul 2024
Conversational Dueling Bandits in Generalized Linear Models Shuhua Yang Hui Yuan Xiaoying Zhang Mengdi Wang Hong Zhang Huazheng Wang 33 1 0 26 Jul 2024
Preference-Guided Reinforcement Learning for Efficient Exploration Guojian Wang Faguo Wu Xiao Zhang Tianyuan Chen Xuyang Chen Lin Zhao 38 0 0 09 Jul 2024
Preference Elicitation for Offline Reinforcement Learning Alizée Pace Bernhard Schölkopf Gunnar Rätsch Giorgia Ramponi OffRL 61 1 0 26 Jun 2024
Bandits with Preference Feedback: A Stackelberg Game Perspective Barna Pásztor Parnian Kassraie Andreas Krause 40 2 0 24 Jun 2024
Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis Qining Zhang Honghao Wei Lei Ying OffRL 62 1 0 11 Jun 2024
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models Xiang Ji Sanjeev Kulkarni Mengdi Wang Tengyang Xie OffRL 43 4 0 06 Jun 2024
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF Tengyang Xie Dylan J. Foster Akshay Krishnamurthy Corby Rosset Ahmed Hassan Awadallah Alexander Rakhlin 43 33 0 31 May 2024
Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity Yan Yang Bin Gao Ya-xiang Yuan 38 2 0 30 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 40 42 0 26 May 2024
Multi-turn Reinforcement Learning from Preference Human Feedback Lior Shani Aviv Rosenberg Asaf B. Cassel Oran Lang Daniele Calandriello ... Bilal Piot Idan Szpektor Avinatan Hassidim Yossi Matias Rémi Munos 47 26 0 23 May 2024
Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedback Asaf B. Cassel Haipeng Luo Aviv A. Rosenberg Dmitry Sotnikov OffRL 29 3 0 13 May 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Guhao Feng Guhao Feng Li Zhao Di He Jiang Bian Liwei Wang Jiang Bian Liwei Wang 55 56 0 29 Apr 2024
Optimal Design for Human Feedback Subhojyoti Mukherjee Anusha Lalitha Kousha Kalantari Aniket Deshmukh Ge Liu Yifei Ma B. Kveton 36 0 0 22 Apr 2024
Dataset Reset Policy Optimization for RLHF Jonathan D. Chang Wenhao Zhan Owen Oertell Kianté Brantley Dipendra Kumar Misra Jason D. Lee Wen Sun OffRL 22 21 0 12 Apr 2024
DP-Dueling: Learning from Preference Feedback without Compromising User Privacy Aadirupa Saha Hilal Asi 36 1 0 22 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 22 55 0 01 Mar 2024
RLVF: Learning from Verbal Feedback without Overgeneralization Moritz Stephan Alexander Khazatsky Eric Mitchell Annie S. Chen Sheryl Hsu Archit Sharma Chelsea Finn 34 11 0 16 Feb 2024
Active Preference Optimization for Sample Efficient RLHF Nirjhar Das Souradip Chakraborty Aldo Pacchiano Sayak Ray Chowdhury 27 13 0 16 Feb 2024
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment Rui Yang Xiaoman Pan Feng Luo Shuang Qiu Han Zhong Dong Yu Jianshu Chen 100 66 0 15 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 38 9 0 11 Feb 2024
Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF Han Shen Zhuoran Yang Tianyi Chen OffRL 32 14 0 10 Feb 2024
Corruption Robust Offline Reinforcement Learning with Human Feedback Debmalya Mandal Andi Nika Parameswaran Kamalaruban Adish Singla Goran Radanović OffRL 28 8 0 09 Feb 2024
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF Banghua Zhu Michael I. Jordan Jiantao Jiao 28 25 0 29 Jan 2024
Think Before You Duel: Understanding Complexities of Preference Learning under Constrained Resources Rohan Deb Aadirupa Saha 25 0 0 28 Dec 2023
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint Wei Xiong Hanze Dong Chen Ye Ziqi Wang Han Zhong Heng Ji Nan Jiang Tong Zhang OffRL 38 157 0 18 Dec 2023
Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF Anand Siththaranjan Cassidy Laidlaw Dylan Hadfield-Menell 29 55 0 13 Dec 2023
Nash Learning from Human Feedback Rémi Munos Michal Valko Daniele Calandriello M. G. Azar Mark Rowland ... Nikola Momchev Olivier Bachem D. Mankowitz Doina Precup Bilal Piot 28 123 0 01 Dec 2023
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model Kai Yang Jian Tao Jiafei Lyu Chunjiang Ge Jiaxin Chen Qimai Li Weihan Shen Xiaolong Zhu Xiu Li EGVM 19 89 0 22 Nov 2023
Conditions on Preference Relations that Guarantee the Existence of Optimal Policies Jonathan Colaco Carr Prakash Panangaden Doina Precup 26 1 0 03 Nov 2023
Differentially Private Reward Estimation with Preference Feedback Sayak Ray Chowdhury Xingyu Zhou Nagarajan Natarajan 38 4 0 30 Oct 2023
COPR: Continual Learning Human Preference through Optimal Policy Regularization Han Zhang Lin Gui Yuanzhao Zhai Hui Wang Yu Lei Ruifeng Xu CLL 43 0 0 24 Oct 2023
A General Theoretical Paradigm to Understand Learning from Human Preferences M. G. Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos 45 532 0 18 Oct 2023
Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks Zihao Li Xiang Ji Minshuo Chen Mengdi Wang OffRL 21 0 0 16 Oct 2023
When is Agnostic Reinforcement Learning Statistically Tractable? Zeyu Jia Gene Li Alexander Rakhlin Ayush Sekhari Nathan Srebro OffRL 27 5 0 09 Oct 2023
Reinforcement Learning-based Mixture of Vision Transformers for Video Violence Recognition Hamid Reza Mohammadi Ehsan Nazerfard Tahereh Firoozi ViT 25 2 0 04 Oct 2023
PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback Souradip Chakraborty Amrit Singh Bedi Alec Koppel Dinesh Manocha Huazheng Wang Mengdi Wang Furong Huang 23 25 0 03 Aug 2023