A Minimaximalist Approach to Reinforcement Learning from Human Feedback

8 January 2024

Papers citing "A Minimaximalist Approach to Reinforcement Learning from Human Feedback"

50 / 78 papers shown

Title
A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future Jialun Zhong Wei Shen Yanzeng Li Songyang Gao Hua Lu Yicheng Chen Yang Zhang Wei Zhou Jinjie Gu Lei Zou LRM 38 2 0 12 Apr 2025
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Kai Ye Hongyi Zhou Jin Zhu Francesco Quinzan C. Shi 20 0 0 03 Apr 2025
A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective Zhuoren Li Guizhe Jin Ran Yu Z. Chen Nan I. Li ... Lu Xiong Bo Leng Jia Hu I. Kolmanovsky Dimitar Filev 44 0 0 31 Mar 2025
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 71 0 0 10 Mar 2025
Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning Hyungkyu Kang Min-hwan Oh OffRL 42 0 0 07 Mar 2025
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning Gokul Swamy Sanjiban Choudhury Wen Sun Zhiwei Steven Wu J. Andrew Bagnell OffRL 42 7 0 03 Mar 2025
Behavior Preference Regression for Offline Reinforcement Learning Padmanaba Srinivasan William J. Knottenbelt OffRL 31 0 0 02 Mar 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 74 0 0 26 Feb 2025
Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems Hao Peng Y. Qi Xiaozhi Wang Zijun Yao Bin Xu Lei Hou Juanzi Li ALM LRM 52 4 0 26 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 72 0 0 25 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 60 23 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 36 4 0 20 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Z. Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 35 1 0 20 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu V. Cevher 49 0 0 18 Feb 2025
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 80 0 0 14 Feb 2025
Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games Tong Yang Bo Dai Lin Xiao Yuejie Chi OffRL 56 2 0 13 Feb 2025
Game Theory Meets Large Language Models: A Systematic Survey Haoran Sun Yusen Wu Yukun Cheng Xu Chu LM&MA OffRL AI4CE 55 1 0 13 Feb 2025
ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates L. Yang Zhaochen Yu Bin Cui Mengdi Wang ReLM LRM AI4CE 94 10 0 10 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 45 0 0 08 Feb 2025
The Battling Influencers Game: Nash Equilibria Structure of a Potential Game and Implications to Value Alignment Young Wu Yancheng Zhu Jin-Yi Cai Xiaojin Zhu 94 0 0 03 Feb 2025
Clone-Robust AI Alignment Ariel D. Procaccia Benjamin G. Schiffer Shirley Zhang 24 1 0 17 Jan 2025
Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models Roberto-Rafael Maura-Rivero Chirag Nagpal Roma Patel Francesco Visin 46 1 0 08 Jan 2025
Geometric-Averaged Preference Optimization for Soft Preference Labels Hiroki Furuta Kuang-Huei Lee Shixiang Shane Gu Y. Matsuo Aleksandra Faust Heiga Zen Izzeddin Gur 50 6 0 31 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang J. Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 123 7 0 05 Dec 2024
Policy Aggregation Parand A. Alamdari Soroush Ebadian Ariel D. Procaccia OffRL 24 4 0 06 Nov 2024
Soft Condorcet Optimization for Ranking of General Agents Marc Lanctot Kate Larson Michael Kaisers Quentin Berthet I. Gemp Manfred Diaz Roberto-Rafael Maura-Rivero Yoram Bachrach Anna Koop Doina Precup 32 0 0 31 Oct 2024
Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data Xinhong Xie Tao Li Quanyan Zhu 16 3 0 27 Oct 2024
Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models Muhan Lin Shuyang Shi Yue (Sophie) Guo Behdad Chalaki Vaishnav Tadiparthi Ehsan Moradi-Pari Simon Stepputtis Joseph Campbell Katia P. Sycara 33 1 0 22 Oct 2024
Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment Mingzhi Wang Chengdong Ma Qizhi Chen Linjian Meng Yang Han Jiancong Xiao Zhaowei Zhang Jing Huo Weijie Su Yaodong Yang 30 4 0 22 Oct 2024
Accelerated Preference Optimization for Large Language Model Alignment Jiafan He Huizhuo Yuan Q. Gu 21 1 0 08 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 54 3 0 06 Oct 2024
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment Yifan Zhang Ge Zhang Yue Wu Kangping Xu Quanquan Gu 37 3 0 03 Oct 2024
The Crucial Role of Samplers in Online Direct Preference Optimization Ruizhe Shi Runlong Zhou Simon S. Du 53 7 0 29 Sep 2024
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment Yan Liu Xiaoyuan Yi Xiaokang Chen Jing Yao Jingwei Yi Daoguang Zan Zheng Liu Xing Xie Tsung-Yi Ho ALM 26 0 0 26 Sep 2024
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do Guijin Son Hyunwoo Ko Hoyoung Lee Yewon Kim Seunghyeok Hong ALM ELM 46 5 0 17 Sep 2024
Critique-out-Loud Reward Models Zachary Ankner Mansheej Paul Brandon Cui Jonathan D. Chang Prithviraj Ammanabrolu ALM LRM 25 25 0 21 Aug 2024
Personality Alignment of Large Language Models Minjun Zhu Linyi Yang Yue Zhang Yue Zhang ALM 52 5 0 21 Aug 2024
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning S. Poddar Yanming Wan Hamish Ivison Abhishek Gupta Natasha Jaques 27 33 0 19 Aug 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 27 4 0 26 Jul 2024
Data-Centric Human Preference Optimization with Rationales H. Just Ming Jin Anit Kumar Sahu Huy Phan Ruoxi Jia 30 3 0 19 Jul 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 38 8 0 25 Jun 2024
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold Amrith Rajagopal Setlur Saurabh Garg Xinyang Geng Naman Garg Virginia Smith Aviral Kumar 35 45 0 20 Jun 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 44 132 0 18 Jun 2024
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning Jifan Zhang Lalit P. Jain Yang Guo Jiayi Chen Kuan Lok Zhou ... Scott Sievert Timothy Rogers Kevin Jamieson Robert Mankoff Robert Nowak 29 5 0 15 Jun 2024
ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions Xu Zhang Xunjian Yin Xiaojun Wan 40 3 0 13 Jun 2024
Discovering Preference Optimization Algorithms with and for Large Language Models Chris Xiaoxuan Lu Samuel Holt Claudio Fanconi Alex J. Chan Jakob Foerster M. Schaar R. T. Lange OffRL 29 14 0 12 Jun 2024
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models Xiang Ji Sanjeev Kulkarni Mengdi Wang Tengyang Xie OffRL 29 4 0 06 Jun 2024
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF Tengyang Xie Dylan J. Foster Akshay Krishnamurthy Corby Rosset Ahmed Hassan Awadallah Alexander Rakhlin 36 32 0 31 May 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 19 21 0 30 May 2024
Offline Regularised Reinforcement Learning for Large Language Models Alignment Pierre Harvey Richemond Yunhao Tang Daniel Guo Daniele Calandriello M. G. Azar ... Gil Shamir Rishabh Joshi Tianqi Liu Rémi Munos Bilal Piot OffRL 40 21 0 29 May 2024