The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

31 October 2023

Papers citing "The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback"

17 / 17 papers shown

Title
AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation Tuhin Chakrabarty Philippe Laban C. Wu 32 1 0 10 Apr 2025
RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Enyu Zhou Guodong Zheng Binghui Wang Zhiheng Xi Shihan Dou ... Yurong Mou Rui Zheng Tao Gui Qi Zhang Xuanjing Huang ALM 59 18 0 13 Oct 2024
Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification Thomas Kwa Drake Thomas Adrià Garriga-Alonso 26 1 0 19 Jul 2024
It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF Taiming Lu Lingfeng Shen Xinyu Yang Weiting Tan Beidi Chen Huaxiu Yao 58 2 0 12 Jun 2024
Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding Kuo Liao Shuang Li Meng Zhao Liqun Liu Mengge Xue Zhenyu Hu Honglin Han Chengguo Yin 40 1 0 30 May 2024
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment Gerald Shen Zhilin Wang Olivier Delalleau Jiaqi Zeng Yi Dong ... Sahil Jain Ali Taghibakhshi Markel Sanz Ausin Ashwath Aithal Oleksii Kuchaiev 40 13 0 02 May 2024
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback Vincent Conitzer Rachel Freedman J. Heitzig Wesley H. Holliday Bob M. Jacobs ... Eric Pacuit Stuart Russell Hailey Schoelkopf Emanuel Tewolde W. Zwicker 33 28 0 16 Apr 2024
Understanding the Effects of RLHF on LLM Generalisation and Diversity Robert Kirk Ishita Mediratta Christoforos Nalmpantis Jelena Luketina Eric Hambro Edward Grefenstette Roberta Raileanu AI4CE ALM 106 122 0 10 Oct 2023
Stabilizing RLHF through Advantage Model and Selective Rehearsal Baolin Peng Linfeng Song Ye Tian Lifeng Jin Haitao Mi Dong Yu 35 17 0 18 Sep 2023
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation Patrick Fernandes Aman Madaan Emmy Liu António Farinhas Pedro Henrique Martins ... José G. C. de Souza Shuyan Zhou Tongshuang Wu Graham Neubig André F. T. Martins ALM 117 56 0 01 May 2023
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 227 502 0 28 Sep 2022
Offline RL for Natural Language Generation with Implicit Language Q Learning Charles Burton Snell Ilya Kostrikov Yi Su Mengjiao Yang Sergey Levine OffRL 125 102 0 05 Jun 2022
Teaching language models to support answers with verified quotes Jacob Menick Maja Trebacz Vladimir Mikulik John Aslanides Francis Song ... Mia Glaese Susannah Young Lucy Campbell-Gillingham G. Irving Nat McAleese ELM RALM 243 257 0 21 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 367 8,495 0 28 Jan 2022
$Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information$ Understanding Dataset Difficulty with $\mathcal{V}$ -Usable Information Kawin Ethayarajh Yejin Choi Swabha Swayamdipta 164 157 0 16 Oct 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019