The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR
Summarization

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

24 March 2024

Michael Noukhovitch

Papers citing "The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization"

11 / 11 papers shown

Title
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 35 0 0 03 Mar 2025
A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning Shashank Gupta Chaitanya Ahuja Tsung-Yu Lin Sreya Dutta Roy Harrie Oosterhuis Maarten de Rijke Satya Narayan Shukla 46 1 0 02 Mar 2025
Cautious Optimizers: Improving Training with One Line of Code Kaizhao Liang Lizhang Chen B. Liu Qiang Liu ODL 98 5 0 25 Nov 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 46 4 0 23 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Aaron C. Courville OffRL 77 5 0 23 Oct 2024
Science Out of Its Ivory Tower: Improving Accessibility with Reinforcement Learning Haining Wang Jason Clark Hannah McKelvey Leila Sterman Zheng Gao Zuoyu Tian Sandra Kübler Xiaozhong Liu 33 1 0 22 Oct 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 59 23 0 23 Aug 2024
Variational Best-of-N Alignment Afra Amini Tim Vieira Ryan Cotterell Ryan Cotterell BDL 43 18 0 08 Jul 2024
Language Model Alignment with Elastic Reset Michael Noukhovitch Samuel Lavoie Florian Strub Aaron Courville KELM 87 25 0 06 Dec 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,881 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,583 0 18 Sep 2019