Scaling Laws for Reward Model Overoptimization

19 October 2022

Papers citing "Scaling Laws for Reward Model Overoptimization"

50 / 89 papers shown

Title
DanceGRPO: Unleashing GRPO on Visual Generation Zeyue Xue Jie Wu Yu Gao Fangyuan Kong Lingting Zhu ... Zhiheng Liu Wei Liu Qiushan Guo Weilin Huang Ping Luo EGVM VGen 47 0 0 12 May 2025
Soft Best-of-n Sampling for Model Alignment C. M. Verdun Alex Oesterling Himabindu Lakkaraju Flavio du Pin Calmon BDL 65 0 0 06 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 37 0 0 05 May 2025
Phi-4-reasoning Technical Report Marah Abdin Sahaj Agarwal Ahmed Hassan Awadallah Vidhisha Balachandran Harkirat Singh Behl ... Vaishnavi Shrivastava Vibhav Vineet Yue Wu Safoora Yousefi Guoqing Zheng ReLM LRM 77 0 0 30 Apr 2025
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Jiabin Fan Guoqing Luo Michael Bowling Lili Mou OffRL 63 0 0 26 Apr 2025
Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL Simone Papicchio Simone Rossi Luca Cagliero Paolo Papotti ReLM LMTD AI4TS LRM 53 0 0 21 Apr 2025
SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM X. Zhang J. Wang Zifei Cheng Wenhao Zhuang Zheng Lin ... Shouyu Yin Chaohang Wen Haotian Zhang Bin Chen Bing Yu LRM 33 2 0 19 Apr 2025
Energy-Based Reward Models for Robust Language Model Alignment Anamika Lochab Ruqi Zhang 59 0 0 17 Apr 2025
Adversarial Training of Reward Models Alexander Bukharin Haifeng Qian Shengyang Sun Adithya Renduchintala Soumye Singhal Z. Wang Oleksii Kuchaiev Olivier Delalleau T. Zhao AAML 29 0 0 08 Apr 2025
Inference-Time Scaling for Generalist Reward Modeling Zijun Liu P. Wang R. Xu Shirong Ma Chong Ruan Peng Li Yang Janet Liu Y. Wu OffRL LRM 46 9 0 03 Apr 2025
Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback Wei Shen Guanlin Liu Zheng Wu Ruofei Zhu Qingping Yang Chao Xin Yu Yue Lin Yan 82 8 0 28 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Y. Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang T. Tan 82 2 0 18 Mar 2025
Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation Bowen Baker Joost Huizinga Leo Gao Zehao Dou M. Guan Aleksander Mądry Wojciech Zaremba J. Pachocki David Farhi LRM 62 11 0 14 Mar 2025
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 71 0 0 10 Mar 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 74 0 0 26 Feb 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 85 1 0 26 Feb 2025
A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics Ting-Ruen Wei Haowei Liu Xuyang Wu Yi Fang LRM AI4CE ReLM KELM 119 1 0 21 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 93 5 0 21 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 60 23 0 20 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Z. Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 35 1 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 36 4 0 20 Feb 2025
Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? Xueru Wen Jie Lou Y. Lu Hongyu Lin Xing Yu Xinyu Lu Ben He Xianpei Han Debing Zhang Le Sun ALM 53 4 0 17 Feb 2025
Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis Kaikai Zhao Zhaoxiang Liu Xuejiao Lei Ning Wang Zhenhong Long ... W. Liu Kai Wang Wen Liu Kai Wang Shiguo Lian ELM LRM 52 1 0 16 Feb 2025
Leveraging Reasoning with Guidelines to Elicit and Utilize Knowledge for Enhancing Safety Alignment Haoyu Wang Zeyu Qin Li Shen Xueqian Wang Minhao Cheng Dacheng Tao 86 1 0 06 Feb 2025
Out-of-Distribution Detection using Synthetic Data Generation Momin Abbas Muneeza Azmat R. Horesh Mikhail Yurochkin 38 1 0 05 Feb 2025
Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning Hao Sun M. Schaar 82 14 0 28 Jan 2025
LiPO: Listwise Preference Optimization through Learning-to-Rank Tianqi Liu Zhen Qin Junru Wu Jiaming Shen Misha Khalman ... Mohammad Saleh Simon Baumgartner Jialu Liu Peter J. Liu Xuanhui Wang 133 47 0 28 Jan 2025
Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Zhenyu Hou Xin Lv Rui Lu J. Zhang Y. Li Zijun Yao Juanzi Li J. Tang Yuxiao Dong OffRL LRM ReLM 49 20 0 20 Jan 2025
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems Hashmath Shaik Alex Doboli OffRL ELM 74 0 0 31 Dec 2024
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization Chia-Yu Hung Navonil Majumder Zhifeng Kong Ambuj Mehrish Rafael Valle Bryan Catanzaro Soujanya Poria Bryan Catanzaro Soujanya Poria 52 5 0 30 Dec 2024
Interpreting Language Reward Models via Contrastive Explanations Junqi Jiang Tom Bewley Saumitra Mishra Freddy Lecue Manuela Veloso 74 0 0 25 Nov 2024
Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward Zhiwei Jia Yuesong Nan Huixi Zhao Gengdai Liu EGVM 84 0 0 22 Nov 2024
Efficient Alignment of Large Language Models via Data Sampling Amrit Khera Rajat Ghosh Debojyoti Dutta 31 1 0 15 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 73 0 0 12 Nov 2024
L3Ms -- Lagrange Large Language Models Guneet S. Dhillon Xingjian Shi Yee Whye Teh Alex Smola 58 0 0 28 Oct 2024
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization Xiyue Peng Hengquan Guo Jiawei Zhang Dongqing Zou Ziyu Shao Honghao Wei Xin Liu 32 0 0 25 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 46 4 0 23 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Aaron C. Courville OffRL 77 5 0 23 Oct 2024
Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen Binxuan Huang Yifan Gao Zhengyang Wang Jingfeng Yang Heng Ji LRM 43 7 0 11 Oct 2024
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment Yuancheng Xu Udari Madhushani Sehwag Alec Koppel Sicheng Zhu Bang An Furong Huang Sumitra Ganesh 52 6 0 10 Oct 2024
RL, but don't do anything I wouldn't do Michael K. Cohen Marcus Hutter Yoshua Bengio Stuart J. Russell OffRL 25 2 0 08 Oct 2024
Frame-Voyager: Learning to Query Frames for Video Large Language Models Sicheng Yu Chengkai Jin Huanyu Wang Zhenghao Chen Sheng Jin ... Zhenbang Sun Bingni Zhang Jiawei Wu Hao Zhang Qianru Sun 64 5 0 04 Oct 2024
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions Yekun Chai Haoran Sun Huang Fang Shuohuan Wang Yu Sun Hua-Hong Wu 56 1 0 03 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 46 4 0 30 Sep 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 84 5 0 13 Sep 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 37 11 0 11 Sep 2024
Efficient Multi-agent Navigation with Lightweight DRL Policy Xingrong Diao Jiankun Wang 40 0 0 29 Aug 2024
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data Chenglong Wang Yang Gan Yifu Huo Yongyu Mu Murun Yang ... Chunliang Zhang Tongran Liu Quan Du Di Yang Jingbo Zhu VLM 61 4 0 22 Aug 2024
Practical Aspects on Solving Differential Equations Using Deep Learning: A Primer Georgios Is. Detorakis 13 0 0 21 Aug 2024