What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret

3 March 2025

Papers citing "What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret"

11 / 11 papers shown

Title
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning Fanqi Wan Weizhou Shen Shengyi Liao Yingcheng Shi Chenliang Li Ziyi Yang Ji Zhang Fei Huang Jingren Zhou Ming Yan OffRL LLMAG ReLM LRM 45 0 0 23 May 2025
Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL Songjun Tu Jiahao Lin Qichao Zhang Xiangyu Tian Linjing Li Xiangyuan Lan Dongbin Zhao OffRL ReLM LRM 56 2 0 16 May 2025
Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models Guanghao Zhou Panjia Qiu Chong Chen Jiadong Wang Zheming Yang Jian Xu Minghui Qiu OffRL LRM 104 4 0 30 Apr 2025
Reinforcement Learning for Reasoning in Large Language Models with One Training Example Yiping Wang Qing Yang Zhiyuan Zeng Liliang Ren Liu Liu ... Jianfeng Gao Weizhu Chen Shuaiqiang Wang Simon Shaolei Du Yelong Shen OffRL ReLM LRM 210 23 0 29 Apr 2025
Concise Reasoning via Reinforcement Learning Mehdi Fatemi Banafsheh Rafiee Mingjie Tang Kartik Talamadupula ReLM OffRL LRM 88 11 0 07 Apr 2025
VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks Yu Yue Yufeng Yuan Qiying Yu Xiaochen Zuo Ruofei Zhu ... Ru Zhang Xin Liu Mingxuan Wang Yonghui Wu Lin Yan OffRL LRM 91 20 0 07 Apr 2025
DAPO: An Open-Source LLM Reinforcement Learning System at Scale Qiying Yu Zheng Zhang Ruofei Zhu Yufeng Yuan Xiaochen Zuo ... Ya Zhang Lin Yan Mu Qiao Yonghui Wu Mingxuan Wang OffRL LRM 125 131 0 18 Mar 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song ... Shiyu Wang S. Yu Shunfeng Zhou Shuting Pan S.S. Li ReLM VLM OffRL AI4TS LRM 256 1,503 0 22 Jan 2025
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 598 9,009 0 28 Jan 2022
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 248 18,685 0 20 Jul 2017
The Arcade Learning Environment: An Evaluation Platform for General Agents Marc G. Bellemare Yavar Naddaf J. Veness Michael Bowling 87 2,992 0 19 Jul 2012