The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

10 January 2022

Papers citing "The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models"

50 / 128 papers shown

Title
Reasoning Models Don't Always Say What They Think Yanda Chen Joe Benton Ansh Radhakrishnan Jonathan Uesato Carson E. Denison ... Vlad Mikulik Samuel R. Bowman Jan Leike Jared Kaplan E. Perez ReLM LRM 65 7 1 08 May 2025
An alignment safety case sketch based on debate Marie Davidsen Buhl Jacob Pfau Benjamin Hilton Geoffrey Irving 30 0 0 06 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 62 0 0 05 May 2025
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Jiabin Fan Guoqing Luo Michael Bowling Lili Mou OffRL 61 0 0 26 Apr 2025
Redefining Superalignment: From Weak-to-Strong Alignment to Human-AI Co-Alignment to Sustainable Symbiotic Society Feifei Zhao Y. Wang Enmeng Lu Dongcheng Zhao Bing Han ... Chao Liu Yaodong Yang Yi Zeng Boyuan Chen Jinyu Fan 80 0 0 24 Apr 2025
Establishing Reliability Metrics for Reward Models in Large Language Models Yizhou Chen Yawen Liu Xuesi Wang Qingtao Yu Guangda Huzhang Anxiang Zeng Han Yu Zhiming Zhou 30 0 0 21 Apr 2025
A Rollout-Based Algorithm and Reward Function for Efficient Resource Allocation in Business Processes Jeroen Middelhuis Z. Bukhsh Ivo Adan R. Dijkman 24 0 0 15 Apr 2025
Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations Pedro Ferreira Wilker Aziz Ivan Titov LRM 26 0 0 07 Apr 2025
Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning Xuerui Su Shufang Xie Guoqing Liu Yingce Xia Renqian Luo Peiran Jin Zhiming Ma Yue Wang Zun Wang Yuting Liu LRM 27 1 0 06 Apr 2025
On the Connection Between Diffusion Models and Molecular Dynamics Liam Harcombe Timothy T. Duignan DiffM 43 1 0 04 Apr 2025
PaperBench: Evaluating AI's Ability to Replicate AI Research Giulio Starace Oliver Jaffe Dane Sherburn James Aung Jun Shern Chan ... Benjamin Kinsella Wyatt Thompson Johannes Heidecke Amelia Glaese Tejal Patwardhan ALM ELM 772 5 0 02 Apr 2025
MultiScale Contextual Bandits for Long Term Objectives Richa Rastogi Yuta Saito Thorsten Joachims OffRL 40 0 0 22 Mar 2025
Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation Bowen Baker Joost Huizinga Leo Gao Zehao Dou M. Guan Aleksander Mądry Wojciech Zaremba J. Pachocki David Farhi LRM 62 11 0 14 Mar 2025
Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners Calarina Muslimani Kerrick Johnstonbaugh Suyog Chandramouli Serena Booth W. B. Knox M. E. Taylor 34 0 0 08 Mar 2025
High-Precision Transformer-Based Visual Servoing for Humanoid Robots in Aligning Tiny Objects Jialong Xue Wei Gao Yu Wang Chao Ji Dongdong Zhao Shi Yan Shiwu Zhang 43 0 0 06 Mar 2025
Adding Alignment Control to Language Models Wenhong Zhu Weinan Zhang Rui Wang 50 0 0 06 Mar 2025
Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarm H. Kim Kanghoon Lee J. Park Jiachen Li Jinkyoo Park 58 1 0 05 Mar 2025
Subtask-Aware Visual Reward Learning from Segmented Demonstrations Changyeon Kim Minho Heo Doohyun Lee Jinwoo Shin Honglak Lee Joseph J. Lim Kimin Lee 32 0 0 28 Feb 2025
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment Tianze Wang Dongnan Gui Yifan Hu Shuhang Lin Linjun Zhang 31 0 0 25 Feb 2025
Unhackable Temporal Rewarding for Scalable Video MLLMs En Yu Kangheng Lin Liang Zhao Yana Wei Zining Zhu ... Jianjian Sun Zheng Ge X. Zhang Jingyu Wang Wenbing Tao 52 4 0 17 Feb 2025
Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization Zishun Yu Tengyu Xu Di Jin Karthik Abinav Sankararaman Yun He ... Eryk Helenowski Chen Zhu Sinong Wang Hao Ma Han Fang LRM 49 4 0 29 Jan 2025
Kimi k1.5: Scaling Reinforcement Learning with LLMs Kimi Team Angang Du Bofei Gao Bowei Xing Changjiu Jiang ... Zhilin Yang Zhiqi Huang Zihao Huang Ziyao Xu Z. Yang VLM ALM OffRL AI4TS LRM 106 128 0 22 Jan 2025
Learning to Assist Humans without Inferring Rewards Vivek Myers Evan Ellis Sergey Levine Benjamin Eysenbach Anca Dragan 28 2 0 17 Jan 2025
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment Chaoqi Wang Zhuokai Zhao Yibo Jiang Zhaorun Chen Chen Zhu ... Jiayi Liu Lizhu Zhang Xiangjun Fan Hao Ma Sinong Wang 70 3 0 17 Jan 2025
When Can Proxies Improve the Sample Complexity of Preference Learning? Yuchen Zhu Daniel Augusto de Souza Zhengyan Shi Mengyue Yang Pasquale Minervini Alexander DÁmour Matt J. Kusner 66 0 0 21 Dec 2024
Test-Time Alignment via Hypothesis Reweighting Yoonho Lee Jonathan Williams Henrik Marklund Archit Sharma E. Mitchell Anikait Singh Chelsea Finn 91 3 0 11 Dec 2024
Towards Data Governance of Frontier AI Models Jason Hausenloy Duncan McClements Madhavendra Thakur 62 1 0 05 Dec 2024
Drowning in Documents: Consequences of Scaling Reranker Inference Mathew Jacob Erik Lindgren Matei A. Zaharia Michael Carbin Omar Khattab Andrew Drozdov OffRL 74 4 0 18 Nov 2024
Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games Usman Anwar Ashish Pandian Jia Wan David M. Krueger Jakob N. Foerster 29 0 0 07 Nov 2024
Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment Weichao Zhou Wenchao Li 23 0 0 31 Oct 2024
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning Jared Joselowitz Arjun Jagota Satyapriya Krishna Sonali Parbhoo Nyal Patel Satyapriya Krishna Sonali Parbhoo 19 0 0 16 Oct 2024
Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack Leo McKee-Reid Christoph Sträter Maria Angelica Martinez Joe Needham Mikita Balesni OffRL 23 1 0 09 Oct 2024
Evaluating Robustness of Reward Models for Mathematical Reasoning Sunghwan Kim Dongjin Kang Taeyoon Kwon Hyungjoo Chae Jungsoo Won Dongha Lee Jinyoung Yeo 23 4 0 02 Oct 2024
The Perfect Blend: Redefining RLHF with Mixture of Judges Tengyu Xu Eryk Helenowski Karthik Abinav Sankararaman Di Jin Kaiyan Peng ... Gabriel Cohen Yuandong Tian Hao Ma Sinong Wang Han Fang 31 9 0 30 Sep 2024
Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations Peixin Qin Chen Huang Yang Deng Wenqiang Lei Tat-Seng Chua LRM 17 3 0 22 Sep 2024
Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback Jiayi Zhou Jiaming Ji Juntao Dai Yaodong Yang 37 4 0 30 Aug 2024
SpecGuard: Specification Aware Recovery for Robotic Autonomous Vehicles from Physical Attacks Pritam Dash Ethan Chan Karthik Pattabiraman AAML 26 3 0 27 Aug 2024
Can a Bayesian Oracle Prevent Harm from an Agent? Yoshua Bengio Michael K. Cohen Nikolay Malkin Matt MacDermott Damiano Fornasiere Pietro Greiner Younesse Kaddar 34 4 0 09 Aug 2024
On the Generalization of Preference Learning with DPO Shawn Im Yixuan Li 33 1 0 06 Aug 2024
Strong and weak alignment of large language models with human values Mehdi Khamassi Marceau Nahon Raja Chatila ALM 27 9 0 05 Aug 2024
Value Internalization: Learning and Generalizing from Social Reward Frieda Rong Max Kleiman-Weiner 31 1 0 19 Jul 2024
AI Safety in Generative AI Large Language Models: A Survey Jaymari Chua Yun Yvonna Li Shiyi Yang Chen Wang Lina Yao LM&MA 34 12 0 06 Jul 2024
Towards shutdownable agents via stochastic choice Elliott Thornley Alexander Roman Christos Ziakas Leyton Ho Louis Thomson 25 0 0 30 Jun 2024
Monitoring Latent World States in Language Models with Propositional Probes Jiahai Feng Stuart Russell Jacob Steinhardt HILM 27 6 0 27 Jun 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 47 13 0 24 Jun 2024
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models Carson E. Denison M. MacDiarmid Fazl Barez D. Duvenaud Shauna Kravec ... Jared Kaplan Buck Shlegeris Samuel R. Bowman Ethan Perez Evan Hubinger 32 35 0 14 Jun 2024
It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF Taiming Lu Lingfeng Shen Xinyu Yang Weiting Tan Beidi Chen Huaxiu Yao 43 2 0 12 Jun 2024
Robust Reward Design for Markov Decision Processes Shuo Wu Haoxiang Ma Jie Fu Shuo Han 19 1 0 07 Jun 2024
Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models Xiang Ji Sanjeev Kulkarni Mengdi Wang Tengyang Xie OffRL 27 4 0 06 Jun 2024
HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning Quentin Delfosse Jannis Blüml Bjarne Gregori Kristian Kersting 26 7 0 06 Jun 2024