Scalable agent alignment via reward modeling: a research direction

19 November 2018

Papers citing "Scalable agent alignment via reward modeling: a research direction"

50 / 73 papers shown

Title
Soft Best-of-n Sampling for Model Alignment C. M. Verdun Alex Oesterling Himabindu Lakkaraju Flavio du Pin Calmon BDL 118 0 0 06 May 2025
An alignment safety case sketch based on debate Marie Davidsen Buhl Jacob Pfau Benjamin Hilton Geoffrey Irving 30 0 0 06 May 2025
Scaling Laws For Scalable Oversight Joshua Engels David D. Baek Subhash Kantamneni Max Tegmark ELM 70 0 0 25 Apr 2025
Redefining Superalignment: From Weak-to-Strong Alignment to Human-AI Co-Alignment to Sustainable Symbiotic Society Feifei Zhao Y. Wang Enmeng Lu Dongcheng Zhao Bing Han ... Chao Liu Yaodong Yang Yi Zeng Boyuan Chen Jinyu Fan 83 0 0 24 Apr 2025
Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution Zen Kit Heng Zimeng Zhao Tianhao Wu Yuanfei Wang Mingdong Wu Yangang Wang Hao Dong 33 0 0 10 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 31 0 0 08 Apr 2025
Adversarial Training of Reward Models Alexander Bukharin Haifeng Qian Shengyang Sun Adithya Renduchintala Soumye Singhal Z. Wang Oleksii Kuchaiev Olivier Delalleau T. Zhao AAML 32 0 0 08 Apr 2025
Representation-based Reward Modeling for Efficient Safety Alignment of Large Language Model Qiyuan Deng X. Bai Kehai Chen Yaowei Wang Liqiang Nie Min Zhang OffRL 59 0 0 13 Mar 2025
Societal Alignment Frameworks Can Improve LLM Alignment Karolina Stañczak Nicholas Meade Mehar Bhatia Hattie Zhou Konstantin Böttinger ... Timothy P. Lillicrap Ana Marasović Sylvie Delacroix Gillian K. Hadfield Siva Reddy 110 0 0 27 Feb 2025
Learning from Active Human Involvement through Proxy Value Propagation Zhenghao Peng Wenjie Mo Chenda Duan Quanyi Li Bolei Zhou 102 14 0 05 Feb 2025
COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models Tobias Materzok LRM 69 0 0 28 Jan 2025
MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking Sebastian Farquhar Vikrant Varma David Lindner David Elson Caleb Biddulph Ian Goodfellow Rohin Shah 82 1 0 22 Jan 2025
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 45 3 0 29 Oct 2024
SPIN: Self-Supervised Prompt INjection Leon Zhou Junfeng Yang Chengzhi Mao AAML SILM 28 0 0 17 Oct 2024
Preference Optimization with Multi-Sample Comparisons Chaoqi Wang Zhuokai Zhao Chen Zhu Karthik Abinav Sankararaman Michal Valko ... Zhaorun Chen Madian Khabsa Yuxin Chen Hao Ma Sinong Wang 62 10 0 16 Oct 2024
CREAM: Consistency Regularized Self-Rewarding Language Models Z. Wang Weilei He Zhiyuan Liang Xuchao Zhang Chetan Bansal Ying Wei Weitong Zhang Huaxiu Yao ALM 96 7 0 16 Oct 2024
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment Yuancheng Xu Udari Madhushani Sehwag Alec Koppel Sicheng Zhu Bang An Furong Huang Sumitra Ganesh 55 6 0 10 Oct 2024
T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design Jiachen Li Qian Long Jian Zheng Xiaofeng Gao Robinson Piramuthu Wenhu Chen William Yang Wang VGen 27 22 0 08 Oct 2024
Prompt Baking Aman Bhargava Cameron Witkowski Alexander Detkov Matt W. Thomson AI4CE 28 0 0 04 Sep 2024
Emergence in Multi-Agent Systems: A Safety Perspective Philipp Altmann Julian Schonberger Steffen Illium Maximilian Zorn Fabian Ritz Tom Haider Simon Burton Thomas Gabor 30 1 0 08 Aug 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 44 72 0 28 Jul 2024
Training Foundation Models as Data Compression: On Information, Model Weights and Copyright Law Giorgio Franceschelli Claudia Cevenini Mirco Musolesi 44 0 0 18 Jul 2024
PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences Daiwei Chen Yi Chen Aniket Rege Ramya Korlakai Vinayak 38 17 0 12 Jun 2024
Diffusion-Reward Adversarial Imitation Learning Chun-Mao Lai Hsiang-Chun Wang Ping-Chun Hsieh Yu-Chiang Frank Wang Min-Hung Chen Shao-Hua Sun 32 8 0 25 May 2024
Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation JoonHo Lee Jae Oh Woo Juree Seok Parisa Hassanzadeh Wooseok Jang ... Hankyu Moon Wenjun Hu Yeong-Dae Kwon Taehee Lee Seungjai Min 45 2 0 10 May 2024
LLM Evaluators Recognize and Favor Their Own Generations Arjun Panickssery Samuel R. Bowman Shi Feng 36 156 0 15 Apr 2024
On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models Xinpeng Wang Shitong Duan Xiaoyuan Yi Jing Yao Shanlin Zhou Zhihua Wei Peng Zhang Dongkuan Xu Maosong Sun Xing Xie OffRL 33 16 0 07 Mar 2024
Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning Xizhou Bu Wenjuan Li Zhengxiong Liu Zhiqiang Ma Panfeng Huang 20 1 0 18 Dec 2023
Vision-Language Models as a Source of Rewards Kate Baumli Satinder Baveja Feryal M. P. Behbahani Harris Chan Gheorghe Comanici ... Yannick Schroecker Stephen Spencer Richie Steigerwald Luyu Wang Lei Zhang VLM LRM 32 26 0 14 Dec 2023
Scalable AI Safety via Doubly-Efficient Debate Jonah Brown-Cohen Geoffrey Irving Georgios Piliouras 19 15 0 23 Nov 2023
From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning Hang Yu Reuben M. Aronson Katherine H. Allen E. Short 40 3 0 17 Nov 2023
When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks Hao Peng Xiaozhi Wang Jianhui Chen Weikai Li Y. Qi ... Zhili Wu Kaisheng Zeng Bin Xu Lei Hou Juanzi Li 24 27 0 15 Nov 2023
Fake Alignment: Are LLMs Really Aligned Well? Yixu Wang Yan Teng Kexin Huang Chengqi Lyu Songyang Zhang Wenwei Zhang Xingjun Ma Yu-Gang Jiang Yu Qiao Yingchun Wang 25 14 0 10 Nov 2023
Towards Understanding Sycophancy in Language Models Mrinank Sharma Meg Tong Tomasz Korbak D. Duvenaud Amanda Askell ... Oliver Rausch Nicholas Schiefer Da Yan Miranda Zhang Ethan Perez 209 178 0 20 Oct 2023
SELF: Self-Evolution with Language Feedback Jianqiao Lu Wanjun Zhong Wenyong Huang Yufei Wang Qi Zhu ... Weichao Wang Xingshan Zeng Lifeng Shang Xin Jiang Qun Liu LRM SyDa 16 6 0 01 Oct 2023
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints Chaoqi Wang Yibo Jiang Yuguang Yang Han Liu Yuxin Chen 19 81 0 28 Sep 2023
Designing Fiduciary Artificial Intelligence Sebastian Benthall David Shekman 43 3 0 27 Jul 2023
Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models S. Phelps Rebecca E. Ranson LLMAG 26 1 0 20 Jul 2023
Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback Tom Bewley J. Lawry Arthur G. Richards 25 1 0 26 May 2023
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment Hanze Dong Wei Xiong Deepanshu Goyal Yihan Zhang Winnie Chow Rui Pan Shizhe Diao Jipeng Zhang Kashun Shum Tong Zhang ALM 11 399 0 13 Apr 2023
A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors Shangding Gu Alap Kshirsagar Yali Du Guang Chen Jan Peters Alois C. Knoll 26 14 0 25 Feb 2023
Discovering Latent Knowledge in Language Models Without Supervision Collin Burns Haotian Ye Dan Klein Jacob Steinhardt 45 321 0 07 Dec 2022
Actively Learning Costly Reward Functions for Reinforcement Learning André Eberhard Houssam Metni G. Fahland A. Stroh Pascal Friederich OffRL 19 0 0 23 Nov 2022
Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning Katherine Metcalf Miguel Sarabia B. Theobald OffRL 25 4 0 12 Nov 2022
Measuring Progress on Scalable Oversight for Large Language Models Sam Bowman Jeeyoon Hyun Ethan Perez Edwin Chen Craig Pettit ... Tristan Hume Yuntao Bai Zac Hatfield-Dodds Benjamin Mann Jared Kaplan ALM ELM 21 121 0 04 Nov 2022
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 225 500 0 28 Sep 2022
Argumentative Reward Learning: Reasoning About Human Preferences Francis Rhys Ward Francesco Belardinelli Francesca Toni HAI 82 2 0 28 Sep 2022
Defining and Characterizing Reward Hacking Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov David M. Krueger 57 54 0 27 Sep 2022
Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans John J. Nay ELM AILaw 84 27 0 14 Sep 2022
The Alignment Problem from a Deep Learning Perspective Richard Ngo Lawrence Chan Sören Mindermann 52 181 0 30 Aug 2022