The Trickle-down Impact of Reward (In-)consistency on RLHF

28 September 2023

Lingfeng Shen

Linfeng Song

Daniel Khashabi

Dong Yu

ArXiv PDF HTML

Papers citing "The Trickle-down Impact of Reward (In-)consistency on RLHF"

23 / 23 papers shown

Title
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 31 0 0 08 Apr 2025
Take Off the Training Wheels Progressive In-Context Learning for Effective Alignment Zhenyu Liu Dongfang Li Xinshuo Hu X. Zhao Yibin Chen Baotian Hu Min-Ling Zhang 46 1 0 13 Mar 2025
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 37 13 0 05 Nov 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF Taiming Lu Lingfeng Shen Xinyu Yang Weiting Tan Beidi Chen Huaxiu Yao 43 2 0 12 Jun 2024
Improving Reward Models with Synthetic Critiques Zihuiwen Ye Fraser Greenlee-Scott Max Bartolo Phil Blunsom Jon Ander Campos Matthias Gallé ALM SyDa LRM 33 16 0 31 May 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James Validad Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 71 210 0 20 Mar 2024
Tur[k]ingBench: A Challenge Benchmark for Web Agents Kevin Xu Yeganeh Kordi Kate Sanders Yizhong Wang Adam Byerly Kate Sanders Adam Byerly Jingyu Zhang Benjamin Van Durme Daniel Khashabi LLMAG 64 6 0 18 Mar 2024
Rethinking the Role of Proxy Rewards in Language Model Alignment Sungdong Kim Minjoon Seo SyDa ALM 23 0 0 02 Feb 2024
Transforming and Combining Rewards for Aligning Large Language Models Zihao Wang Chirag Nagpal Jonathan Berant Jacob Eisenstein Alex DÁmour Oluwasanmi Koyejo Victor Veitch 6 11 0 01 Feb 2024
ARGS: Alignment as Reward-Guided Search Maxim Khanov Jirayu Burapacheep Yixuan Li 26 43 0 23 Jan 2024
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts Lingfeng Shen Weiting Tan Sihao Chen Yunmo Chen Jingyu Zhang Haoran Xu Boyuan Zheng Philipp Koehn Daniel Khashabi 19 37 0 23 Jan 2024
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning Bill Yuchen Lin Abhilasha Ravichander Ximing Lu Nouha Dziri Melanie Sclar Khyathi Raghavi Chandu Chandra Bhagavatula Yejin Choi 17 163 0 04 Dec 2023
The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback Nathan Lambert Roberto Calandra ALM 13 30 0 31 Oct 2023
Evaluating Large Language Models at Evaluating Instruction Following Zhiyuan Zeng Jiatong Yu Tianyu Gao Yu Meng Tanya Goyal Danqi Chen ELM ALM 15 159 0 11 Oct 2023
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 225 495 0 28 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Measure and Improve Robustness in NLP Models: A Survey Xuezhi Wang Haohan Wang Diyi Yang 129 130 0 15 Dec 2021
Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer Fanchao Qi Yangyi Chen Xurui Zhang Mukai Li Zhiyuan Liu Maosong Sun AAML SILM 77 171 0 14 Oct 2021
Robustness Gym: Unifying the NLP Evaluation Landscape Karan Goel Nazneen Rajani Jesse Vig Samson Tan Jason M. Wu Stephan Zheng Caiming Xiong Mohit Bansal Christopher Ré AAML OffRL OOD 138 136 0 13 Jan 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019
Certified Robustness to Adversarial Word Substitutions Robin Jia Aditi Raghunathan Kerem Göksel Percy Liang AAML 167 289 0 03 Sep 2019
Generating Natural Language Adversarial Examples M. Alzantot Yash Sharma Ahmed Elgohary Bo-Jhang Ho Mani B. Srivastava Kai-Wei Chang AAML 233 909 0 21 Apr 2018