RRHF: Rank Responses to Align Language Models with Human Feedback without tears

11 April 2023

Zheng Yuan

Hongyi Yuan

Chuanqi Tan

Papers citing "RRHF: Rank Responses to Align Language Models with Human Feedback without tears"

50 / 282 papers shown

Title
PERL: Parameter Efficient Reinforcement Learning from Human Feedback Hakim Sidahmed Samrat Phatale Alex Hutcheson Zhuonan Lin Zhan Chen ... Jessica Hoffmann Hassan Mansoor Wei Li Abhinav Rastogi Lucas Dixon 24 4 0 15 Mar 2024
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization Renjie Pi Tianyang Han Wei Xiong Jipeng Zhang Runtao Liu Rui Pan Tong Zhang MLLM 35 33 0 13 Mar 2024
Human Alignment of Large Language Models through Online Preference Optimisation Daniele Calandriello Daniel Guo Rémi Munos Mark Rowland Yunhao Tang ... Michal Valko Tianqi Liu Rishabh Joshi Zeyu Zheng Bilal Piot 44 60 0 13 Mar 2024
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards Wei Shen Xiaoying Zhang Yuanshun Yao Rui Zheng Hongyi Guo Yang Liu ALM 38 11 0 12 Mar 2024
Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences Pulkit Pattnaik Rishabh Maheshwary Kelechi Ogueji Vikas Yadav Sathwik Tejaswi Madhusudhan 31 18 0 12 Mar 2024
From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification Fei Wang Chao Shang Sarthak Jain Shuai Wang Qiang Ning Bonan Min Vittorio Castelli Yassine Benajiba Dan Roth ALM 22 7 0 10 Mar 2024
Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation Xiaoying Zhang Jean-François Ton Wei Shen Hongning Wang Yang Liu 32 13 0 08 Mar 2024
On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models Xinpeng Wang Shitong Duan Xiaoyuan Yi Jing Yao Shanlin Zhou Zhihua Wei Peng Zhang Dongkuan Xu Maosong Sun Xing Xie OffRL 33 16 0 07 Mar 2024
Negating Negatives: Alignment without Human Positive Samples via Distributional Dispreference Optimization Shitong Duan Xiaoyuan Yi Peng Zhang T. Lu Xing Xie Ning Gu 32 4 0 06 Mar 2024
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling Shanghaoran Quan MoE OffRL 43 7 0 02 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 22 55 0 01 Mar 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 40 70 0 28 Feb 2024
TroubleLLM: Align to Red Team Expert Zhuoer Xu Jianping Zhang Shiwen Cui Changhua Meng Weiqiang Wang 33 1 0 28 Feb 2024
Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration Xin Mao Fengming Li Huimin Xu Wei Zhang A. Luu ALM 45 6 0 25 Feb 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Tianyi Zhou KELM VLM 42 100 0 20 Feb 2024
A Critical Evaluation of AI Feedback for Aligning Large Language Models Archit Sharma Sedrick Scott Keh Eric Mitchell Chelsea Finn Kushal Arora Thomas Kollar ALM LLMAG 21 23 0 19 Feb 2024
Amplifying Training Data Exposure through Fine-Tuning with Pseudo-Labeled Memberships Myung Gyo Oh Hong Eun Ahn L. Park T.-H. Kwon MIALM AAML 27 0 0 19 Feb 2024
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment Rui Yang Xiaoman Pan Feng Luo Shuang Qiu Han Zhong Dong Yu Jianshu Chen 95 66 0 15 Feb 2024
Reward Generalization in RLHF: A Topological Perspective Tianyi Qiu Fanzhi Zeng Jiaming Ji Dong Yan Kaile Wang Jiayi Zhou Yang Han Josef Dai Xuehai Pan Yaodong Yang AI4CE 25 3 0 15 Feb 2024
Aligning Crowd Feedback via Distributional Preference Reward Modeling Dexun Li Cong Zhang Kuicai Dong Derrick-Goh-Xin Deik Ruiming Tang Yong Liu 8 14 0 15 Feb 2024
ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization Feifan Song Yuxuan Fan Xin Zhang Peiyi Wang Houfeng Wang 30 8 0 14 Feb 2024
Rethinking Machine Unlearning for Large Language Models Sijia Liu Yuanshun Yao Jinghan Jia Stephen Casper Nathalie Baracaldo ... Hang Li Kush R. Varshney Mohit Bansal Sanmi Koyejo Yang Liu AILaw MU 65 81 0 13 Feb 2024
A Dense Reward View on Aligning Text-to-Image Diffusion with Preference Shentao Yang Tianqi Chen Mingyuan Zhou EGVM 30 22 0 13 Feb 2024
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts Yueqin Yin Zhendong Wang Yi Gu Hai Huang Weizhu Chen Mingyuan Zhou 19 19 0 12 Feb 2024
Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs Víctor Gallego SyDa 25 6 0 12 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 38 9 0 11 Feb 2024
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation Xianghe Pang Shuo Tang Rui Ye Yuxin Xiong Bolun Zhang Yanfeng Wang Siheng Chen 114 28 0 08 Feb 2024
Noise Contrastive Alignment of Language Models with Explicit Rewards Huayu Chen Guande He Lifan Yuan Ganqu Cui Hang Su Jun Zhu 55 37 0 08 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Y. K. Li Yu-Huan Wu Daya Guo ReLM LRM 26 631 0 05 Feb 2024
DeAL: Decoding-time Alignment for Large Language Models James Y. Huang Sailik Sengupta Daniele Bonadiman Yi-An Lai Arshit Gupta Nikolaos Pappas Saab Mansour Katrin Kirchoff Dan Roth 47 29 0 05 Feb 2024
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback Gaurav Pandey Yatin Nandwani Tahira Naseem Mayank Mishra Guangxuan Xu Dinesh Raghu Sachindra Joshi Asim Munawar Ramón Fernández Astudillo BDL 37 3 0 04 Feb 2024
Aligner: Efficient Alignment by Learning to Correct Jiaming Ji Boyuan Chen Hantao Lou Donghai Hong Borong Zhang Xuehai Pan Juntao Dai Tianyi Qiu Yaodong Yang 29 28 0 04 Feb 2024
Panacea: Pareto Alignment via Preference Adaptation for LLMs Yifan Zhong Chengdong Ma Xiaoyuan Zhang Ziran Yang Haojun Chen Qingfu Zhang Siyuan Qi Yaodong Yang 54 30 0 03 Feb 2024
Vaccine: Perturbation-aware Alignment for Large Language Model Tiansheng Huang Sihao Hu Ling Liu 44 33 0 02 Feb 2024
Towards Efficient Exact Optimization of Language Model Alignment Haozhe Ji Cheng Lu Yilin Niu Pei Ke Hongning Wang Jun Zhu Jie Tang Minlie Huang 50 11 0 01 Feb 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 6 30 0 01 Feb 2024
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF Banghua Zhu Michael I. Jordan Jiantao Jiao 26 23 0 29 Jan 2024
ARGS: Alignment as Reward-Guided Search Maxim Khanov Jirayu Burapacheep Yixuan Li 29 44 0 23 Jan 2024
Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model Zhiwei He Xing Wang Wenxiang Jiao Zhuosheng Zhang Rui Wang Shuming Shi Zhaopeng Tu ALM 29 24 0 23 Jan 2024
GRATH: Gradual Self-Truthifying for Large Language Models Weixin Chen D. Song Bo-wen Li HILM SyDa 23 5 0 22 Jan 2024
Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback Songyang Gao Qiming Ge Wei Shen Shihan Dou Junjie Ye ... Yicheng Zou Zhi Chen Hang Yan Qi Zhang Dahua Lin 47 10 0 21 Jan 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 235 298 0 18 Jan 2024
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems Tianyu Cui Yanling Wang Chuanpu Fu Yong Xiao Sijia Li ... Junwu Xiong Xinyu Kong Zujie Wen Ke Xu Qi Li 55 56 0 11 Jan 2024
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback Chengfeng Dou Zhi Jin Wenpin Jiao Haiyan Zhao Yongqiang Zhao Zhenwei Tao LM&MA 74 4 0 11 Jan 2024
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance Renjie Pi Tianyang Han Jianshu Zhang Yueqi Xie Rui Pan Qing Lian Hanze Dong Jipeng Zhang Tong Zhang AAML 23 59 0 05 Jan 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 66 32 0 30 Dec 2023
Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs Shaojie Zhu Zhaobin Wang Chengxiang Zhuo Hui Lu Bo Hu Zang Li LRM 27 0 0 29 Dec 2023
Improving In-context Learning via Bidirectional Alignment Chengwei Qin Wenhan Xia Fangkai Jiao Chen Chen Yuchen Hu Bosheng Ding Shafiq R. Joty 35 7 0 28 Dec 2023
Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss Jing Xu Andrew Lee Sainbayar Sukhbaatar Jason Weston 15 86 0 27 Dec 2023
LLMLight: Large Language Models as Traffic Signal Control Agents Siqi Lai Zhao Xu Weijiao Zhang Hao Liu Hui Xiong LLMAG 18 7 0 26 Dec 2023