SLiC-HF: Sequence Likelihood Calibration with Human Feedback

17 May 2023

Yao-Min Zhao

Rishabh Joshi

Tianqi Liu

Misha Khalman

Mohammad Saleh

Peter J. Liu

ArXiv PDF HTML

Papers citing "SLiC-HF: Sequence Likelihood Calibration with Human Feedback"

50 / 220 papers shown

Title
ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback Zhenyu Hou Yiin Niu Zhengxiao Du Xiaohan Zhang Xiao Liu ... Qinkai Zheng Minlie Huang Hongning Wang Jie Tang Yuxiao Dong ALM 22 17 0 01 Apr 2024
Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model Qi Gou Cam-Tu Nguyen 27 8 0 28 Mar 2024
Improving the Robustness of Large Language Models via Consistency Alignment Zhao Yukun Lingyong Yan Weiwei Sun Guoliang Xing Shuaiqiang Wang Meng Chong Zhicong Cheng Zhaochun Ren Yin Dawei 35 18 0 21 Mar 2024
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision Zhiqing Sun Longhui Yu Yikang Shen Weiyang Liu Yiming Yang Sean Welleck Chuang Gan 23 50 0 14 Mar 2024
Human Alignment of Large Language Models through Online Preference Optimisation Daniele Calandriello Daniel Guo Rémi Munos Mark Rowland Yunhao Tang ... Michal Valko Tianqi Liu Rishabh Joshi Zeyu Zheng Bilal Piot 44 60 0 13 Mar 2024
Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences Pulkit Pattnaik Rishabh Maheshwary Kelechi Ogueji Vikas Yadav Sathwik Tejaswi Madhusudhan 31 18 0 12 Mar 2024
On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models Xinpeng Wang Shitong Duan Xiaoyuan Yi Jing Yao Shanlin Zhou Zhihua Wei Peng Zhang Dongkuan Xu Maosong Sun Xing Xie OffRL 33 16 0 07 Mar 2024
Negating Negatives: Alignment without Human Positive Samples via Distributional Dispreference Optimization Shitong Duan Xiaoyuan Yi Peng Zhang T. Lu Xing Xie Ning Gu 32 4 0 06 Mar 2024
CURATRON: Complete Robust Preference Data for Robust Alignment of Large Language Models S. Nguyen Uma-Naresh Niranjan Theja Tulabandhula 31 0 0 05 Mar 2024
Enhancing LLM Safety via Constrained Direct Preference Optimization Zixuan Liu Xiaolin Sun Zizhan Zheng 28 20 0 04 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 22 55 0 01 Mar 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 40 70 0 28 Feb 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 56 17 0 28 Feb 2024
Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration Xin Mao Fengming Li Huimin Xu Wei Zhang A. Luu ALM 45 6 0 25 Feb 2024
SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization Prakamya Mishra Zonghai Yao Parth Vashisht Feiyun Ouyang Beining Wang Vidhi Mody Hong-ye Yu SyDa MedIm 36 4 0 21 Feb 2024
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive Arka Pal Deep Karkhanis Samuel Dooley Manley Roberts Siddartha Naidu Colin White OSLM 31 125 0 20 Feb 2024
Amplifying Training Data Exposure through Fine-Tuning with Pseudo-Labeled Memberships Myung Gyo Oh Hong Eun Ahn L. Park T.-H. Kwon MIALM AAML 27 0 0 19 Feb 2024
Aligning Large Language Models by On-Policy Self-Judgment Sangkyu Lee Sungdong Kim Ashkan Yousefpour Minjoon Seo Kang Min Yoo Youngjae Yu OSLM 33 9 0 17 Feb 2024
Direct Preference Optimization with an Offset Afra Amini Tim Vieira Ryan Cotterell 71 55 0 16 Feb 2024
A Dense Reward View on Aligning Text-to-Image Diffusion with Preference Shentao Yang Tianqi Chen Mingyuan Zhou EGVM 30 22 0 13 Feb 2024
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts Yueqin Yin Zhendong Wang Yi Gu Hai Huang Weizhu Chen Mingyuan Zhou 19 19 0 12 Feb 2024
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model Chen Ye Wei Xiong Yuheng Zhang Nan Jiang Tong Zhang OffRL 38 9 0 11 Feb 2024
Generalized Preference Optimization: A Unified Approach to Offline Alignment Yunhao Tang Z. Guo Zeyu Zheng Daniele Calandriello Rémi Munos Mark Rowland Pierre Harvey Richemond Michal Valko Bernardo Avila-Pires Bilal Piot 32 85 0 08 Feb 2024
Noise Contrastive Alignment of Language Models with Explicit Rewards Huayu Chen Guande He Lifan Yuan Ganqu Cui Hang Su Jun Zhu 55 37 0 08 Feb 2024
Direct Language Model Alignment from Online AI Feedback Shangmin Guo Biao Zhang Tianlin Liu Tianqi Liu Misha Khalman ... Thomas Mesnard Yao-Min Zhao Bilal Piot Johan Ferret Mathieu Blondel ALM 29 130 0 07 Feb 2024
Decoding-time Realignment of Language Models Tianlin Liu Shangmin Guo Leonardo Bianco Daniele Calandriello Quentin Berthet Felipe Llinares-López Jessica Hoffmann Lucas Dixon Michal Valko Mathieu Blondel AI4CE 54 35 0 05 Feb 2024
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback Gaurav Pandey Yatin Nandwani Tahira Naseem Mayank Mishra Guangxuan Xu Dinesh Raghu Sachindra Joshi Asim Munawar Ramón Fernández Astudillo BDL 37 3 0 04 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 159 444 0 02 Feb 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 6 30 0 01 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 235 298 0 18 Jan 2024
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty Kaitlyn Zhou Jena D. Hwang Xiang Ren Maarten Sap 28 54 0 12 Jan 2024
A Minimaximalist Approach to Reinforcement Learning from Human Feedback Gokul Swamy Christoph Dann Rahul Kidambi Zhiwei Steven Wu Alekh Agarwal OffRL 28 94 0 08 Jan 2024
Large Language Models for Social Networks: Applications, Challenges, and Solutions Jingying Zeng Richard Huang Waleed Malik Langxuan Yin Bojan Babic Danny Shacham Xiao Yan Jaewon Yang Qi He 22 6 0 04 Jan 2024
Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss Jing Xu Andrew Lee Sainbayar Sukhbaatar Jason Weston 15 86 0 27 Dec 2023
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint Wei Xiong Hanze Dong Chen Ye Ziqi Wang Han Zhong Heng Ji Nan Jiang Tong Zhang OffRL 36 155 0 18 Dec 2023
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling Jingying Zeng Jaewon Yang Waleed Malik Xiao Yan Richard Huang Qi He 22 1 0 16 Dec 2023
Self-Evaluation Improves Selective Generation in Large Language Models Jie Jessie Ren Yao-Min Zhao Tu Vu Peter J. Liu Balaji Lakshminarayanan ELM 23 34 0 14 Dec 2023
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking Jacob Eisenstein Chirag Nagpal Alekh Agarwal Ahmad Beirami Alex DÁmour ... Katherine Heller Stephen R. Pfohl Deepak Ramachandran Peter Shaw Jonathan Berant 24 82 0 14 Dec 2023
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 30 16 0 12 Dec 2023
RLHF and IIA: Perverse Incentives Wanqiao Xu Shi Dong Xiuyuan Lu Grace Lam Zheng Wen Benjamin Van Roy 24 2 0 02 Dec 2023
Nash Learning from Human Feedback Rémi Munos Michal Valko Daniele Calandriello M. G. Azar Mark Rowland ... Nikola Momchev Olivier Bachem D. Mankowitz Doina Precup Bilal Piot 17 123 0 01 Dec 2023
A density estimation perspective on learning from pairwise human preferences Vincent Dumoulin Daniel D. Johnson Pablo Samuel Castro Hugo Larochelle Yann Dauphin 29 12 0 23 Nov 2023
Diffusion Model Alignment Using Direct Preference Optimization Bram Wallace Meihua Dang Rafael Rafailov Linqi Zhou Aaron Lou Senthil Purushwalkam Stefano Ermon Caiming Xiong Shafiq R. Joty Nikhil Naik EGVM 33 224 0 21 Nov 2023
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game Pengyu Cheng Yifan Yang Jian Li Yong Dai Tianhao Hu Peixin Cao Nan Du Xiaolong Li 21 28 0 14 Nov 2023
Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering Yichi Zhang Zhuo Chen Yin Fang Yanxi Lu Fangming Li Wen Zhang Hua-zeng Chen 53 30 0 11 Nov 2023
Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment Geyang Guo Ranchi Zhao Tianyi Tang Wayne Xin Zhao Ji-Rong Wen ALM 27 27 0 07 Nov 2023
Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization Prakamya Mishra Zonghai Yao Shuwei Chen Beining Wang Rohan Mittal Hong-ye Yu KELM ALM HILM 20 7 0 30 Oct 2023
A General Theoretical Paradigm to Understand Learning from Human Preferences M. G. Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos 16 524 0 18 Oct 2023
Calibrating Likelihoods towards Consistency in Summarization Models Polina Zablotskaia Misha Khalman Rishabh Joshi Livio Baldini Soares Shoshana Jakobovits Joshua Maynez Shashi Narayan 26 3 0 12 Oct 2023
Evaluating Large Language Models at Evaluating Instruction Following Zhiyuan Zeng Jiatong Yu Tianyu Gao Yu Meng Tanya Goyal Danqi Chen ELM ALM 33 166 0 11 Oct 2023