SLiC-HF: Sequence Likelihood Calibration with Human Feedback

17 May 2023

Yao-Min Zhao

Rishabh Joshi

Tianqi Liu

Misha Khalman

Mohammad Saleh

Peter J. Liu

ArXiv PDF HTML

Papers citing "SLiC-HF: Sequence Likelihood Calibration with Human Feedback"

50 / 220 papers shown

Title
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 46 135 0 18 Jun 2024
Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency Leonidas Gee Milan Gritta Gerasimos Lampouras Ignacio Iacobacci 21 10 0 18 Jun 2024
WPO: Enhancing RLHF with Weighted Preference Optimization Wenxuan Zhou Ravi Agrawal Shujian Zhang Sathish Indurthi Sanqiang Zhao Kaiqiang Song Silei Xu Chenguang Zhu 35 17 0 17 Jun 2024
A Survey on Human Preference Learning for Large Language Models Ruili Jiang Kehai Chen Xuefeng Bai Zhixuan He Juntao Li Muyun Yang Tiejun Zhao Liqiang Nie Min Zhang 39 8 0 17 Jun 2024
Self-Evolution Fine-Tuning for Policy Optimization Ruijun Chen Jiehao Liang Shiping Gao Fanqi Wan Xiaojun Quan 35 0 0 16 Jun 2024
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning Jifan Zhang Lalit P. Jain Yang Guo Jiayi Chen Kuan Lok Zhou ... Scott Sievert Timothy Rogers Kevin Jamieson Robert Mankoff Robert Nowak 29 5 0 15 Jun 2024
Bootstrapping Language Models with DPO Implicit Rewards Changyu Chen Zichen Liu Chao Du Tianyu Pang Qian Liu Arunesh Sinha Pradeep Varakantham Min-Bin Lin SyDa ALM 62 23 0 14 Jun 2024
ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions Xu Zhang Xunjian Yin Xiaojun Wan 40 3 0 13 Jun 2024
Discovering Preference Optimization Algorithms with and for Large Language Models Chris Xiaoxuan Lu Samuel Holt Claudio Fanconi Alex J. Chan Jakob Foerster M. Schaar R. T. Lange OffRL 32 14 0 12 Jun 2024
OPTune: Efficient Online Preference Tuning Lichang Chen Jiuhai Chen Chenxi Liu John Kirchenbauer Davit Soselia Chen Zhu Tom Goldstein Tianyi Zhou Heng Huang 34 4 0 11 Jun 2024
3D-Properties: Identifying Challenges in DPO and Charting a Path Forward Yuzi Yan Yibo Miao J. Li Yipin Zhang Jian Xie Zhijie Deng Dong Yan 44 11 0 11 Jun 2024
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms Rafael Rafailov Yaswanth Chittepu Ryan Park Harshit S. Sikchi Joey Hejna Bradley Knox Chelsea Finn S. Niekum 50 47 0 05 Jun 2024
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs Rongzhi Zhang Jiaming Shen Tianqi Liu Haorui Wang Zhen Qin Feng Han Jialu Liu Simon Baumgartner Michael Bendersky Chao Zhang 37 6 0 05 Jun 2024
Adaptive Preference Scaling for Reinforcement Learning with Human Feedback Ilgee Hong Zichong Li Alexander Bukharin Yixiao Li Haoming Jiang Tianbao Yang Tuo Zhao 24 4 0 04 Jun 2024
Self-Improving Robust Preference Optimization Eugene Choi Arash Ahmadian Matthieu Geist Oilvier Pietquin M. G. Azar 26 8 0 03 Jun 2024
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling Lin Gui Cristina Garbacea Victor Veitch BDL LM&MA 41 36 0 02 Jun 2024
Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training Maximillian Chen Ruoxi Sun Sercan Ö. Arik Tomas Pfister LLMAG 27 6 0 31 May 2024
Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment Yueqin Yin Zhendong Wang Yujia Xie Weizhu Chen Mingyuan Zhou 33 4 0 31 May 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 19 22 0 30 May 2024
Enhancing Large Vision Language Models with Self-Training on Image Comprehension Yihe Deng Pan Lu Fan Yin Ziniu Hu Sheng Shen James Y. Zou Kai-Wei Chang Wei Wang SyDa VLM LRM 36 36 0 30 May 2024
Preference Learning Algorithms Do Not Learn Preference Rankings Angelica Chen Sadhika Malladi Lily H. Zhang Xinyi Chen Qiuyi Zhang Rajesh Ranganath Kyunghyun Cho 25 22 0 29 May 2024
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment Shenao Zhang Donghan Yu Hiteshi Sharma Ziyi Yang Shuohang Wang Hany Hassan Zhaoran Wang LRM 40 28 0 29 May 2024
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models Zhanhui Zhou Zhixuan Liu Jie Liu Zhichen Dong Chao Yang Yu Qiao ALM 36 20 0 29 May 2024
Offline Regularised Reinforcement Learning for Large Language Models Alignment Pierre Harvey Richemond Yunhao Tang Daniel Guo Daniele Calandriello M. G. Azar ... Gil Shamir Rishabh Joshi Tianqi Liu Rémi Munos Bilal Piot OffRL 40 21 0 29 May 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 73 21 0 29 May 2024
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization Yuanpu Cao Tianrong Zhang Bochuan Cao Ziyi Yin Lu Lin Fenglong Ma Jinghui Chen LLMSV 19 19 0 28 May 2024
Prompt Optimization with Human Feedback Xiaoqiang Lin Zhongxiang Dai Arun Verma See-Kiong Ng P. Jaillet K. H. Low AAML 29 8 0 27 May 2024
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer Zhihan Liu Miao Lu Shenao Zhang Boyi Liu Hongyi Guo Yingxiang Yang Jose H. Blanchet Zhaoran Wang 38 41 0 26 May 2024
Multi-Reference Preference Optimization for Large Language Models Hung Le Quan Tran D. Nguyen Kien Do Saloni Mittal Kelechi Ogueji Svetha Venkatesh 53 0 0 26 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 57 345 0 23 May 2024
Online Self-Preferring Language Models Yuanzhao Zhai Zhuo Zhang Kele Xu Hanyang Peng Yue Yu Dawei Feng Cheng Yang Bo Ding Huaimin Wang 41 0 0 23 May 2024
Annotation-Efficient Preference Optimization for Language Model Alignment Yuu Jinnai Ukyo Honda 40 0 0 22 May 2024
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process Ermo Hua Biqing Qi Kaiyan Zhang Yue Yu Ning Ding Xingtai Lv Kai Tian Bowen Zhou 32 3 0 20 May 2024
Hummer: Towards Limited Competitive Preference Dataset Li Jiang Yusen Wu Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo Zujie Wen Jun Zhou Xiaotie Deng 29 6 0 19 May 2024
Understanding the performance gap between online and offline alignment algorithms Yunhao Tang Daniel Guo Zeyu Zheng Daniele Calandriello Yuan Cao ... Rémi Munos Bernardo Avila-Pires Michal Valko Yong Cheng Will Dabney OffRL OnRL 25 61 0 14 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 21 95 0 13 May 2024
Integrating Emotional and Linguistic Models for Ethical Compliance in Large Language Models Edward Y. Chang 22 3 0 11 May 2024
Self-Play Preference Optimization for Language Model Alignment Yue Wu Zhiqing Sun Huizhuo Yuan Kaixuan Ji Yiming Yang Quanquan Gu 28 113 0 01 May 2024
Soft Preference Optimization: Aligning Language Models to Expert Distributions Arsalan Sharifnassab Sina Ghiassian Saber Salehkaleybar Surya Kanoria Dale Schuurmans 28 2 0 30 Apr 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Guhao Feng Guhao Feng Li Zhao Di He Jiang Bian Liwei Wang Jiang Bian Liwei Wang 55 56 0 29 Apr 2024
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data Fahim Tajwar Anika Singh Archit Sharma Rafael Rafailov Jeff Schneider Tengyang Xie Stefano Ermon Chelsea Finn Aviral Kumar 33 105 0 22 Apr 2024
Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels Jan-Philipp Fränken E. Zelikman Rafael Rafailov Kanishk Gandhi Tobias Gerstenberg Noah D. Goodman 23 9 0 22 Apr 2024
Impact of Preference Noise on the Alignment Performance of Generative Language Models Yang Gao Dana Alon Donald Metzler 21 16 0 15 Apr 2024
Exploring Text-to-Motion Generation with Human Preference Jenny Sheng Matthieu Lin Andrew Zhao Kevin Pruvost Yu-Hui Wen Yangguang Li Gao Huang Yong-Jin Liu VGen 32 1 0 15 Apr 2024
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari Pranjal Aggarwal Vishvak Murahari Tanmay Rajpurohit A. Kalyan Karthik Narasimhan A. Deshpande Bruno Castro da Silva 21 34 0 12 Apr 2024
Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data Tim Baumgärtner Yang Gao Dana Alon Donald Metzler AAML 18 18 0 08 Apr 2024
Aligning Diffusion Models by Optimizing Human Utility Shufan Li Konstantinos Kallidromitis Akash Gokul Yusuke Kato Kazuki Kozuka 105 27 0 06 Apr 2024
ROPO: Robust Preference Optimization for Large Language Models Xize Liang Chao Chen Shuang Qiu Jie Wang Yue-bo Wu Zhihang Fu Zhihao Shi Feng Wu Jieping Ye 43 1 0 05 Apr 2024
Investigating Regularization of Self-Play Language Models Réda Alami Abdalgader Abubaker Mastane Achab M. Seddik Salem Lahlou 30 3 0 04 Apr 2024
Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment Yuu Jinnai Tetsuro Morimura Kaito Ariu Kenshi Abe 57 7 0 01 Apr 2024