Disentangling Length from Quality in Direct Preference Optimization

28 March 2024

Stefano Ermon

Papers citing "Disentangling Length from Quality in Direct Preference Optimization"

24 / 24 papers shown

Title
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul M. Chilimbi 19 0 0 13 May 2025
LookAlike: Consistent Distractor Generation in Math MCQs Nisarg Parikh Nigel Fernandez Alexander Scarlatos Simon Woodhead Andrew S. Lan 41 0 0 03 May 2025
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors Ren-Wei Liang Chin-Ting Hsu Chan-Hung Yu Saransh Agrawal Shih-Cheng Huang Shang-Tse Chen Kuan-Hao Huang Shao-Hua Sun 76 0 0 27 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong X. Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Yu Jiang ALM ELM 84 0 0 26 Apr 2025
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal 71 0 0 10 Mar 2025
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition H. A. Alyahya Haidar Khan Yazeed Alnumay M Saiful Bari B. Yener LRM 60 1 0 10 Mar 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 45 0 0 08 Feb 2025
REFA: Reference Free Alignment for multi-preference optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 83 1 0 20 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 98 2 0 01 Dec 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 45 3 0 29 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 87 2 0 10 Oct 2024
RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization Hanyang Zhao Genta Indra Winata Anirban Das Shi-Xiong Zhang D. Yao Wenpin Tang Sambit Sahu 54 4 0 05 Oct 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 10 0 18 Sep 2024
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization Yuxin Jiang Bo Huang Yufei Wang Xingshan Zeng Liangyou Li Yasheng Wang Xin Jiang Lifeng Shang Ruiming Tang Wei Wang 42 5 0 14 Aug 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 44 71 0 28 Jul 2024
Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization Wenkai Yang Shiqi Shen Guangyao Shen Zhi Gong Yankai Lin Zhi Gong Yankai Lin Ji-Rong Wen 50 13 0 17 Jun 2024
Bootstrapping Language Models with DPO Implicit Rewards Changyu Chen Zichen Liu Chao Du Tianyu Pang Qian Liu Arunesh Sinha Pradeep Varakantham Min-Bin Lin SyDa ALM 62 23 0 14 Jun 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 73 21 0 29 May 2024
Learn Your Reference Model for Real Good Alignment Alexey Gorbatovski Boris Shaposhnikov Alexey Malakhov Nikita Surnachev Yaroslav Aksenov Ian Maksimov Nikita Balagansky Daniil Gavrilov OffRL 47 25 0 15 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 53 315 0 06 Apr 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 61 31 0 30 Dec 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019