KTO: Model Alignment as Prospect Theoretic Optimization

KTO: Model Alignment as Prospect Theoretic Optimization

2 February 2024

Kawin Ethayarajh

Niklas Muennighoff

Dan Jurafsky

Douwe Kiela

Papers citing "KTO: Model Alignment as Prospect Theoretic Optimization"

17 / 17 papers shown

Title
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 7 0 0 05 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 11 0 0 05 May 2025
Calibrating Translation Decoding with Quality Estimation on LLMs Di Wu Yibin Lei Christof Monz 41 46 0 26 Apr 2025
Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization Kesen Zhao B. Zhu Qianru Sun Hanwang Zhang MLLM LRM 51 0 0 25 Apr 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 30 0 0 22 Apr 2025
SR-Reward: Taking The Path More Traveled Seyed Mahdi Basiri Azad Zahra Padar Gabriel Kalweit Joschka Boedecker OffRL 36 0 0 04 Jan 2025
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 44 10 0 11 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 42 2 0 10 Oct 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 42 5 0 13 Sep 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 73 124 0 20 May 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 110 104 0 04 Apr 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 171 133 0 18 Jan 2024
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 185 327 0 23 Aug 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 270 8,441 0 04 Mar 2022
$Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information$ Understanding Dataset Difficulty with $\mathcal{V}$ -Usable Information Kawin Ethayarajh Yejin Choi Swabha Swayamdipta 125 157 0 16 Oct 2021
Towards Ecologically Valid Research on Language User Interfaces H. D. Vries Dzmitry Bahdanau Christopher D. Manning 167 47 0 28 Jul 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 245 1,151 0 18 Sep 2019