KTO: Model Alignment as Prospect Theoretic Optimization

KTO: Model Alignment as Prospect Theoretic Optimization

2 February 2024

Kawin Ethayarajh

Niklas Muennighoff

Dan Jurafsky

Douwe Kiela

Papers citing "KTO: Model Alignment as Prospect Theoretic Optimization"

18 / 18 papers shown

Title
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 17 88 0 05 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 37 65 0 05 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 39 219 0 05 May 2025
Calibrating Translation Decoding with Quality Estimation on LLMs Di Wu Yibin Lei Christof Monz 55 46 0 26 Apr 2025
Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization Kesen Zhao B. Zhu Qianru Sun Hanwang Zhang MLLM LRM 67 40 0 25 Apr 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 47 30 0 22 Apr 2025
SR-Reward: Taking The Path More Traveled Seyed Mahdi Basiri Azad Zahra Padar Gabriel Kalweit Joschka Boedecker OffRL 46 55 0 04 Jan 2025
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 57 69 0 11 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 54 75 0 10 Oct 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 59 53 0 13 Sep 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 87 124 0 20 May 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 114 43 0 04 Apr 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 185 133 0 18 Jan 2024
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 200 327 0 23 Aug 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 298 7,763 0 04 Mar 2022
$Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information$ Understanding Dataset Difficulty with $\mathcal{V}$ -Usable Information Kawin Ethayarajh Yejin Choi Swabha Swayamdipta 129 144 0 16 Oct 2021
Towards Ecologically Valid Research on Language User Interfaces H. D. Vries Dzmitry Bahdanau Christopher D. Manning 174 47 0 28 Jul 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 262 1,123 0 18 Sep 2019