Bootstrapping Language Models with DPO Implicit Rewards

Bootstrapping Language Models with DPO Implicit Rewards

14 June 2024

Zichen Liu

Qian Liu

Arunesh Sinha

Pradeep Varakantham

Papers citing "Bootstrapping Language Models with DPO Implicit Rewards"

18 / 18 papers shown

Title
Direct Advantage Regression: Aligning LLMs with Online AI Reward Li He He Zhao Stephen Wan Dadong Wang Lina Yao Tongliang Liu 22 0 0 19 Apr 2025
REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective Zhihao Xu Yongqi Tong Xin Zhang Jun Zhou Xiting Wang 31 0 0 15 Apr 2025
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment Wen Yang Junhong Wu Chen Wang Chengqing Zong J. Zhang 61 1 0 06 Mar 2025
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation Songming Zhang Xue Zhang Tong Zhang Bojie Hu Yufeng Chen Jinan Xu 40 1 0 04 Mar 2025
Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data Siqi Guo Ilgee Hong Vicente Balmaseda Changlong Yu Liang Qiu Xin Liu Haoming Jiang Tuo Zhao Tianbao Yang 36 0 0 25 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 82 1 0 18 Feb 2025
Preference Optimization for Reasoning with Pseudo Feedback Fangkai Jiao Geyang Guo Xingxing Zhang Nancy F. Chen Shafiq R. Joty Furu Wei LRM 95 8 0 17 Feb 2025
Bag of Tricks for Inference-time Computation of LLM Reasoning Fan Liu Wenshuo Chao Naiqiang Tan Hao Liu OffRL LRM 69 3 0 11 Feb 2025
Process Reinforcement through Implicit Rewards Ganqu Cui Lifan Yuan Z. Wang Hanbin Wang Wendi Li ... Yu Cheng Zhiyuan Liu Maosong Sun Bowen Zhou Ning Ding OffRL LRM 65 50 0 03 Feb 2025
Geometric-Averaged Preference Optimization for Soft Preference Labels Hiroki Furuta Kuang-Huei Lee Shixiang Shane Gu Y. Matsuo Aleksandra Faust Heiga Zen Izzeddin Gur 46 6 0 31 Dec 2024
Sample-Efficient Alignment for LLMs Zichen Liu Changyu Chen Chao Du Wee Sun Lee Min-Bin Lin 26 3 0 03 Nov 2024
Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Jing Jiang Min-Bin Lin 25 8 0 09 Oct 2024
Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning Yifang Chen Shuohang Wang Ziyi Yang Hiteshi Sharma Nikos Karampatziakis Donghan Yu Kevin G. Jamieson Simon Shaolei Du Yelong Shen OffRL 33 4 0 02 Jul 2024
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning Sen Yang Leyang Cui Deng Cai Xinting Huang Shuming Shi Wai Lam 30 8 0 25 Jun 2024
Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs Xuan Zhang Chao Du Tianyu Pang Qian Liu Wei Gao Min-Bin Lin LRM AI4CE 36 34 0 13 Jun 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 153 437 0 02 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 215 291 0 18 Jan 2024
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022