v1v2 (latest)

ORPO: Monolithic Preference Optimization without Reference Model

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

12 March 2024

ArXiv (abs)PDF HTML HuggingFace (67 upvotes)

Papers citing "ORPO: Monolithic Preference Optimization without Reference Model"

50 / 252 papers shown

Title
ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates L. Yang Zhaochen Yu Tengjiao Wang Mengdi Wang ReLM LRM AI4CE 524 42 0 10 Feb 2025
Evolving LLMs' Self-Refinement Capability via Iterative Preference Optimization Yongcheng Zeng Xinyu Cui Xuanfa Jin Guoqing Liu Guoqing Liu ... Ning Yang Jun Wang Jianye Hao Haifeng Zhang Jun Wang LLMAG LRM 366 5 0 08 Feb 2025
QExplorer: Large Language Model Based Query Extraction for Toxic Content Exploration Shaola Ren Li Ke Longtao Huang Dehong Gao Hui Xue 151 0 0 06 Feb 2025
COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models Tobias Materzok LRM 294 1 0 28 Jan 2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment BenchmarkingInternational Conference on Learning Representations (ICLR), 2024 Benjamin Feuer Micah Goldblum Teresa Datta Sanjana Nambiar Raz Besaleli Samuel Dooley Max Cembalest John P. Dickerson ALM 334 26 0 28 Jan 2025
HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor Zihui Wu Haichang Gao Jiacheng Luo Zhaoxiang Liu 412 1 0 23 Jan 2025
In-situ graph reasoning and knowledge expansion using Graph-PReFLexOR Markus J. Buehler 147 7 0 14 Jan 2025
FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference RankingsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Tong Liu Xiao Yu Wenxuan Zhou Jindong Gu Volker Tresp 372 3 0 11 Jan 2025
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled CriteriaAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Joonwon Jang Jaehee Kim Wonbin Kweon Seonghyeon Lee Hwanjo Yu LRM 493 1 0 30 Dec 2024
REFA: Reference Free Alignment for multi-preference optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 435 1 0 20 Dec 2024
Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model Yuzhong Hong Hanshan Zhang Junwei Bao Hongfei Jiang Yang Song OffRL 225 6 0 18 Dec 2024
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative SamplesInternational Conference on Computational Linguistics (COLING), 2024 Shuo Xie Fangzhi Zhu Jiahui Wang Lulu Wen Wei Dai Xiaowei Chen Junxiong Zhu Kai Zhou Bo Zheng 158 0 0 13 Dec 2024
Learning to Reason via Self-Iterative Process Feedback for Small Language ModelsInternational Conference on Computational Linguistics (COLING), 2024 Kaiyuan Chen Jin Wang Xuejie Zhang LRM ReLM 192 2 0 11 Dec 2024
Classifier-free guidance in LLMs Safety Roman Smirnov MU 154 1 0 08 Dec 2024
FANAL -- Financial Activity News Alerting Language Modeling FrameworkBigData Congress [Services Society] (BSS), 2024 Urjitkumar Patel Fang-Chun Yeh Chinmay Gondhalekar Hari Nalluri AIFin 215 4 0 04 Dec 2024
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs Zhihan Liu Shenao Zhang Yongfei Liu Boyi Liu Yingxiang Yang Zhaoran Wang 381 6 0 20 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 482 178 1 15 Nov 2024
1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs Jebish Purbey Siddartha Pullakhandam Kanwal Mehreen Muhammad Arham Drishti Sharma Ashay Srivastava Ram Mohan Rao Kadiyala 130 2 0 11 Nov 2024
Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs Megh Thakkar Yash More Quentin Fournier Matthew D Riemer Pin-Yu Chen Payel Das Payel Das MoMe 178 7 0 11 Nov 2024
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings Miguel Moura Ramos Tomás Almeida Daniel Vareta Filipe Azevedo Sweta Agrawal Patrick Fernandes Marcely Zanon Boito 376 6 0 08 Nov 2024
GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models Nizar Islah Justine Gehring Diganta Misra Eilif B. Muller Irina Rish Terry Yue Zhuo Massimo Caccia SyDa 134 4 0 05 Nov 2024
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment Dongxu Liu Bing Xu Yinzhuo Chen Bufan Xu Wenpeng Lu Muyun Yang Tiejun Zhao MoE 176 1 0 02 Nov 2024
TODO: Enhancing LLM Alignment with Ternary PreferencesInternational Conference on Learning Representations (ICLR), 2024 Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 327 5 0 02 Nov 2024
VPO: Leveraging the Number of Votes in Preference Optimization Jae Hyeon Cho Minkyung Park Byung-Jun Lee 78 2 0 30 Oct 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence MinimizationInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2024 Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 334 9 0 29 Oct 2024
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function Zhichao Wang Bin Bi Z. Zhu Xiangbo Mao Jun Wang Shiyu Wang CLL 231 5 0 28 Oct 2024
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned ModelInternational Conference on Learning Representations (ICLR), 2024 Wenhong Zhu Zhiwei He Xiaofeng Wang Pengfei Liu Rui Wang OSLM 330 12 0 24 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual AlignmentNorth American Chapter of the Association for Computational Linguistics (NAACL), 2024 Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 392 15 0 23 Oct 2024
Augmenting Legal Decision Support Systems with LLM-based NLI for Analyzing Social Media Evidence Ram Mohan Rao Kadiyala Siddartha Pullakhandam Kanwal Mehreen Subhasya Tippareddy Ashay Srivastava AILaw 124 1 0 21 Oct 2024
Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning - A Convex Optimization Perspective H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 414 10 0 20 Oct 2024
Holistic Utility Preference Learning for Listwise Alignment Jiacong Zhou Xianyun Wang Jun Yu Jun Yu 312 3 0 17 Oct 2024
Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models Iaroslav Chelombitko Egor Safronov Aleksey Komissarov 186 2 0 16 Oct 2024
PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking Markus J. Buehler ReLM LRM 184 19 0 16 Oct 2024
CREAM: Consistency Regularized Self-Rewarding Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Zhaoxiang Wang Weilei He Zhiyuan Liang Xuchao Zhang Chetan Bansal Ying Wei Weitong Zhang Huaxiu Yao ALM 521 24 0 16 Oct 2024
Understanding Likelihood Over-optimisation in Direct Alignment Algorithms Zhengyan Shi Sander Land Acyr Locatelli Matthieu Geist Max Bartolo 311 9 0 15 Oct 2024
Varying Shades of Wrong: Aligning LLMs with Wrong Answers OnlyInternational Conference on Learning Representations (ICLR), 2024 Jihan Yao Wenxuan Ding Shangbin Feng Lucy Lu Wang Yulia Tsvetkov 221 4 0 14 Oct 2024
Taming Overconfidence in LLMs: Reward Calibration in RLHFInternational Conference on Learning Representations (ICLR), 2024 Jixuan Leng Chengsong Huang Banghua Zhu Jiaxin Huang 349 35 0 13 Oct 2024
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both Abhijnan Nath Changsoo Jung Ethan Seefried Nikhil Krishnaswamy 1.0K 5 0 11 Oct 2024
Evolutionary Contrastive Distillation for Language Model AlignmentConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Julian Katz-Samuels Zheng Li Hyokun Yun Priyanka Nigam Yi Xu Vaclav Petricek Bing Yin Trishul Chilimbi ALM SyDa 88 1 0 10 Oct 2024
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference TreesInternational Conference on Learning Representations (ICLR), 2024 Weibin Liao Xu Chu Yasha Wang LRM 417 13 0 10 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Yanzhe Zhang Yingxiang Yang Yunxing Liu Liyu Chen Tao Sun Ziyi Wang 563 5 0 10 Oct 2024
Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editingAnnual Meeting of the Association for Computational Linguistics (ACL), 2024 Kaishuai Xu Tiezheng YU Wenjun Hou Yi Cheng Chak Tou Leong Liangyou Li Xin Jiang Lifeng Shang Qun Liu Wenjie Li LRM 1.0K 0 0 09 Oct 2024
Reasoning Paths Optimization: Learning to Reason and Explore From Diverse PathsConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Yew Ken Chia Guizhen Chen Weiwen Xu Luu Anh Tuan Soujanya Poria Lidong Bing LRM 192 3 0 07 Oct 2024
SparsePO: Controlling Preference Alignment of LLMs via Sparse Token MasksConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Fenia Christopoulou Ronald Cardenas Gerasimos Lampouras H. Ammar Jun Wang 187 5 0 07 Oct 2024
MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Guanzhen Li Yuxi Xie Min-Yen Kan VLM 793 8 0 06 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFInternational Conference on Learning Representations (ICLR), 2024 Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 383 13 0 06 Oct 2024
Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification Zhenwen Liang Ye Liu Tong Niu Xiangliang Zhang Yingbo Zhou Semih Yavuz LRM 214 35 0 05 Oct 2024
RainbowPO: A Unified Framework for Combining Improvements in Preference OptimizationInternational Conference on Learning Representations (ICLR), 2024 Hanyang Zhao Genta Indra Winata Anirban Das Shi-Xiong Zhang D. Yao Wenpin Tang Sambit Sahu 357 17 0 05 Oct 2024
X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at ScaleInternational Conference on Learning Representations (ICLR), 2024 Haoran Xu Kenton W. Murray Philipp Koehn Hieu T. Hoang Akiko Eriguchi Huda Khayrallah 295 27 0 04 Oct 2024
Investigating on RLHF methodology Alexey Kutalev Sergei Markoff 89 0 0 02 Oct 2024