v1v2 (latest)

ORPO: Monolithic Preference Optimization without Reference Model

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

12 March 2024

ArXiv (abs)PDF HTML HuggingFace (67 upvotes)

Papers citing "ORPO: Monolithic Preference Optimization without Reference Model"

50 / 252 papers shown

Title
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tur Yuan Yao 247 14 0 16 May 2025
Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO Peter Chen Xiaopeng Li Zhiyu Li Xi Chen Tianyi Lin 499 0 0 16 May 2025
Preference Optimization for Combinatorial Optimization Problems Mingjun Pan Guanquan Lin You-Wei Luo Bin Zhu Zhien Dai Lijun Sun Chun Yuan 234 3 0 13 May 2025
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang Hyunjung Shim 257 5 0 12 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 390 0 0 05 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 320 9 0 05 May 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 264 5 0 22 Apr 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 418 0 0 18 Apr 2025
Continual Pre-Training is (not) What You Need in Domain Adaption Pin-Er Chen Da-Chen Lian S. Hsieh Sieh-Chuen Huang Hsuan-Lei Shao ... Yang-Hsien Lin Zih-Ching Chen Cheng-Kuang Eddie TC Huang Simon See CLL AILaw 278 1 0 18 Apr 2025
REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective Zhihao Xu Yongqi Tong Xin Zhang Jun Zhou Xiting Wang 188 1 0 15 Apr 2025
Improving In-Context Learning with Reasoning Distillation Nafis Sadeq Xin Xu Zhouhang Xie Julian McAuley Byungkyu Kang Prarit Lamba Xiang Gao RALM ReLM LRM 214 0 0 14 Apr 2025
Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data Shuai Zhao Linchao Zhu Yi Yang Yi Yang 397 4 0 14 Apr 2025
A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future Jialun Zhong Wei Shen Yanzeng Li Songyang Gao Hua Lu Yicheng Chen Yang Zhang Wei Zhou Jinjie Gu Lei Zou LRM 324 27 0 12 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 285 1 0 09 Apr 2025
R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation Martin Weyssow Chengran Yang Junkai Chen Ratnadira Widyasari Ting Zhang ... Ang Han Wei Frank Liauw Eng Lieh Ouh Lwin Khin Shar David Lo LRM 452 4 0 07 Apr 2025
Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration Ran Xu W. Shi Yuchen Zhuang Yue Yu Joyce C. Ho Haoyu Wang Carl Yang 217 11 0 07 Apr 2025
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Kai Ye Hongyi Zhou Jin Zhu Francesco Quinzan C. Shi 407 5 0 03 Apr 2025
More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment Yifan Wang Runjin Chen Bolian Li David Cho Yihe Deng Ruqi Zhang Tianlong Chen Zhangyang Wang A. Grama Junyuan Hong SyDa 239 4 0 03 Apr 2025
DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models S. Jung Donghun Lee Shinbok Lee Gaeun Seo Daniel Lee Byeongil Ko Junrae Cho Kihyun Kim EungGyun Kim M. Shin 291 3 0 02 Apr 2025
InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model AlignmentComputer Vision and Pattern Recognition (CVPR), 2025 Yaojie Lu Qichao Wang H. Cao Xierui Wang Xiaoyin Xu Min Zhang 295 7 0 24 Mar 2025
Debiasing Multimodal Large Language Models via Noise-Aware Preference OptimizationComputer Vision and Pattern Recognition (CVPR), 2025 Zefeng Zhang Hengzhu Tang Shuaiyi Nie Ying Tai Yiming Ren Zhenyang Li Dawei Yin Duohe Ma Tingwen Liu 268 7 0 23 Mar 2025
Modifying Large Language Model Post-Training for Diverse Creative Writing John Joon Young Chung Vishakh Padmakumar Melissa Roemmele Yuqian Sun Max Kreminski MoMe 178 17 0 21 Mar 2025
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization Yunan Wang Jijie Li Bo Zhang Liangdong Wang Guang Liu 206 3 0 20 Mar 2025
CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain SettingsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Daniil Orel Dilshod Azizov Preslav Nakov DeLMO 306 4 0 17 Mar 2025
Aligning to What? Limits to RLHF Based AlignmentNorth American Chapter of the Association for Computational Linguistics (NAACL), 2025 Logan Barnhart Reza Akbarian Bafghi Stephen Becker M. Raissi 179 3 0 12 Mar 2025
Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation Wenlong Meng Fan Zhang Wendao Yao Zhenyuan Guo Yongqian Li Chengkun Wei Wenzhi Chen AAML 286 8 0 11 Mar 2025
RePO: Understanding Preference Learning Through ReLU-Based Optimization Junkang Wu Kexin Huang Qingsong Wen Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xiang Wang 265 3 0 10 Mar 2025
SOLAR: Scalable Optimization of Large-scale Architecture for Reasoning Chen Li Yinyi Luo Anudeep Bolimera Uzair Ahmed Siyang Song Hrishikesh Gokhale Marios Savvides LRM AI4CE 368 1 0 06 Mar 2025
DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models Ruizhe Chen Wenhao Chai Zhifei Yang Xiaotian Zhang Qiufeng Wang Tony Q.S. Quek Soujanya Poria Zuozhu Liu 497 3 0 06 Mar 2025
Process-based Self-Rewarding Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Shimao Zhang Xiao Liu Xin Zhang Junxiao Liu Zheheng Luo Shujian Huang Yeyun Gong ReLM SyDa LRM 295 21 0 05 Mar 2025
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 206 0 0 03 Mar 2025
Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-EvolutionAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Keliang Li Tianhua Zhang Yunxiang Li Hongyin Luo Abdalla Moustafa Xixin Wu James Glass Helen Meng 244 5 0 03 Mar 2025
The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue AgentsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Yihong Tang Kehai Chen X. Bai Zhengyu Niu Binghai Wang J. Tang Min Zhang LLMAG 221 3 0 28 Feb 2025
Optimizing Large Language Models for ESG Activity Detection in Financial Texts Mattia Birti Francesco Osborne Andrea Maurino 163 2 0 28 Feb 2025
Controlled Diversity: Length-optimized Natural Language Generation Diana Marie Schenke Timo Baumann 142 0 0 26 Feb 2025
Conformal Linguistic Calibration: Trading-off between Factuality and Specificity Zhengping Jiang Anqi Liu Benjamin Van Durme 484 7 0 26 Feb 2025
ZEBRA: Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Construction Jeesu Jung Chanjun Park Sangkeun Jung 234 0 0 26 Feb 2025
AMPO: Active Multi-Preference Optimization for Self-play Preference Selection Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 314 0 0 25 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 391 2 0 25 Feb 2025
CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization Shuming Shi Ruobing Zuo Gaolei He Jianlin Wang Chenyang Xu Zhengfeng Yang 316 0 0 25 Feb 2025
Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance Chenghua Huang Lu Wang Fangkai Yang Pu Zhao Hao Sun Qingwei Lin Dongmei Zhang Saravan Rajmohan Qi Zhang OffRL 195 3 0 24 Feb 2025
CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale Chenlong Wang Zhaoyang Chu Zhengxiang Cheng Xuyi Yang Kaiyue Qiu Yao Wan Zhou Zhao Xuanhua Shi Benlin Liu ALM SyDa 309 3 0 23 Feb 2025
C3AI: Crafting and Evaluating Constitutions for Constitutional AIThe Web Conference (WWW), 2025 Yara Kyrychenko Ke Zhou Edyta Bogucka Daniele Quercia ELM 202 10 0 21 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without HyperparametersInternational Conference on Learning Representations (ICLR), 2025 Teng Xiao Yige Yuan Ziyang Chen Mingxiao Li Shangsong Liang Zhaochun Ren V. Honavar 600 21 0 21 Feb 2025
SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text GenerationInternational Conference on Learning Representations (ICLR), 2025 Song Duong Florian Le Bronnec Alexandre Allauzen Vincent Guigue Alberto Lumbreras Laure Soulier Patrick Gallinari HILM 241 3 0 20 Feb 2025
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic DataAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Cehao Yang Xueyuan Lin Chengjin Xu Xuhui Jiang Shengjie Ma Aofan Liu Hui Xiong Jian Guo LRM 278 7 0 18 Feb 2025
Personalized Top-k Set Queries Over Predicted Scores Sohrab Namazi Nia Subhodeep Ghosh Senjuti Basu Roy S. Amer-Yahia 247 0 0 18 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu Volkan Cevher 484 2 0 18 Feb 2025
Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL), 2025 Yingqian Cui Pengfei He Jingying Zeng Hui Liu Xianfeng Tang ... Zhen Li Suhang Wang Yue Xing Shucheng Zhou Qi He LRM 506 28 0 18 Feb 2025
RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation C. Zhou Xinyu Zhang Dandan Song Xiancai Chen Wanli Gu Huipeng Ma Yuhang Tian Hao Fei Linmei Hu 267 4 0 13 Feb 2025