Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 641 papers shown

Title
TimeHF: Billion-Scale Time Series Models Guided by Human Feedback Yongzhi Qi Hao Hu Dazhou Lei Jianshen Zhang Zhengxin Shi Yulin Huang Zhengyu Chen Xiaoming Lin Zuo-jun Shen AI4TS AI4CE 39 1 0 28 Jan 2025
DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data Ke-Han Lu Zhehuai Chen Szu-Wei Fu Chao-Han Huck Yang Jagadeesh Balam Boris Ginsburg Yu-Te Wang Hung-yi Lee AuLLM SyDa 99 5 0 28 Jan 2025
Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation Takyoung Kim Kyungjae Lee Y. Jang Ji Yong Cho Gangwoo Kim Minseok Cho Moontae Lee 89 0 0 28 Jan 2025
BoKDiff: Best-of-K Diffusion Alignment for Target-Specific 3D Molecule Generation Ali Khodabandeh Yalabadi Mehdi Yazdani-Jahromi O. Garibay 33 0 0 28 Jan 2025
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning Zehan Qi Xiao-Chang Liu Iat Long Iong Hanyu Lai X. Sun ... Shuntian Yao Tianjie Zhang Wei Xu J. Tang Yuxiao Dong 93 14 0 28 Jan 2025
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics Kai He Rui Mao Qika Lin Yucheng Ruan Xiang Lan Mengling Feng Erik Cambria LM&MA AILaw 90 151 0 28 Jan 2025
Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework Yulong Hu Tingting Dong Sen Li OffRL OnRL 54 0 0 24 Jan 2025
Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment Melissa Kazemi Rad Huy Nghiem Andy Luo Sahil Wadhwa Mohammad Sorower Stephen Rawls AAML 91 2 0 22 Jan 2025
Evolution and The Knightian Blindspot of Machine Learning Joel Lehman Elliot Meyerson Tarek El-Gaaly Kenneth O. Stanley Tarin Ziyaee 78 1 0 22 Jan 2025
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback Yafu Li Xuyang Hu Xiaoye Qu Linjie Li Yu-Xi Cheng 51 3 0 22 Jan 2025
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning Haotian Luo Li Shen Haiying He Y. Wang Shiwei Liu Wei Li Naiqiang Tan Xiaochun Cao Dacheng Tao VLM LRM 86 36 0 22 Jan 2025
Kimi k1.5: Scaling Reinforcement Learning with LLMs Kimi Team Angang Du Bofei Gao Bowei Xing Changjiu Jiang ... Zhilin Yang Zhiqi Huang Zihao Huang Ziyao Xu Z. Yang VLM ALM OffRL AI4TS LRM 106 132 0 22 Jan 2025
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling Yi Wang Xinhao Li Ziang Yan Yinan He Jiashuo Yu ... Kai Chen Wenhai Wang Yu Qiao Yali Wang Limin Wang 73 19 0 21 Jan 2025
DiffDoctor: Diagnosing Image Diffusion Models Before Treating Yiyang Wang Xi Chen Xiaogang Xu S. Ji Y. Liu Yujun Shen Hengshuang Zhao DiffM 49 0 0 21 Jan 2025
Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Zhenyu Hou Xin Lv Rui Lu J. Zhang Y. Li Zijun Yao Juanzi Li J. Tang Yuxiao Dong OffRL LRM ReLM 49 20 0 20 Jan 2025
Revisiting Ensemble Methods for Stock Trading and Crypto Trading Tasks at ACM ICAIF FinRL Contest 2023-2024 Nikolaus Holzer Keyi Wang Kairong Xiao Xiao-Yang Liu Yanglet AIFin 30 1 0 18 Jan 2025
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion Yannis Flet-Berliac Nathan Grinsztajn Florian Strub Bill Wu Eugene Choi ... Arash Ahmadian Yash Chandak M. G. Azar Olivier Pietquin Matthieu Geist OffRL 62 4 0 17 Jan 2025
Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis Lanling Xu Junjie Zhang Bingqian Li Jinpeng Wang Sheng Chen Wayne Xin Zhao Ji-Rong Wen 74 18 0 17 Jan 2025
A General Framework for Inference-time Scaling and Steering of Diffusion Models R. Singhal Zachary Horvitz Ryan Teehan Mengye Ren Zhou Yu Kathleen McKeown Rajesh Ranganath DiffM 61 15 0 17 Jan 2025
Direct Unlearning Optimization for Robust and Safe Text-to-Image Models Yong-Hyun Park Sangdoo Yun Jin-Hwa Kim Junho Kim Geonhui Jang Yonghyun Jeong Junghyo Jo Gayoung Lee 73 13 0 17 Jan 2025
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words Junyi Ao Yuancheng Wang Xiaohai Tian Dekun Chen J. Zhang Lu Lu Y. Wang Haizhou Li Z. Wu AuLLM 80 16 0 17 Jan 2025
Foundation Models at Work: Fine-Tuning for Fairness in Algorithmic Hiring Buse Sibel Korkmaz Rahul Nair Elizabeth M. Daly Evangelos Anagnostopoulos Christos Varytimidis Antonio del Rio Chanona 40 0 0 13 Jan 2025
Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questions Aidan Hogan Xin Luna Dong Denny Vrandečić Gerhard Weikum 50 1 0 12 Jan 2025
Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning Zhen Li Yupeng Su Runming Yang C. Xie Z. Wang Zhongwei Xie Ngai Wong Hongxia Yang MQ LRM 30 3 0 06 Jan 2025
SR-Reward: Taking The Path More Traveled Seyed Mahdi Basiri Azad Zahra Padar Gabriel Kalweit Joschka Boedecker OffRL 67 0 0 04 Jan 2025
Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection Yachao Zhao Bo Wang Yan Wang 48 2 0 04 Jan 2025
DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learning Utsav Singh Souradip Chakraborty Wesley A. Suttle Brian M. Sadler Vinay P. Namboodiri Amrit Singh Bedi OffRL 51 0 0 03 Jan 2025
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria Joonwon Jang Jaehee Kim Wonbin Kweon Hwanjo Yu LRM 31 1 0 03 Jan 2025
Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey Junqiao Wang Zeng Zhang Yangfan He Yuyang Song Tianyu Shi ... Hengyuan Xu Kunyu Wu Guangwu Qian Qiuwu Chen Lewei He 38 8 0 03 Jan 2025
ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze Chunyu Xuan Yazhe Niu Yuan Pu Shuai Hu Yu Liu Jing Yang 51 0 0 03 Jan 2025
Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search Shuangtao Li Shuaihao Dong Kexin Luan Xinhan Di Chaofan Ding LRM 43 1 0 02 Jan 2025
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems Hashmath Shaik Alex Doboli OffRL ELM 71 0 0 31 Dec 2024
ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantation Weilong Dong Xinwei Wu Renren Jin Shaoyang Xu Deyi Xiong 54 6 0 31 Dec 2024
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment Jianfei Zhang Jun Bai B. Li Yanmeng Wang Rumei Li Chenghua Lin Wenge Rong 41 0 0 31 Dec 2024
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 88 11 0 31 Dec 2024
LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robots Dongge Han Trevor A. McInroe Adam Jelley Stefano V. Albrecht Peter Bell Amos Storkey 51 9 0 31 Dec 2024
TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization Chia-Yu Hung Navonil Majumder Zhifeng Kong Ambuj Mehrish Rafael Valle Bryan Catanzaro Soujanya Poria Bryan Catanzaro Soujanya Poria 52 5 0 30 Dec 2024
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models Huawen Feng Pu Zhao Qingfeng Sun Can Xu Fangkai Yang ... Qianli Ma Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang AAML ALM 62 0 0 23 Dec 2024
Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval Luo Ji Feixiang Guo Teng Chen Qingqing Gu Xiaoyu Wang ... Peng Yu Yue Zhao Hongyang Lei Zhonglin Jiang Yong Chen RALM LRM 89 0 0 21 Dec 2024
REFA: Reference Free Alignment for multi-preference optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 83 1 0 20 Dec 2024
SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage Xiaoning Dong Wenbo Hu Wei Xu Tianxing He 67 0 0 19 Dec 2024
Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection Le Yang Ziwei Zheng Boxu Chen Zhengyu Zhao Chenhao Lin Chao Shen VLM 135 3 0 18 Dec 2024
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models Jiale Cheng Xiao-Chang Liu C. Wang Xiaotao Gu Y. Lu Dan Zhang Yuxiao Dong J. Tang Hongning Wang Minlie Huang LRM 123 3 0 16 Dec 2024
Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets Zhen Liu Tim Z. Xiao Weiyang Liu Yoshua Bengio Dinghuai Zhang 118 2 0 10 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang J. Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 126 7 0 05 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 98 1 0 03 Dec 2024
Time-Reversal Provides Unsupervised Feedback to LLMs Yerram Varun Rahul Madhavan Sravanti Addepalli A. Suggala Karthikeyan Shanmugam Prateek Jain LRM SyDa 64 0 0 03 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 97 3 0 02 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 98 2 0 01 Dec 2024
DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling Xin Xie Dong Gong 82 1 0 01 Dec 2024