Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 643 papers shown

Title
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 98 2 0 01 Dec 2024
DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Scheduling Xin Xie Dong Gong 82 1 0 01 Dec 2024
Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-Language Classifiers Chancharik Mitra Brandon Huang Tianning Chai Zhiqiu Lin Assaf Arbelle Rogerio Feris Leonid Karlinsky Trevor Darrell Deva Ramanan Roei Herzig VLM 121 4 0 28 Nov 2024
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning Di Zhang Jingdi Lei Junxian Li Xunzhi Wang Y. Liu ... S. M. I. Simon X. Yang Jianbo Wu Peng Ye Wanli Ouyang Dongzhan Zhou OffRL LRM 105 6 0 27 Nov 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 108 63 0 25 Nov 2024
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages Ashmal Vayani Dinura Dissanayake Hasindri Watawana Noor Ahsan Nevasini Sasikumar ... Monojit Choudhury Ivan Laptev Mubarak Shah Salman Khan Fahad A Khan 124 8 0 25 Nov 2024
Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward Zhiwei Jia Yuesong Nan Huixi Zhao Gengdai Liu EGVM 86 0 0 22 Nov 2024
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning Elita Lobo Chirag Agarwal Himabindu Lakkaraju LRM 70 5 0 22 Nov 2024
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection Gabriel Chua Shing Yee Chan Shaun Khoo 75 1 0 20 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 57 46 1 15 Nov 2024
Efficient Alignment of Large Language Models via Data Sampling Amrit Khera Rajat Ghosh Debojyoti Dutta 31 1 0 15 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 73 0 0 12 Nov 2024
Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching A. Jain Harley Wiltzer Jesse Farebrother Irina Rish Glen Berseth Sanjiban Choudhury 39 1 0 11 Nov 2024
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings Miguel Moura Ramos Tomás Almeida Daniel Vareta Filipe Azevedo Sweta Agrawal Patrick Fernandes André F. T. Martins 31 1 0 08 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 57 3 0 07 Nov 2024
TODO: Enhancing LLM Alignment with Ternary Preferences Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 33 1 0 02 Nov 2024
LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation Bowen Li Zhaoyu Li Qiwei Du Jinqi Luo Wenshan Wang ... Katia P. Sycara Pradeep Kumar Ravikumar Alexander G. Gray X. Si Sebastian A. Scherer AI4CE LRM 74 3 0 01 Nov 2024
Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation Bohan Lyu Yadi Cao Duncan Watson-Parris Leon Bergen Taylor Berg-Kirkpatrick Rose Yu 58 3 0 01 Nov 2024
DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios Junchao Wu Runzhe Zhan Derek F. Wong Shu Yang Xinyi Yang Yulin Yuan Lidia S. Chao DeLMO 43 1 0 31 Oct 2024
Constraint Back-translation Improves Complex Instruction Following of Large Language Models Y. Qi Hao Peng X. Wang Bin Xu Lei Hou Juanzi Li 56 0 0 31 Oct 2024
Smaller Large Language Models Can Do Moral Self-Correction Guangliang Liu Zhiyu Xue Rongrong Wang K. Johnson Kristen Marie Johnson LRM 23 0 0 30 Oct 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 58 3 0 30 Oct 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 45 3 0 29 Oct 2024
Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate Zhiqi Bu Xiaomeng Jin Bhanukiran Vinzamuri Anil Ramakrishna Kai-Wei Chang V. Cevher Mingyi Hong MU 83 6 0 29 Oct 2024
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function Zhichao Wang Bin Bi Z. Zhu Xiangbo Mao Jun Wang Shiyu Wang CLL 23 1 0 28 Oct 2024
L3Ms -- Lagrange Large Language Models Guneet S. Dhillon Xingjian Shi Yee Whye Teh Alex Smola 67 0 0 28 Oct 2024
Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation Jaechang Kim Jinmin Goh Inseok Hwang Jaewoong Cho Jungseul Ok ELM 18 1 0 28 Oct 2024
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring Honglin Mu Han He Yuxin Zhou Yunlong Feng Yang Xu ... Zeming Liu Xudong Han Qi Shi Qingfu Zhu Wanxiang Che AAML 33 1 0 28 Oct 2024
Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks? Xuan He Da Yin Nanyun Peng LRM 38 0 0 27 Oct 2024
Vulnerability of LLMs to Vertically Aligned Text Manipulations Zhecheng Li Y. Wang Bryan Hooi Yujun Cai Zhen Xiong Nanyun Peng Kai-Wei Chang 51 1 0 26 Oct 2024
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization Xiyue Peng Hengquan Guo Jiawei Zhang Dongqing Zou Ziyu Shao Honghao Wei Xin Liu 34 0 0 25 Oct 2024
Improving Model Factuality with Fine-grained Critique-based Evaluator Yiqing Xie Wenxuan Zhou Pradyot Prakash Di Jin Yuning Mao ... Sinong Wang Han Fang Carolyn Rose Daniel Fried Hejia Zhang HILM 33 5 0 24 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies L. Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 44 3 0 24 Oct 2024
End-to-end Training for Recommendation with Language-based User Profiles Zhaolin Gao Joyce Zhou Yijia Dai Thorsten Joachims AI4Ed 49 3 0 24 Oct 2024
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model Wenhong Zhu Zhiwei He Xiaofeng Wang Pengfei Liu Rui Wang OSLM 47 3 0 24 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 46 4 0 23 Oct 2024
Understanding Layer Significance in LLM Alignment Guangyuan Shi Zexin Lu Xiaoyu Dong Wenlong Zhang Xuanyu Zhang Yujie Feng Xiao-Ming Wu 48 2 0 23 Oct 2024
CLEAR: Character Unlearning in Textual and Visual Modalities Alexey Dontsov Dmitrii Korzh Alexey Zhavoronkin Boris Mikheev Denis Bobkov Aibek Alanov Oleg Y. Rogov Ivan V. Oseledets Elena Tutubalina AILaw VLM MU 66 5 0 23 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Aaron C. Courville OffRL 77 5 0 23 Oct 2024
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models Jinghan Jia Jiancheng Liu Yihua Zhang Parikshit Ram Nathalie Baracaldo Sijia Liu MU 35 2 0 23 Oct 2024
Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks Samuele Poppi Zheng-Xin Yong Yifei He Bobbie Chern Han Zhao Aobo Yang Jianfeng Chi AAML 45 12 0 23 Oct 2024
Large Language Models Empowered Personalized Web Agents Hongru Cai Yongqi Li W. Wang Fengbin Zhu Xiaoyu Shen Wenjie Li Tat-Seng Chua LLMAG 43 12 0 22 Oct 2024
Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment Mingzhi Wang Chengdong Ma Qizhi Chen Linjian Meng Yang Han Jiancong Xiao Zhaowei Zhang Jing Huo Weijie Su Yaodong Yang 32 4 0 22 Oct 2024
BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data Wenkai Li Jiarui Liu Andy Liu Xuhui Zhou Mona Diab Maarten Sap 48 6 0 21 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 50 2 0 20 Oct 2024
Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens Zhepeng Cen Yao Liu Siliang Zeng Pratik Chaudhar Huzefa Rangwala George Karypis Rasool Fakoor SyDa AIFin 18 3 0 18 Oct 2024
Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning Xiaochuan Li Zichun Yu Chenyan Xiong SyDa 27 1 0 18 Oct 2024
Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus Raviraj Joshi Kanishk Singla Anusha Kamath Raunak Kalani Rakesh Paul Utkarsh Vaidya Sanjay Singh Chauhan Niranjan Wartikar Eileen Long SyDa CLL 31 2 0 18 Oct 2024
Teaching Models to Balance Resisting and Accepting Persuasion Elias Stengel-Eskin Peter Hase Mohit Bansal MU 29 4 0 18 Oct 2024
SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment Qin Liu Fei Wang Chaowei Xiao Muhao Chen 62 0 0 18 Oct 2024