Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 641 papers shown

Title
Moral Alignment for LLM Agents Elizaveta Tennant Stephen Hailes Mirco Musolesi 35 0 0 02 Oct 2024
Reasoning Elicitation in Language Models via Counterfactual Feedback Alihan Hüyük Xinnuo Xu Jacqueline Maasch Aditya V. Nori Javier González ReLM LRM 65 1 0 02 Oct 2024
Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models Angela Lopez-Cardona Carlos Segura Alexandros Karatzoglou Sergi Abadal Ioannis Arapakis ALM 48 2 0 02 Oct 2024
HelpSteer2-Preference: Complementing Ratings with Preferences Zhilin Wang Alexander Bukharin Olivier Delalleau Daniel Egert Gerald Shen Jiaqi Zeng Oleksii Kuchaiev Yi Dong ALM 42 39 0 02 Oct 2024
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown Xingzhou Lou Dong Yan Wei Shen Yuzi Yan Jian Xie Junge Zhang 45 21 0 01 Oct 2024
Recent Advances in Speech Language Models: A Survey Wenqian Cui Dianzhi Yu Xiaoqi Jiao Ziqiao Meng Guangyan Zhang Qichao Wang Yiwen Guo Irwin King AuLLM 59 14 0 01 Oct 2024
PersonalLLM: Tailoring LLMs to Individual Preferences Thomas P. Zollo Andrew Siah Naimeng Ye Ang Li Hongseok Namkoong 30 10 0 30 Sep 2024
The Crucial Role of Samplers in Online Direct Preference Optimization Ruizhe Shi Runlong Zhou Simon S. Du 53 7 0 29 Sep 2024
CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering Yike Wu Yi Huang Nan Hu Yuncheng Hua Guilin Qi Jiaoyan Chen Jeff Z. Pan 33 6 0 29 Sep 2024
An Adversarial Perspective on Machine Unlearning for AI Safety Jakub Łucki Boyi Wei Yangsibo Huang Peter Henderson F. Tramèr Javier Rando MU AAML 71 31 0 26 Sep 2024
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference Qining Zhang Lei Ying OffRL 35 1 0 25 Sep 2024
Speechworthy Instruction-tuned Language Models Hyundong Justin Cho Nicolaas Jedema Leonardo F. R. Ribeiro Karishma Sharma Pedro Szekely Alessandro Moschitti Ruben Janssen Jonathan May ALM 40 1 0 23 Sep 2024
Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension Chenxu Wang Ping Jian Zhen Yang LRM 22 0 0 22 Sep 2024
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models Anmol Mekala Vineeth Dorna Shreya Dubey Abhishek Lalwani David Koleczek Mukund Rungta Sadid Hasan Elita Lobo KELM MU 31 1 0 20 Sep 2024
Learning to Compare Hardware Designs for High-Level Synthesis Yunsheng Bai Atefeh Sohrabizadeh Zijian Ding Rongjian Liang Weikai Li Ding Wang Haoxing Ren Yizhou Sun Jason Cong 30 0 0 20 Sep 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 29 2 0 20 Sep 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair Mingjie Liu Yun-Da Tsai Wenfei Zhou Haoxing Ren SyDa 3DV 45 5 0 19 Sep 2024
MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts Tianle Gu Kexin Huang Ruilin Luo Yuanqi Yao Yujiu Yang Yan Teng Yingchun Wang MU 21 4 0 18 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 10 0 18 Sep 2024
AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robots Zhaxizhuoma Pengan Chen Ziniu Wu Jiawei Sun Dong Wang Peng Zhou Nieqing Cao Yan Ding Bin Zhao Xuelong Li 41 4 0 18 Sep 2024
Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse Maojia Song Shang Hong Sim Rishabh Bhardwaj Hai Leong Chieu Navonil Majumder Soujanya Poria 29 6 0 17 Sep 2024
KodeXv0.1: A Family of State-of-the-Art Financial Large Language Models Neel Rajani Lilli Kiessling Aleksandr Ogaltsov Claus Lang ALM 21 0 0 13 Sep 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 84 5 0 13 Sep 2024
Propaganda is all you need Paul Kronlund-Drouault 54 0 0 13 Sep 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 37 11 0 11 Sep 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 58 23 0 10 Sep 2024
Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models Xingyun Hong Yan Shao Zhilin Wang Manni Duan Jin Xiongnan 26 0 0 09 Sep 2024
Programming Refusal with Conditional Activation Steering Bruce W. Lee Inkit Padhi K. Ramamurthy Erik Miehling Pierre L. Dognin Manish Nagireddy Amit Dhurandhar LLMSV 91 13 0 06 Sep 2024
Recent Advances in Attack and Defense Approaches of Large Language Models Jing Cui Yishi Xu Zhewei Huang Shuchang Zhou Jianbin Jiao Junge Zhang PILM AAML 50 1 0 05 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 51 0 0 28 Aug 2024
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models Wenxuan Zhang Philip H. S. Torr Mohamed Elhoseiny Adel Bibi 48 9 0 27 Aug 2024
Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates Hui Wei Shenghua He Tian Xia Andy H. Wong Jingyang Lin Mei Han Mei Han ALM ELM 59 23 0 23 Aug 2024
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data Chenglong Wang Yang Gan Yifu Huo Yongyu Mu Murun Yang ... Chunliang Zhang Tongran Liu Quan Du Di Yang Jingbo Zhu VLM 61 4 0 22 Aug 2024
Personality Alignment of Large Language Models Minjun Zhu Linyi Yang Yue Zhang Yue Zhang ALM 55 5 0 21 Aug 2024
Understanding Generative AI Content with Embedding Models Max Vargas Reilly Cannon A. Engel Anand D. Sarwate Tony Chiang 42 3 0 19 Aug 2024
How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments Yusuke Ide Yuto Nishida Miyu Oba Miyu Oba Justin Vasselli Hidetaka Kamigaito Taro Watanabe 31 2 0 19 Aug 2024
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization Yuxin Jiang Bo Huang Yufei Wang Xingshan Zeng Liangyou Li Yasheng Wang Xin Jiang Lifeng Shang Ruiming Tang Wei Wang 42 5 0 14 Aug 2024
The advantages of context specific language models: the case of the Erasmian Language Model João Gonçalves Nick Jelicic Michele Murgia Evert Stamhuis 26 0 0 13 Aug 2024
HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes Xuanyu Su Yansong Li Diana Inkpen Nathalie Japkowicz VLM 81 2 0 11 Aug 2024
Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts Tingchen Fu Yupeng Hou Julian McAuley Rui Yan 28 3 0 09 Aug 2024
Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate Yiqun Zhang Xiaocui Yang Shi Feng Daling Wang Yifei Zhang Kaisong Song LLMAG 27 4 0 08 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 27 5 0 06 Aug 2024
Intermediate direct preference optimization Atsushi Kojima 18 0 0 06 Aug 2024
Tamper-Resistant Safeguards for Open-Weight LLMs Rishub Tamirisa Bhrugu Bharathi Long Phan Andy Zhou Alice Gatti ... Andy Zou Dawn Song Bo Li Dan Hendrycks Mantas Mazeika AAML MU 47 37 0 01 Aug 2024
TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods Gabriel Loiseau Damien Sileo Damien Riquet Maxime Meyer Marc Tommasi 25 0 0 31 Jul 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 44 71 0 28 Jul 2024
Grammar-based Game Description Generation using Large Language Models Tsunehiko Tanaka Edgar Simo-Serra 48 2 0 24 Jul 2024
Towards Aligning Language Models with Textual Feedback Sauc Abadal Lloret S. Dhuliawala K. Murugesan Mrinmaya Sachan VLM 33 1 0 24 Jul 2024
Leveraging Large Language Models to Geolocate Linguistic Variations in Social Media Posts Davide Savarro Davide Zago Stefano Zoia 13 0 0 22 Jul 2024