Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

28 February 2024

Haoxiang Wang

Yong Lin

Wei Xiong

Tong Zhang

Papers citing "Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards"

50 / 63 papers shown

Title
Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning Xiaokun Wang Chris Jiangbo Pei Wei Shen Yi Peng ... Ai Jian Tianyidan Xie Xuchen Song Yang Liu Yahui Zhou OffRL LRM 18 0 0 12 May 2025
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes Zhuocheng Gong Jian-Yu Guan Wei Yu Wu Huishuai Zhang Dongyan Zhao 56 1 0 08 May 2025
PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model Baijiong Lin Weisen Jiang Yuancheng Xu Hao Chen Ying Chen 21 0 0 06 May 2025
ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data Haoran Gu Handing Wang Yi Mei Mengjie Zhang Yaochu Jin 27 0 0 23 Apr 2025
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment Xiaotian Zhang Ruizhe Chen Yang Feng Zuozhu Liu 40 0 0 17 Apr 2025
A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models Carlos Peláez-González Andrés Herrera-Poyatos Cristina Zuheros David Herrera-Poyatos Virilo Tejedor F. Herrera AAML 19 0 0 07 Apr 2025
Robust Multi-Objective Controlled Decoding of Large Language Models Seongho Son William Bankes Sangwoong Yoon Shyam Sundhar Ramesh Xiaohang Tang Ilija Bogunovic 39 0 0 11 Mar 2025
Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models Kefan Song Jin Yao Runnan Jiang Rohan Chandra Shangtong Zhang ALM 46 0 0 10 Mar 2025
Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustness Tingchen Fu Fazl Barez AAML 58 0 0 03 Mar 2025
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 35 0 0 03 Mar 2025
Robust Multi-Objective Preference Alignment with Online DPO Raghav Gupta Ryan Sullivan Yunxuan Li Samrat Phatale Abhinav Rastogi 32 0 0 01 Mar 2025
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment Tianze Wang Dongnan Gui Yifan Hu Shuhang Lin Linjun Zhang 31 0 0 25 Feb 2025
Drift: Decoding-time Personalized Alignments with Implicit User Preferences Minbeom Kim Kang-il Lee Seongho Joo Hwaran Lee Thibaut Thonet Kyomin Jung AI4TS 105 1 0 20 Feb 2025
Multi-Attribute Steering of Language Models via Targeted Intervention Duy Nguyen Archiki Prasad Elias Stengel-Eskin Mohit Bansal LLMSV 106 0 0 18 Feb 2025
Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond Weiyu Chen Xiaoyuan Zhang Baijiong Lin Xi Victoria Lin Han Zhao Qingfu Zhang James T. Kwok 73 1 0 19 Jan 2025
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd Shang Liu Yu Pan Guanting Chen Xiaocheng Li 75 2 0 19 Nov 2024
Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization Zhuotong Chen Fang Liu Jennifer Zhu Wanyu Du Yanjun Qi 33 0 0 07 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 52 3 0 07 Nov 2024
Active Preference-based Learning for Multi-dimensional Personalization Minhyeon Oh Seungjoon Lee Jungseul Ok 26 1 0 01 Nov 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 58 3 0 30 Oct 2024
2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision Shilong Li Yancheng He Hui Huang Xingyuan Bu J. Liu Hangyu Guo Weixun Wang Jihao Gu Wenbo Su Bo Zheng 29 5 0 25 Oct 2024
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs Chris Liu Liang Zeng J. Liu Rui Yan Jujie He Chaojie Wang Shuicheng Yan Yang Liu Yahui Zhou AI4TS 37 62 0 24 Oct 2024
POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization Batuhan K. Karaman Ishmam Zabir Alon Benhaim Vishrav Chaudhary M. Sabuncu Xia Song AI4CE 29 0 0 16 Oct 2024
Taming Overconfidence in LLMs: Reward Calibration in RLHF Jixuan Leng Chengsong Huang Banghua Zhu Jiaxin Huang 26 7 0 13 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 87 2 0 10 Oct 2024
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment Yuancheng Xu Udari Madhushani Sehwag Alec Koppel Sicheng Zhu Bang An Furong Huang Sumitra Ganesh 47 5 0 10 Oct 2024
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits Duy Nguyen Archiki Prasad Elias Stengel-Eskin Mohit Bansal 23 2 0 02 Oct 2024
HelpSteer2-Preference: Complementing Ratings with Preferences Zhilin Wang Alexander Bukharin Olivier Delalleau Daniel Egert Gerald Shen Jiaqi Zeng Oleksii Kuchaiev Yi Dong ALM 42 37 0 02 Oct 2024
Inference-Time Language Model Alignment via Integrated Value Guidance Zhixuan Liu Zhanhui Zhou Yuanfu Wang Chao Yang Yu Qiao 24 7 0 26 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 10 0 18 Sep 2024
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do Guijin Son Hyunwoo Ko Hoyoung Lee Yewon Kim Seunghyeok Hong ALM ELM 43 5 0 17 Sep 2024
On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization Yong Lin Skyler Seto Maartje ter Hoeve Katherine Metcalf B. Theobald Xuan Wang Yizhe Zhang Chen Huang Tong Zhang 29 12 0 05 Sep 2024
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models Wenxuan Zhang Philip H. S. Torr Mohamed Elhoseiny Adel Bibi 46 9 0 27 Aug 2024
Personality Alignment of Large Language Models Minjun Zhu Linyi Yang Yue Zhang Yue Zhang ALM 50 5 0 21 Aug 2024
Orchestrating LLMs with Different Personalizations Jin Peng Zhou Katie Z Luo Jingwen Gu Jason Yuan Kilian Q. Weinberger Wen Sun 46 2 0 04 Jul 2024
DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging Tzu-Han Lin Chen An Li Hung-yi Lee Yun-Nung Chen VLM ALM 26 4 0 01 Jul 2024
BAPO: Base-Anchored Preference Optimization for Personalized Alignment in Large Language Models Gihun Lee Minchan Jeong Yujin Kim Hojung Jung Jaehoon Oh Sangmook Kim Se-Young Yun 24 1 0 30 Jun 2024
Decoding-Time Language Model Alignment with Multiple Objectives Ruizhe Shi Yifang Chen Yushi Hu Alisa Liu Hannaneh Hajishirzi Noah A. Smith Simon Du 44 30 0 27 Jun 2024
VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation Xuan He Dongfu Jiang Ge Zhang Max W.F. Ku Achint Soni ... Yaswanth Narsupalli Rongqi Fan Zhiheng Lyu Yuchen Lin Wenhu Chen EGVM VGen ALM 43 41 0 21 Jun 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 44 132 0 18 Jun 2024
WPO: Enhancing RLHF with Weighted Preference Optimization Wenxuan Zhou Ravi Agrawal Shujian Zhang Sathish Indurthi Sanqiang Zhao Kaiqiang Song Silei Xu Chenguang Zhu 32 16 0 17 Jun 2024
Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs Rui Yang Ruomeng Ding Yong Lin Huan Zhang Tong Zhang 21 42 0 14 Jun 2024
PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences Daiwei Chen Yi Chen Aniket Rege Ramya Korlakai Vinayak 35 16 0 12 Jun 2024
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation Lu Li T. Zhang Zhiqi Bu Suyuchen Wang Huan He Jie Fu Yonghui Wu Jiang Bian Yong Chen Yoshua Bengio FedML MoMe 92 3 0 11 Jun 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 16 13 0 30 May 2024
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models Zhanhui Zhou Zhixuan Liu Jie Liu Zhichen Dong Chao Yang Yu Qiao ALM 36 20 0 29 May 2024
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization Jiancong Xiao Ziniu Li Xingyu Xie E. Getzen Cong Fang Qi Long Weijie J. Su 41 10 0 26 May 2024
Direct Preference Optimization With Unobserved Preference Heterogeneity Keertana Chidambaram Karthik Vinay Seetharaman Vasilis Syrgkanis 31 7 0 23 May 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 54 335 0 23 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 21 92 0 13 May 2024