Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

17 October 2023

Luke Zettlemoyer

Yejin Choi

Prithviraj Ammanabrolu

MoMe

ArXiv PDF HTML

Papers citing "Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging"

50 / 117 papers shown

Title
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment Yuancheng Xu Udari Madhushani Sehwag Alec Koppel Sicheng Zhu Bang An Furong Huang Sumitra Ganesh 52 6 0 10 Oct 2024
Can Language Models Reason about Individualistic Human Values and Preferences? Liwei Jiang Taylor Sorensen Sydney Levine Yejin Choi 28 7 0 04 Oct 2024
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits Duy Nguyen Archiki Prasad Elias Stengel-Eskin Mohit Bansal 23 2 0 02 Oct 2024
PersonalLLM: Tailoring LLMs to Individual Preferences Thomas P. Zollo Andrew Siah Naimeng Ye Ang Li Hongseok Namkoong 30 10 0 30 Sep 2024
Inference-Time Language Model Alignment via Integrated Value Guidance Zhixuan Liu Zhanhui Zhou Yuanfu Wang Chao Yang Yu Qiao 24 7 0 26 Sep 2024
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering Ziyu Zhao Tao Shen Didi Zhu Zexi Li Jing Su Xuwu Wang Kun Kuang Fei Wu MoMe 16 5 0 24 Sep 2024
Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models Alireza Salemi Hamed Zamani 35 9 0 14 Sep 2024
Personality Alignment of Large Language Models Minjun Zhu Linyi Yang Yue Zhang Yue Zhang ALM 52 5 0 21 Aug 2024
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning S. Poddar Yanming Wan Hamish Ivison Abhishek Gupta Natasha Jaques 27 33 0 19 Aug 2024
Diffusion Guided Language Modeling Justin Lovelace Varsha Kishore Yiwei Chen Kilian Q. Weinberger 31 6 0 08 Aug 2024
Improving Context-Aware Preference Modeling for Language Models Silviu Pitis Ziang Xiao Nicolas Le Roux Alessandro Sordoni 21 8 0 20 Jul 2024
SoupLM: Model Integration in Large Language and Multi-Modal Models Yue Bai Zichen Zhang Jiasen Lu Yun Fu MoMe 22 1 0 11 Jul 2024
HAF-RM: A Hybrid Alignment Framework for Reward Model Training Shujun Liu Xiaoyu Shen Yuhang Lai Siyuan Wang Shengbin Yue Zengfeng Huang Xuanjing Huang Zhongyu Wei 13 1 0 04 Jul 2024
Orchestrating LLMs with Different Personalizations Jin Peng Zhou Katie Z Luo Jingwen Gu Jason Yuan Kilian Q. Weinberger Wen Sun 49 2 0 04 Jul 2024
DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging Tzu-Han Lin Chen An Li Hung-yi Lee Yun-Nung Chen VLM ALM 26 4 0 01 Jul 2024
BAPO: Base-Anchored Preference Optimization for Personalized Alignment in Large Language Models Gihun Lee Minchan Jeong Yujin Kim Hojung Jung Jaehoon Oh Sangmook Kim Se-Young Yun 24 1 0 30 Jun 2024
Decoding-Time Language Model Alignment with Multiple Objectives Ruizhe Shi Yifang Chen Yushi Hu Alisa Liu Hannaneh Hajishirzi Noah A. Smith Simon Du 44 30 0 27 Jun 2024
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment Thom Lake Eunsol Choi Greg Durrett 37 9 0 25 Jun 2024
Understanding the Role of User Profile in the Personalization of Large Language Models Bin Wu Zhengyan Shi Hossein A. Rahmani Varsha Ramineni Emine Yilmaz 41 5 0 22 Jun 2024
Pareto-Optimal Learning from Preferences with Hidden Context Ryan Boldi Li Ding Lee Spector S. Niekum 51 6 0 21 Jun 2024
Can LLM be a Personalized Judge? Yijiang River Dong Tiancheng Hu Nigel Collier ELM 35 34 0 17 Jun 2024
Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts Zhaoxuan Tan Zheyuan Liu Meng-Long Jiang 27 19 0 15 Jun 2024
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation Lu Li T. Zhang Zhiqi Bu Suyuchen Wang Huan He Jie Fu Yonghui Wu Jiang Bian Yong Chen Yoshua Bengio FedML MoMe 92 3 0 11 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 90 29 0 09 Jun 2024
Quantifying Misalignment Between Agents Aidan Kierans Avijit Ghosh Hananel Hazan Shiri Dori-Hacohen 25 1 0 06 Jun 2024
FusionBench: A Comprehensive Benchmark of Deep Model Fusion A. Tang Li Shen Yong Luo Han Hu Bo Du Dacheng Tao ELM MoMe VLM 34 19 0 05 Jun 2024
HYDRA: Model Factorization Framework for Black-Box LLM Personalization Yuchen Zhuang Haotian Sun Yue Yu Rushi Qiang Qifan Wang Chao Zhang Bo Dai AAML 31 14 0 05 Jun 2024
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization Yu-Min Tseng Yu-Chao Huang Teng-Yun Hsiao Yu-Ching Hsu Chao-Wei Huang Jia-Yin Foo Yun-Nung Chen LLMAG 246 63 0 03 Jun 2024
Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models Zhanhui Zhou Zhixuan Liu Jie Liu Zhichen Dong Chao Yang Yu Qiao ALM 36 20 0 29 May 2024
Aligning to Thousands of Preferences via System Message Generalization Seongyun Lee Sue Hyun Park Seungone Kim Minjoon Seo ALM 27 35 0 28 May 2024
Multi-Reference Preference Optimization for Large Language Models Hung Le Quan Tran D. Nguyen Kien Do Saloni Mittal Kelechi Ogueji Svetha Venkatesh 48 0 0 26 May 2024
Direct Preference Optimization With Unobserved Preference Heterogeneity Keertana Chidambaram Karthik Vinay Seetharaman Vasilis Syrgkanis 33 7 0 23 May 2024
Annotation-Efficient Preference Optimization for Language Model Alignment Yuu Jinnai Ukyo Honda 38 0 0 22 May 2024
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction Tingchen Fu Deng Cai Lemao Liu Shuming Shi Rui Yan MoMe 45 13 0 22 May 2024
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling Xingzhou Lou Junge Zhang Jian Xie Lifeng Liu Dong Yan Kaiqi Huang 29 11 0 21 May 2024
Hummer: Towards Limited Competitive Preference Dataset Li Jiang Yusen Wu Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo Zujie Wen Jun Zhou Xiaotie Deng 29 6 0 19 May 2024
Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing Letian Peng Jingbo Shang 40 1 0 13 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 49 163 0 02 May 2024
RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation Chanwoo Park Mingyang Liu Dingwen Kong Kaiqing Zhang Asuman Ozdaglar 23 28 0 30 Apr 2024
Stylus: Automatic Adapter Selection for Diffusion Models Michael Luo Justin Wong Brandon Trabucco Yanping Huang Joseph E. Gonzalez Zhifeng Chen Ruslan Salakhutdinov Ion Stoica DiffM 25 6 0 29 Apr 2024
DPO Meets PPO: Reinforced Token Optimization for RLHF Han Zhong Guhao Feng Guhao Feng Li Zhao Di He Jiang Bian Liwei Wang Jiang Bian Liwei Wang 47 56 0 29 Apr 2024
From Persona to Personalization: A Survey on Role-Playing Language Agents Jiangjie Chen Xintao Wang Rui Xu Siyu Yuan Yikai Zhang ... Caiyu Hu Siye Wu Scott Ren Ziquan Fu Yanghua Xiao 50 72 0 28 Apr 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 64 45 0 23 Apr 2024
AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence Minbeom Kim Hwanhee Lee Joonsuk Park Hwaran Lee Kyomin Jung 27 1 0 18 Apr 2024
Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment Yuu Jinnai Tetsuro Morimura Kaito Ariu Kenshi Abe 57 7 0 01 Apr 2024
Contextual Moral Value Alignment Through Context-Based Aggregation Pierre L. Dognin Jesus Rios Ronny Luss Inkit Padhi Matthew D Riemer Miao Liu P. Sattigeri Manish Nagireddy Kush R. Varshney Djallel Bouneffouf 36 5 0 19 Mar 2024
ALaRM: Align Language Models via Hierarchical Rewards Modeling Yuhang Lai Siyuan Wang Shujun Liu Xuanjing Huang Zhongyu Wei 16 4 0 11 Mar 2024
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment Yiju Guo Ganqu Cui Lifan Yuan Ning Ding Jiexin Wang ... Ruobing Xie Jie Zhou Yankai Lin Zhiyuan Liu Maosong Sun 36 56 0 29 Feb 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 40 70 0 28 Feb 2024
MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization Yasaman Jafari Dheeraj Mekala Rose Yu Taylor Berg-Kirkpatrick 34 7 0 18 Feb 2024