Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

7 June 2023

Papers citing "Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards"

42 / 42 papers shown

Title
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 31 0 0 05 May 2025
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors Ren-Wei Liang Chin-Ting Hsu Chan-Hung Yu Saransh Agrawal Shih-Cheng Huang Shang-Tse Chen Kuan-Hao Huang Shao-Hua Sun 76 0 0 27 Apr 2025
HF4Rec: Human-Like Feedback-Driven Optimization Framework for Explainable Recommendation Jiakai Tang Jingsen Zhang Zihang Tian Xueyang Feng Lei Wang Xu Chen OffRL 32 0 0 19 Apr 2025
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications Jian-Yu Guan J. Wu J. Li Chuanqi Cheng Wei Yu Wu LM&MA 69 0 0 21 Mar 2025
Drift: Decoding-time Personalized Alignments with Implicit User Preferences Minbeom Kim Kang-il Lee Seongho Joo Hwaran Lee Thibaut Thonet Kyomin Jung AI4TS 95 1 0 20 Feb 2025
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts Helia Hashemi J. Eisner Corby Rosset Benjamin Van Durme Chris Kedzie 61 1 0 03 Jan 2025
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 68 0 0 12 Nov 2024
L3Ms -- Lagrange Large Language Models Guneet S. Dhillon Xingjian Shi Yee Whye Teh Alex Smola 30 0 0 28 Oct 2024
How to Find the Exact Pareto Front for Multi-Objective MDPs? Yining Li Peizhong Ju Ness B. Shroff 32 0 0 21 Oct 2024
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both Abhijnan Nath Changsoo Jung Ethan Seefried Nikhil Krishnaswamy 43 1 0 11 Oct 2024
Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements Jingyu Zhang Ahmed Elgohary Ahmed Magooda Daniel Khashabi Benjamin Van Durme 38 2 0 11 Oct 2024
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment Yuancheng Xu Udari Madhushani Sehwag Alec Koppel Sicheng Zhu Bang An Furong Huang Sumitra Ganesh 40 5 0 10 Oct 2024
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions Yekun Chai Haoran Sun Huang Fang Shuohuan Wang Yu Sun Hua-Hong Wu 34 1 0 03 Oct 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 37 11 0 11 Sep 2024
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models Wenxuan Zhang Philip H. S. Torr Mohamed Elhoseiny Adel Bibi 38 9 0 27 Aug 2024
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation Katherine M. Collins Najoung Kim Yonatan Bitton Verena Rieser Shayegan Omidshafiei ... Gang Li Adrian Weller Junfeng He Deepak Ramachandran Krishnamurthy Dvijotham EGVM 41 3 0 24 Jun 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 47 13 0 24 Jun 2024
Pareto-Optimal Learning from Preferences with Hidden Context Ryan Boldi Li Ding Lee Spector S. Niekum 42 6 0 21 Jun 2024
PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences Daiwei Chen Yi Chen Aniket Rege Ramya Korlakai Vinayak 32 16 0 12 Jun 2024
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation Lu Li T. Zhang Zhiqi Bu Suyuchen Wang Huan He Jie Fu Yonghui Wu Jiang Bian Yong Chen Yoshua Bengio FedML MoMe 89 3 0 11 Jun 2024
Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oracles Jiesong Lian Yucong Huang Chengdong Ma Mingzhi Wang Ying Wen Long Hu Yixue Hao 42 0 0 31 May 2024
One-Shot Safety Alignment for Large Language Models via Optimal Dualization Xinmeng Huang Shuo Li Edgar Dobriban Osbert Bastani Hamed Hassani Dongsheng Ding 30 3 0 29 May 2024
Robust Preference Optimization through Reward Model Distillation Adam Fisch Jacob Eisenstein Vicky Zayats Alekh Agarwal Ahmad Beirami Chirag Nagpal Peter Shaw Jonathan Berant 68 20 0 29 May 2024
Multi-Reference Preference Optimization for Large Language Models Hung Le Quan Tran D. Nguyen Kien Do Saloni Mittal Kelechi Ogueji Svetha Venkatesh 48 0 0 26 May 2024
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling Xingzhou Lou Junge Zhang Jian Xie Lifeng Liu Dong Yan Kaiqi Huang 26 11 0 21 May 2024
FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback Liqiang Jing Xinya Du 68 17 0 07 Apr 2024
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data Jingyu Zhang Marc Marone Tianjian Li Benjamin Van Durme Daniel Khashabi 70 9 0 05 Apr 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 29 70 0 28 Feb 2024
$π$ -Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation Chengyue Wu Teng Wang Yixiao Ge Zeyu Lu Rui-Zhi Zhou Ying Shan Ping Luo MoMe 68 35 0 27 Apr 2023
Foundation Models for Decision Making: Problems, Methods, and Opportunities Sherry Yang Ofir Nachum Yilun Du Jason W. Wei Pieter Abbeel Dale Schuurmans LM&Ro OffRL LRM AI4CE 87 148 0 07 Mar 2023
Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition Steven Vander Eeckt Hugo Van hamme CLL MoMe 56 14 0 27 Oct 2022
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 222 495 0 28 Sep 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 239 313 0 11 Sep 2022
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 216 327 0 23 Aug 2022
Linear Connectivity Reveals Generalization Strategies Jeevesh Juneja Rachit Bansal Kyunghyun Cho João Sedoc Naomi Saphra 232 45 0 24 May 2022
Diverse Weight Averaging for Out-of-Distribution Generalization Alexandre Ramé Matthieu Kirchmeyer Thibaud Rahier A. Rakotomamonjy Patrick Gallinari Matthieu Cord OOD 186 128 0 19 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble Gaon An Seungyong Moon Jang-Hyun Kim Hyun Oh Song OffRL 92 261 0 04 Oct 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 283 5,723 0 29 Apr 2021
Multi-expert learning of adaptive legged locomotion Chuanyu Yang Kai Yuan Qiuguo Zhu Wanming Yu Zhibin Li 94 147 0 10 Dec 2020
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 273 1,561 0 18 Sep 2019
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles Balaji Lakshminarayanan Alexander Pritzel Charles Blundell UQCV BDL 268 5,635 0 05 Dec 2016