Statistical Rejection Sampling Improves Preference Optimization

13 September 2023

Tianqi Liu

Yao-Min Zhao

Rishabh Joshi

Misha Khalman

Mohammad Saleh

Peter J. Liu

Jialu Liu

ArXiv PDF HTML

Papers citing "Statistical Rejection Sampling Improves Preference Optimization"

50 / 168 papers shown

Title
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both Abhijnan Nath Changsoo Jung Ethan Seefried Nikhil Krishnaswamy 54 1 0 11 Oct 2024
MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans? Guanzhen Li Yuxi Xie Min-Yen Kan VLM 27 2 0 06 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 54 3 0 06 Oct 2024
RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization Hanyang Zhao Genta Indra Winata Anirban Das Shi-Xiong Zhang D. Yao Wenpin Tang Sambit Sahu 54 4 0 05 Oct 2024
Learning Code Preference via Synthetic Evolution Jiawei Liu Thanh Nguyen Mingyue Shang Hantian Ding Xiaopeng Li Yu Yu Varun Kumar Zijian Wang SyDa ALM AAML 23 3 0 04 Oct 2024
Strong Preferences Affect the Robustness of Preference Models and Value Alignment Ziwei Xu Mohan Kankanhalli AAML 19 0 0 03 Oct 2024
Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models Angela Lopez-Cardona Carlos Segura Alexandros Karatzoglou Sergi Abadal Ioannis Arapakis ALM 48 2 0 02 Oct 2024
FlashMask: Efficient and Rich Mask Extension of FlashAttention Guoxia Wang Jinle Zeng Xiyuan Xiao Siming Wu Jiabin Yang Lujing Zheng Zeyu Chen Jiang Bian Dianhai Yu Haifeng Wang 43 2 0 02 Oct 2024
The Crucial Role of Samplers in Online Direct Preference Optimization Ruizhe Shi Runlong Zhou Simon S. Du 53 7 0 29 Sep 2024
Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review Emma Croxford Yanjun Gao Nicholas Pellegrino Karen K. Wong Graham Wills Elliot First Frank J. Liao Cherodeep Goswami Brian Patterson Majid Afshar HILM ELM LM&MA 32 1 0 26 Sep 2024
Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness Jian Li Haojing Huang Yujia Zhang Pengfei Xu Xi Chen Rui Song Lida Shi Jingwen Wang Hao Xu 21 0 0 26 Sep 2024
Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization Ruijie Xu Zhihan Liu Yongfei Liu Shipeng Yan Zhaoran Wang Zhi-Li Zhang Xuming He ALM 28 1 0 26 Sep 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 50 12 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 10 0 18 Sep 2024
CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration Jiahui Gao Renjie Pi Tianyang Han Han Wu Lanqing Hong Lingpeng Kong Xin Jiang Zhenguo Li 34 5 0 17 Sep 2024
Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization Jianing Wang Yang Zhou Xiaocheng Zhang Mengjiao Bao Peng Yan 25 1 0 17 Sep 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 84 5 0 13 Sep 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 37 11 0 11 Sep 2024
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation Wei Shen Chuheng Zhang OffRL 30 6 0 11 Sep 2024
On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization Yong Lin Skyler Seto Maartje ter Hoeve Katherine Metcalf B. Theobald Xuan Wang Yizhe Zhang Chen Huang Tong Zhang 29 12 0 05 Sep 2024
Towards a Unified View of Preference Learning for Large Language Models: A Survey Bofei Gao Feifan Song Yibo Miao Zefan Cai Z. Yang ... Houfeng Wang Zhifang Sui Peiyi Wang Baobao Chang Baobao Chang 41 11 0 04 Sep 2024
Critique-out-Loud Reward Models Zachary Ankner Mansheej Paul Brandon Cui Jonathan D. Chang Prithviraj Ammanabrolu ALM LRM 25 25 0 21 Aug 2024
CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs Yassine Ouali Adrian Bulat Brais Martínez Georgios Tzimiropoulos VLM MLLM 20 18 0 19 Aug 2024
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization Yuxin Jiang Bo Huang Yufei Wang Xingshan Zeng Liangyou Li Yasheng Wang Xin Jiang Lifeng Shang Ruiming Tang Wei Wang 40 5 0 14 Aug 2024
On the Generalization of Preference Learning with DPO Shawn Im Yixuan Li 33 1 0 06 Aug 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 27 4 0 26 Jul 2024
Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation Jiaming Shen Ran Xu Yennie Jun Zhen Qin Tianqi Liu Carl Yang Yi Liang Simon Baumgartner Michael Bendersky SyDa 53 4 0 22 Jul 2024
Clinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization Md Sultan al Nahian R. Kavuluru MedIm AI4CE 31 0 0 19 Jul 2024
Hindsight Preference Learning for Offline Preference-based Reinforcement Learning Chen-Xiao Gao Shengjun Fang Chenjun Xiao Yang Yu Zongzhang Zhang OffRL 25 0 0 05 Jul 2024
Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment Janghwan Lee Seongmin Park S. Hong Minsoo Kim Du-Seong Chang Jungwook Choi 27 4 0 03 Jul 2024
Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning Zimu Lu Aojun Zhou Ke Wang Houxing Ren Weikang Shi Junting Pan Mingjie Zhan Hongsheng Li LRM 24 22 0 30 Jun 2024
BAPO: Base-Anchored Preference Optimization for Personalized Alignment in Large Language Models Gihun Lee Minchan Jeong Yujin Kim Hojung Jung Jaehoon Oh Sangmook Kim Se-Young Yun 24 1 0 30 Jun 2024
ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation Peiyang Wu Nan Guo Xiao Xiao Wenming Li Xiaochun Ye Dongrui Fan 30 0 0 28 Jun 2024
Aligning Teacher with Student Preferences for Tailored Training Data Generation Yantao Liu Zhao Zhang Zijun Yao S. Cao Lei Hou Juanzi Li 42 1 0 27 Jun 2024
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation Guanting Dong Yutao Zhu Chenghao Zhang Zechen Wang Zhicheng Dou Ji-Rong Wen RALM 42 10 0 26 Jun 2024
Aligning Diffusion Models with Noise-Conditioned Perception Alexander Gambashidze Anton Kulikov Yuriy Sosnin Ilya Makarov 35 5 0 25 Jun 2024
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models Sean Welleck Amanda Bertsch Matthew Finlayson Hailey Schoelkopf Alex Xie Graham Neubig Ilia Kulikov Zaid Harchaoui 33 45 0 24 Jun 2024
On the Transformations across Reward Model, Parameter Update, and In-Context Prompt Deng Cai Huayang Li Tingchen Fu Siheng Li Weiwen Xu ... Leyang Cui Yan Wang Lemao Liu Taro Watanabe Shuming Shi KELM 26 2 0 24 Jun 2024
Cascade Reward Sampling for Efficient Decoding-Time Alignment Bolian Li Yifan Wang A. Grama Ruqi Zhang Ruqi Zhang AI4TS 47 9 0 24 Jun 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 44 132 0 18 Jun 2024
Low-Redundant Optimization for Large Language Model Alignment Zhipeng Chen Kun Zhou Wayne Xin Zhao Jingyuan Wang Ji-Rong Wen 29 2 0 18 Jun 2024
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models Hongbang Yuan Yubo Chen Pengfei Cao Zhuoran Jin Kang Liu Jun Zhao 28 0 0 18 Jun 2024
Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency Leonidas Gee Milan Gritta Gerasimos Lampouras Ignacio Iacobacci 21 10 0 18 Jun 2024
Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models Fangzhi Xu Qiushi Sun Kanzhi Cheng J. Liu Yu Qiao Zhiyong Wu LLMAG 29 5 0 17 Jun 2024
A Survey on Human Preference Learning for Large Language Models Ruili Jiang Kehai Chen Xuefeng Bai Zhixuan He Juntao Li Muyun Yang Tiejun Zhao Liqiang Nie Min Zhang 39 8 0 17 Jun 2024
Self-Evolution Fine-Tuning for Policy Optimization Ruijun Chen Jiehao Liang Shiping Gao Fanqi Wan Xiaojun Quan 35 0 0 16 Jun 2024
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning Jifan Zhang Lalit P. Jain Yang Guo Jiayi Chen Kuan Lok Zhou ... Scott Sievert Timothy Rogers Kevin Jamieson Robert Mankoff Robert Nowak 29 5 0 15 Jun 2024
On Softmax Direct Preference Optimization for Recommendation Yuxin Chen Junfei Tan An Zhang Zhengyi Yang Leheng Sheng Enzhi Zhang Xiang Wang Tat-Seng Chua 29 23 0 13 Jun 2024
3D-Properties: Identifying Challenges in DPO and Charting a Path Forward Yuzi Yan Yibo Miao J. Li Yipin Zhang Jian Xie Zhijie Deng Dong Yan 39 11 0 11 Jun 2024
Aligning Large Language Models with Representation Editing: A Control Perspective Lingkai Kong Haorui Wang Wenhao Mu Yuanqi Du Yuchen Zhuang Yifei Zhou Yue Song Rongzhi Zhang Kai Wang Chao Zhang 18 21 0 10 Jun 2024