Statistical Rejection Sampling Improves Preference Optimization

13 September 2023

Tianqi Liu

Yao-Min Zhao

Rishabh Joshi

Misha Khalman

Mohammad Saleh

Peter J. Liu

Jialu Liu

ArXiv PDF HTML

Papers citing "Statistical Rejection Sampling Improves Preference Optimization"

50 / 168 papers shown

Title
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul M. Chilimbi 16 0 0 13 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 62 0 0 05 May 2025
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach Jiancong Xiao Bojian Hou Zhanliang Wang Ruochen Jin Q. Long Weijie Su Li Shen 28 0 0 04 May 2025
CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design Weitao Feng Hang Zhou Jing Liao Li Cheng Wenbo Zhou 3DV 58 0 0 28 Apr 2025
ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data Haoran Gu Handing Wang Yi Mei Mengjie Zhang Yaochu Jin 27 0 0 23 Apr 2025
Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback N. Wang Bingkun Yao Jie Zhou Yuchen Hu Xi Wang Nan Guan Zhe Jiang OffRL 22 0 0 22 Apr 2025
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce Wei Xiong Jiarui Yao Yuhui Xu Bo Pang Lei Wang ... Junnan Li Nan Jiang Tong Zhang Caiming Xiong Hanze Dong OffRL LRM 33 2 0 15 Apr 2025
A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future Jialun Zhong Wei Shen Yanzeng Li Songyang Gao Hua Lu Yicheng Chen Yang Zhang Wei Zhou Jinjie Gu Lei Zou LRM 38 1 0 12 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 57 0 0 09 Apr 2025
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Kai Ye Hongyi Zhou Jin Zhu Francesco Quinzan C. Shi 20 0 0 03 Apr 2025
An Illusion of Progress? Assessing the Current State of Web Agents Tianci Xue Weijian Qi Tianneng Shi Chan Hee Song Boyu Gou D. Song Huan Sun Yu Su LLMAG ELM 82 4 1 02 Apr 2025
POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation Lanyun Zhu Tianrun Chen Qianxiong Xu Xuanyi Liu Deyi Ji Haiyang Wu De Wen Soh J. Liu VLM LRM 44 0 0 01 Apr 2025
InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment Y. Lu Qichao Wang H. Cao Xierui Wang Xiaoyin Xu Min Zhang 56 0 0 24 Mar 2025
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization Zefeng Zhang Hengzhu Tang Jiawei Sheng Zhenyu Zhang Yiming Ren Zhenyang Li Dawei Yin Duohe Ma Tingwen Liu 43 0 0 23 Mar 2025
TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning Sheng Wang Pengan Chen Jingqi Zhou Qintong Li Jingwei Dong Jiahui Gao Boyang Xue Jiyue Jiang Lingpeng Kong Chuan Wu SyDa 61 0 0 21 Mar 2025
Regulatory DNA sequence Design with Reinforcement Learning Zhao-Qing Yang Bing-Huang Su Chuan Cao Ji-Rong Wen 51 0 0 11 Mar 2025
UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality Zelei Cheng Xin-Qiang Cai Yuting Tang Pushi Zhang Boming Yang Xinyu Xing 39 0 0 10 Mar 2025
Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models Alessio Galatolo Zhenbang Dai Katie Winkle Meriem Beloucif 47 0 0 05 Mar 2025
Self-rewarding correction for mathematical reasoning Wei Xiong Hanning Zhang Chenlu Ye Lichang Chen Nan Jiang Tong Zhang ReLM KELM LRM 64 9 0 26 Feb 2025
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment Tianze Wang Dongnan Gui Yifan Hu Shuhang Lin Linjun Zhang 31 0 0 25 Feb 2025
Debt Collection Negotiations with Large Language Models: An Evaluation System and Optimizing Decision Making with Multi-Agent Xiaofeng Wang Z. Zhang Jinguang Zheng Yiming Ai Rui Wang 37 1 0 25 Feb 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He X. Wang 40 0 0 25 Feb 2025
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment Sizhe Wang Yongqi Tong Hengyuan Zhang Dawei Li Xin Zhang Tianlong Chen 85 5 0 21 Feb 2025
IPO: Iterative Preference Optimization for Text-to-Video Generation Xiaomeng Yang Zhiyu Tan Xuecheng Nie VGen 101 1 0 04 Feb 2025
LiPO: Listwise Preference Optimization through Learning-to-Rank Tianqi Liu Zhen Qin Junru Wu Jiaming Shen Misha Khalman ... Mohammad Saleh Simon Baumgartner Jialu Liu Peter J. Liu Xuanhui Wang 133 47 0 28 Jan 2025
Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning Nirav Diwan Tolga Ergen Dongsub Shim Honglak Lee 29 0 0 28 Jan 2025
Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model Yueqin Yin Shentao Yang Yujia Xie Ziyi Yang Yuting Sun Hany Awadalla Weizhu Chen Mingyuan Zhou 48 0 0 07 Jan 2025
Geometric-Averaged Preference Optimization for Soft Preference Labels Hiroki Furuta Kuang-Huei Lee Shixiang Shane Gu Y. Matsuo Aleksandra Faust Heiga Zen Izzeddin Gur 50 6 0 31 Dec 2024
The Superalignment of Superhuman Intelligence with Large Language Models Minlie Huang Yingkang Wang Shiyao Cui Pei Ke J. Tang 103 1 0 15 Dec 2024
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Avinandan Bose Zhihan Xiong Aadirupa Saha S. Du Maryam Fazel 66 1 0 13 Dec 2024
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang J. Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 121 6 0 05 Dec 2024
Harnessing Preference Optimisation in Protein LMs for Hit Maturation in Cell Therapy Katarzyna Janocha Annabel Ling Alice Godson Yulia Lampi Simon Bornschein Nils Y. Hammerla 67 2 0 02 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 93 2 0 01 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 108 61 0 25 Nov 2024
Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models Hao Yi Qingyang Li Y. Hu Fuzheng Zhang Di Zhang Yong Liu VGen 67 0 0 25 Nov 2024
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs Zhihan Liu Shenao Zhang Yongfei Liu Boyi Liu Yingxiang Yang Zhaoran Wang 111 2 0 20 Nov 2024
Reward Modeling with Ordinal Feedback: Wisdom of the Crowd Shang Liu Yu Pan Guanting Chen Xiaocheng Li 75 2 0 19 Nov 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 55 45 1 15 Nov 2024
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset Khaoula Chehbouni Jonathan Colaço-Carr Yash More Jackie CK Cheung G. Farnadi 71 0 0 12 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 54 3 0 07 Nov 2024
Active Preference-based Learning for Multi-dimensional Personalization Minhyeon Oh Seungjoon Lee Jungseul Ok 26 1 0 01 Nov 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 45 3 0 29 Oct 2024
LongReward: Improving Long-context Large Language Models with AI Feedback J. Zhang Zhongni Hou Xin Lv S. Cao Zhenyu Hou Yilin Niu Lei Hou Yuxiao Dong Ling Feng Juanzi Li OffRL LRM 30 7 0 28 Oct 2024
Fast Best-of-N Decoding via Speculative Rejection Hanshi Sun Momin Haider Ruiqi Zhang Huitao Yang Jiahao Qiu Ming Yin Mengdi Wang Peter L. Bartlett Andrea Zanette BDL 40 26 0 26 Oct 2024
Keep Guessing? When Considering Inference Scaling, Mind the Baselines G. Yona Or Honovich Omer Levy Roee Aharoni UQLM LRM 33 0 0 20 Oct 2024
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling Jiahao Qiu Yifu Lu Yifan Zeng Jiacheng Guo Jiayi Geng Huazheng Wang Kaixuan Huang Yue Wu Mengdi Wang 34 22 0 18 Oct 2024
Optimizing Preference Alignment with Differentiable NDCG Ranking Jiacong Zhou Xianyun Wang Jun Yu 20 2 0 17 Oct 2024
Understanding Likelihood Over-optimisation in Direct Alignment Algorithms Zhengyan Shi Sander Land Acyr F. Locatelli Matthieu Geist Max Bartolo 46 3 0 15 Oct 2024
How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective Teng Xiao Mingxiao Li Yige Yuan Huaisheng Zhu Chao Cui V. Honavar ALM 26 7 0 14 Oct 2024
Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps Han Wang Yilin Zhao Dian Li Xiaohan Wang Gang Liu Xuguang Lan H. Wang LRM 36 1 0 14 Oct 2024