Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

30 September 2023

Papers citing "Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment"

47 / 47 papers shown

Title
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 31 0 0 05 May 2025
A Domain-Agnostic Scalable AI Safety Ensuring Framework Beomjun Kim Kangyeon Kim Sunwoo Kim Heejin Ahn 41 0 0 29 Apr 2025
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment J. Li Jian-Yu Guan Songhao Wu Wei Yu Wu Rui Yan 59 1 0 19 Mar 2025
OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses Angela Lopez-Cardona Sebastian Idesis Miguel Barreda-Ángeles Sergi Abadal Ioannis Arapakis 46 0 0 13 Mar 2025
Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Zachary Yahn Yichang Xu Ling Liu 51 8 0 01 Mar 2025
Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance Chenghua Huang Lu Wang Fangkai Yang Pu Zhao Z. Li Qingwei Lin Dongmei Zhang Saravan Rajmohan Qi Zhang OffRL 52 1 0 24 Feb 2025
Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation Y. Wang Tiansheng Huang Li Shen H. Yao Haotian Luo Rui Liu Naiqiang Tan Jiaxing Huang Dacheng Tao AAML MoMe CLL 109 1 0 30 Jan 2025
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion Yannis Flet-Berliac Nathan Grinsztajn Florian Strub Bill Wu Eugene Choi ... Arash Ahmadian Yash Chandak M. G. Azar Olivier Pietquin Matthieu Geist OffRL 62 4 0 17 Jan 2025
A Statistical Framework for Ranking LLM-Based Chatbots Siavash Ameli Siyuan Zhuang Ion Stoica Michael W. Mahoney ELM 38 1 0 24 Dec 2024
$H^3$ Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs Selim Furkan Tekin Fatih Ilhan Tiansheng Huang Sihao Hu Zachary Yahn Ling Liu MoMe 63 3 0 26 Nov 2024
TODO: Enhancing LLM Alignment with Ternary Preferences Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 33 1 0 02 Nov 2024
Generative Reward Models Dakota Mahan Duy Phung Rafael Rafailov Chase Blagden Nathan Lile Louis Castricato Jan-Philipp Fränken Chelsea Finn Alon Albalak VLM SyDa OffRL 27 24 0 02 Oct 2024
Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models Angela Lopez-Cardona Carlos Segura Alexandros Karatzoglou Sergi Abadal Ioannis Arapakis ALM 48 2 0 02 Oct 2024
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation Wei Shen Chuheng Zhang OffRL 28 6 0 11 Sep 2024
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Tiansheng Huang Gautam Bhattacharya Pratik Joshi Josh Kimball Ling Liu AAML MoMe 47 16 0 18 Aug 2024
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Tianhao Wu Weizhe Yuan O. Yu. Golovneva Jing Xu Yuandong Tian Jiantao Jiao Jason Weston Sainbayar Sukhbaatar ALM KELM LRM 44 71 0 28 Jul 2024
When Search Engine Services meet Large Language Models: Visions and Challenges Haoyi Xiong Jiang Bian Yuchen Li Xuhong Li Mengnan Du Shuaiqiang Wang Dawei Yin Sumi Helal 43 28 0 28 Jun 2024
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold Amrith Rajagopal Setlur Saurabh Garg Xinyang Geng Naman Garg Virginia Smith Aviral Kumar 35 45 0 20 Jun 2024
A Survey on Human Preference Learning for Large Language Models Ruili Jiang Kehai Chen Xuefeng Bai Zhixuan He Juntao Li Muyun Yang Tiejun Zhao Liqiang Nie Min Zhang 39 8 0 17 Jun 2024
Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas Chengyuan Deng Yiqun Duan Xin Jin Heng Chang Yijun Tian ... Kuofeng Gao Sihong He Jun Zhuang Lu Cheng Haohan Wang AILaw 38 16 0 08 Jun 2024
Group Robust Preference Optimization in Reward-free RLHF Shyam Sundhar Ramesh Yifan Hu Iason Chaimalas Viraj Mehta Pier Giuseppe Sessa Haitham Bou-Ammar Ilija Bogunovic 16 13 0 30 May 2024
Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Ling Liu 40 23 0 28 May 2024
Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization Amir Saeidi Shivanshu Verma Aswin Rrv Chitta Baral 27 5 0 26 May 2024
Online Self-Preferring Language Models Yuanzhao Zhai Zhuo Zhang Kele Xu Hanyang Peng Yue Yu Dawei Feng Cheng Yang Bo Ding Huaimin Wang 36 0 0 23 May 2024
Large Language Models for Education: A Survey Hanyi Xu Wensheng Gan Zhenlian Qi Jiayang Wu Philip S. Yu AI4Ed ELM 51 13 0 12 May 2024
Value Augmented Sampling for Language Model Alignment and Personalization Seungwook Han Idan Shenfeld Akash Srivastava Yoon Kim Pulkit Agrawal OffRL 26 23 0 10 May 2024
Active Preference Learning for Ordering Items In- and Out-of-sample Herman Bergström Emil Carlsson Devdatt Dubhashi Fredrik D. Johansson 22 0 0 05 May 2024
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks Amir Saeidi Shivanshu Verma Chitta Baral Chitta Baral ALM 30 21 0 23 Apr 2024
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari Pranjal Aggarwal Vishvak Murahari Tanmay Rajpurohit A. Kalyan Karthik Narasimhan A. Deshpande Bruno Castro da Silva 21 33 0 12 Apr 2024
Dataset Reset Policy Optimization for RLHF Jonathan D. Chang Wenhao Zhan Owen Oertell Kianté Brantley Dipendra Kumar Misra Jason D. Lee Wen Sun OffRL 19 21 0 12 Apr 2024
ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback Zhenyu Hou Yiin Niu Zhengxiao Du Xiaohan Zhang Xiao Liu ... Qinkai Zheng Minlie Huang Hongning Wang Jie Tang Yuxiao Dong ALM 20 17 0 01 Apr 2024
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards Wei Shen Xiaoying Zhang Yuanshun Yao Rui Zheng Hongyi Guo Yang Liu ALM 25 11 0 12 Mar 2024
ORPO: Monolithic Preference Optimization without Reference Model Jiwoo Hong Noah Lee James Thorne OSLM 27 54 0 12 Mar 2024
COPR: Continual Human Preference Learning via Optimal Policy Regularization Han Zhang Lin Gui Yu Lei Yuanzhao Zhai Yehong Zhang ... Hui Wang Yue Yu Kam-Fai Wong Bin Liang Ruifeng Xu CLL 29 4 0 22 Feb 2024
Vaccine: Perturbation-aware Alignment for Large Language Model Tiansheng Huang Sihao Hu Ling Liu 42 32 0 02 Feb 2024
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF Banghua Zhu Michael I. Jordan Jiantao Jiao 21 22 0 29 Jan 2024
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles Yuanzhao Zhai Han Zhang Yu Lei Yue Yu Kele Xu Dawei Feng Bo Ding Huaimin Wang AI4CE 58 31 0 30 Dec 2023
An Invitation to Deep Reinforcement Learning Bernhard Jaeger Andreas Geiger OffRL OOD 59 5 0 13 Dec 2023
The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback Nathan Lambert Roberto Calandra ALM 13 30 0 31 Oct 2023
The Expressibility of Polynomial based Attention Scheme Zhao-quan Song Guangyi Xu Junze Yin 24 5 0 30 Oct 2023
COPR: Continual Learning Human Preference through Optimal Policy Regularization Han Zhang Lin Gui Yuanzhao Zhai Hui Wang Yu Lei Ruifeng Xu CLL 27 0 0 24 Oct 2023
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models Ashutosh Baheti Ximing Lu Faeze Brahman Ronan Le Bras Maarten Sap Mark O. Riedl 20 9 0 24 May 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 203 2,232 0 22 Mar 2023
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 218 441 0 23 Aug 2022
Offline RL for Natural Language Generation with Implicit Language Q Learning Charles Burton Snell Ilya Kostrikov Yi Su Mengjiao Yang Sergey Levine OffRL 121 101 0 05 Jun 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,561 0 18 Sep 2019