Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

3 October 2022

Rajkumar Ramamurthy

Prithviraj Ammanabrolu

Yejin Choi

Papers citing "Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization"

50 / 202 papers shown

Title
RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs Xuan Chen Yuzhou Nie Lu Yan Yunshu Mao Wenbo Guo Xiangyu Zhang 16 6 0 13 Jun 2024
Prompt-Based Length Controlled Generation with Multiple Control Types Renlong Jie Xiaojun Meng Lifeng Shang Xin Jiang Qun Liu 19 6 0 12 Jun 2024
Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing Biqing Qi Pengfei Li Fangyuan Li Junqi Gao Kaiyan Zhang Bowen Zhou 29 11 0 08 Jun 2024
Aligning Large Language Models via Fine-grained Supervision Dehong Xu Liang Qiu Minseok Kim Faisal Ladhak Jaeyoung Do 19 2 0 04 Jun 2024
Using RL to Identify Divisive Perspectives Improves LLMs Abilities to Identify Communities on Social Media Nikhil Mehta Dan Goldwasser 16 0 0 03 Jun 2024
Preference Learning Algorithms Do Not Learn Preference Rankings Angelica Chen Sadhika Malladi Lily H. Zhang Xinyi Chen Qiuyi Zhang Rajesh Ranganath Kyunghyun Cho 20 22 0 29 May 2024
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications Dingkang Yang Jinjie Wei Dongling Xiao Shunli Wang Tong Wu ... Yue Jiang Qingyao Xu Ke Li Peng Zhai Lihua Zhang LM&MA 30 6 0 29 May 2024
Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment Jiaxiang Li Siliang Zeng Hoi-To Wai Chenliang Li Alfredo García Mingyi Hong 55 15 0 28 May 2024
BWArea Model: Learning World Model, Inverse Dynamics, and Policy for Controllable Language Generation Chengxing Jia Pengyuan Wang Ziniu Li Yi-Chen Li Zhilong Zhang Nan Tang Yang Yu OffRL 25 1 0 27 May 2024
InstructPatentGPT: Training patent language models to follow instructions with human feedback Jieh-Sheng Lee ALM 36 5 0 25 May 2024
RaFe: Ranking Feedback Improves Query Rewriting for RAG Shengyu Mao Yong-jia Jiang Boli Chen Xiao Li Peng Wang Xinyu Wang Pengjun Xie Fei Huang Huajun Chen Ningyu Zhang RALM 26 6 0 23 May 2024
Online Self-Preferring Language Models Yuanzhao Zhai Zhuo Zhang Kele Xu Hanyang Peng Yue Yu Dawei Feng Cheng Yang Bo Ding Huaimin Wang 22 0 0 23 May 2024
Hummer: Towards Limited Competitive Preference Dataset Li Jiang Yusen Wu Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo Zujie Wen Jun Zhou Xiaotie Deng 18 6 0 19 May 2024
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning Yuexiang Zhai Hao Bai Zipeng Lin Jiayi Pan Shengbang Tong ... Alane Suhr Saining Xie Yann LeCun Yi-An Ma Sergey Levine LLMAG LRM 27 54 0 16 May 2024
Value Augmented Sampling for Language Model Alignment and Personalization Seungwook Han Idan Shenfeld Akash Srivastava Yoon Kim Pulkit Agrawal OffRL 21 23 0 10 May 2024
Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation JoonHo Lee Jae Oh Woo Juree Seok Parisa Hassanzadeh Wooseok Jang ... Hankyu Moon Wenjun Hu Yeong-Dae Kwon Taehee Lee Seungjai Min 40 2 0 10 May 2024
Position: Leverage Foundational Models for Black-Box Optimization Xingyou Song Yingtao Tian Robert Tjarko Lange Chansoo Lee Yujin Tang Yutian Chen 32 3 0 06 May 2024
ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces Libing Yang Yang Li Long Chen 18 3 0 05 May 2024
WildChat: 1M ChatGPT Interaction Logs in the Wild Wenting Zhao Xiang Ren Jack Hessel Claire Cardie Yejin Choi Yuntian Deng 40 171 0 02 May 2024
Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning Mathieu Rita Florian Strub Rahma Chaabouni Paul Michel Emmanuel Dupoux Olivier Pietquin 34 2 0 30 Apr 2024
When to Trust LLMs: Aligning Confidence with Response Quality Shuchang Tao Liuyi Yao Hanxing Ding Yuexiang Xie Qi Cao Fei Sun Jinyang Gao Huawei Shen Bolin Ding 22 0 0 26 Apr 2024
REBEL: Reinforcement Learning via Regressing Relative Rewards Zhaolin Gao Jonathan D. Chang Wenhao Zhan Owen Oertell Gokul Swamy Kianté Brantley Thorsten Joachims J. Andrew Bagnell Jason D. Lee Wen Sun OffRL 23 31 0 25 Apr 2024
Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing Ye Tian Baolin Peng Linfeng Song Lifeng Jin Dian Yu Haitao Mi Dong Yu LRM ReLM 33 62 0 18 Apr 2024
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study Shusheng Xu Wei Fu Jiaxuan Gao Wenjie Ye Weiling Liu Zhiyu Mei Guangju Wang Chao Yu Yi Wu 19 131 0 16 Apr 2024
Self-playing Adversarial Language Game Enhances LLM Reasoning Pengyu Cheng Tianhao Hu Han Xu Zhisong Zhang Yong Dai Lei Han Nan Du Nan Du Xiaolong Li SyDa LRM ReLM 77 28 0 16 Apr 2024
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari Pranjal Aggarwal Vishvak Murahari Tanmay Rajpurohit A. Kalyan Karthik Narasimhan A. Deshpande Bruno Castro da Silva 21 33 0 12 Apr 2024
Dataset Reset Policy Optimization for RLHF Jonathan D. Chang Wenhao Zhan Owen Oertell Kianté Brantley Dipendra Kumar Misra Jason D. Lee Wen Sun OffRL 14 21 0 12 Apr 2024
Aligning Diffusion Models by Optimizing Human Utility Shufan Li Konstantinos Kallidromitis Akash Gokul Yusuke Kato Kazuki Kozuka 97 27 0 06 Apr 2024
Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs Shu Yang Jiayuan Su Han Jiang Mengdi Li Keyuan Cheng Muhammad Asif Ali Lijie Hu Di Wang 16 5 0 30 Mar 2024
If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions Reza Esfandiarpoor Cristina Menghini Stephen H. Bach CoGe VLM 16 8 0 25 Mar 2024
Qibo: A Large Language Model for Traditional Chinese Medicine Heyi Zhang Xin Wang Zhaopeng Meng Zhe Chen Pengwei Zhuang Yongzhe Jia Dawei Xu Wenbin Guo LM&MA 18 6 0 24 Mar 2024
Locating and Mitigating Gender Bias in Large Language Models Yuchen Cai Ding Cao Rongxi Guo Yaqin Wen Guiquan Liu Enhong Chen 19 5 0 21 Mar 2024
Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection Kyungjae Lee Dasol Hwang Sunghyun Park Youngsoo Jang Moontae Lee 27 8 0 21 Mar 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James Validad Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 62 210 0 20 Mar 2024
Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation Do June Min Verónica Pérez-Rosas Kenneth Resnicow Rada Mihalcea OffRL 33 2 0 20 Mar 2024
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization Renjie Pi Tianyang Han Wei Xiong Jipeng Zhang Runtao Liu Rui Pan Tong Zhang MLLM 20 27 0 13 Mar 2024
Teaching Large Language Models to Reason with Reinforcement Learning Alex Havrilla Yuqing Du Sharath Chandra Raparthy Christoforos Nalmpantis Jane Dwivedi-Yu Maksym Zhuravinskyi Eric Hambro Sainbayar Sukhbaatar Roberta Raileanu ReLM LRM 29 67 0 07 Mar 2024
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy Yu Zhu Chuxiong Sun Wenfei Yang Wenqiang Wei Bo Tang ... Zhiyu Li Shifeng Zhang Feiyu Xiong Jie Hu Mingchuan Yang 21 3 0 07 Mar 2024
Improving Socratic Question Generation using Data Augmentation and Preference Optimization Nischal Ashok Kumar Andrew S. Lan 14 8 0 01 Mar 2024
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL Yifei Zhou Andrea Zanette Jiayi Pan Sergey Levine Aviral Kumar 59 47 0 29 Feb 2024
Generalizing Reward Modeling for Out-of-Distribution Preference Learning Chen Jia 12 2 0 22 Feb 2024
Q-Probe: A Lightweight Approach to Reward Maximization for Language Models Kenneth Li Samy Jelassi Hugh Zhang Sham Kakade Martin Wattenberg David Brandfonbrener 19 9 0 22 Feb 2024
COPR: Continual Human Preference Learning via Optimal Policy Regularization Han Zhang Lin Gui Yu Lei Yuanzhao Zhai Yehong Zhang ... Hui Wang Yue Yu Kam-Fai Wong Bin Liang Ruifeng Xu CLL 29 4 0 22 Feb 2024
Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models Younghun Lee Sungchul Kim Tong Yu Ryan A. Rossi Xiang Chen LLMAG 28 0 0 22 Feb 2024
GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations Jinhao Duan Renming Zhang James Diffenderfer B. Kailkhura Lichao Sun Elias Stengel-Eskin Mohit Bansal Tianlong Chen Kaidi Xu ELM LRM 21 55 0 19 Feb 2024
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition Zhaorun Chen Zhuokai Zhao Zhihong Zhu Ruiqi Zhang Xiang Li Bhiksha Raj Huaxiu Yao LRM 19 17 0 18 Feb 2024
A Dense Reward View on Aligning Text-to-Image Diffusion with Preference Shentao Yang Tianqi Chen Mingyuan Zhou EGVM 18 22 0 13 Feb 2024
Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts Yueqin Yin Zhendong Wang Yi Gu Hai Huang Weizhu Chen Mingyuan Zhou 8 10 0 12 Feb 2024
Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement Muning Wen Junwei Liao Cheng Deng Jun Wang Weinan Zhang Ying Wen 13 1 0 09 Feb 2024
Decoding-time Realignment of Language Models Tianlin Liu Shangmin Guo Leonardo Bianco Daniele Calandriello Quentin Berthet Felipe Llinares-López Jessica Hoffmann Lucas Dixon Michal Valko Mathieu Blondel AI4CE 45 11 0 05 Feb 2024