BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

North American Chapter of the Association for Computational Linguistics (NAACL), 2024

21 February 2025

Papers citing "BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment"

39 / 39 papers shown

Title
Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation Hengyuan Zhang Shiping Yang Xiao Liang Chenming Shang Yuxuan Jiang ... Jing Xiong Hayden Kwok-Hay So Ruobing Xie Angel X. Chang Ngai Wong 52 0 0 13 Oct 2025
Who's Your Judge? On the Detectability of LLM-Generated Judgments Dawei Li Zhen Tan Chengshuai Zhao Bohan Jiang Baixiang Huang Pingchuan Ma Abdullah Alnaibari Kai Shu Huan Liu 97 0 0 29 Sep 2025
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors Hengyuan Zhang Xinrong Chen Yingmin Qiu Xiao Liang Ziyue Li Guanyu Wang Weiping Li Tong Mo Wenyue Li Hayden Kwok-Hay So MoE ALM 145 2 0 17 Jun 2025
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review Yuan Chang Ziyue Li Hengyuan Zhang Yuanbo Kong Yanru Wu Hayden Kwok-Hay So Zhijiang Guo Liya Zhu Ngai Wong 193 8 0 09 Jun 2025
The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation Ruichen Zhang Rana Muhammad Shahroz Khan Zhen Tan Dawei Li Song Wang Tianlong Chen LRM 172 1 0 24 May 2025
Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary Licheng Pan Yongqi Tong Xin Zhang Xiaolu Zhang Jun Zhou Zhixuan Chu 186 1 0 23 May 2025
Quantifying the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data Shiping Yang Jie Wu Wenbiao Ding Ning Wu Shining Liang Ming Gong Hengyuan Zhang Dongmei Zhang AAML 248 10 0 07 Mar 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Qingsong Wen Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xiang Wang 125 2 0 25 Feb 2025
Preference Leakage: A Contamination Problem in LLM-as-a-judge Dawei Li Renliang Sun Yue Huang Ming Zhong Bohan Jiang Jiawei Han Wei Wei Wei Wang Huan Liu 434 60 0 03 Feb 2025
Assessing the Impact of Conspiracy Theories Using Large Language Models Bohan Jiang Dawei Li Zhen Tan Xinyi Zhou Ashwin Rao Kristina Lerman H. Bernard Huan Liu 297 4 0 09 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 708 210 0 25 Nov 2024
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning Tianduo Wang Shichen Li Wei Lu LRM AI4CE 145 32 1 25 Jul 2024
SimPO: Simple Preference Optimization with a Reference-Free RewardNeural Information Processing Systems (NeurIPS), 2024 Yu Meng Mengzhou Xia Danqi Chen 373 713 0 23 May 2024
DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature Dawei Li Shu Yang Zhen Tan Jae Young Baik Sunkwon Yun ... D. Duong-Tran Ying Ding Huan Liu Li Shen Tianlong Chen 221 59 0 08 May 2024
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks Amir Saeidi Shivanshu Verma Chitta Baral Chitta Baral ALM 246 29 0 23 Apr 2024
Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment Feifan Song Bowen Yu Hao Lang Haiyang Yu Fei Huang Houfeng Wang Yongbin Li ALM 140 22 0 17 Mar 2024
Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked PreferencesConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 Pulkit Pattnaik Rishabh Maheshwary Kelechi Ogueji Vikas Yadav Sathwik Tejaswi Madhusudhan 134 32 0 12 Mar 2024
Large Language Models for Data Annotation: A Survey Zhen Tan Dawei Li Song Wang Alimohammad Beigi Bohan Jiang Amrita Bhattacharjee Mansooreh Karami Wenlin Yao Lu Cheng Huan Liu SyDa 260 166 0 21 Feb 2024
LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia Sadhika Malladi Suchin Gururangan Sanjeev Arora Danqi Chen 372 349 0 06 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 634 770 0 02 Feb 2024
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 285 305 0 25 Dec 2023
Safe RLHF: Safe Reinforcement Learning from Human Feedback Josef Dai Xuehai Pan Ruiyang Sun Jiaming Ji Xinbo Xu Mickel Liu Yizhou Wang Yaodong Yang 239 486 0 19 Oct 2023
Statistical Rejection Sampling Improves Preference OptimizationInternational Conference on Learning Representations (ICLR), 2023 Tianqi Liu Yao-Min Zhao Rishabh Joshi Misha Khalman Mohammad Saleh Peter J. Liu Jialu Liu 226 304 0 13 Sep 2023
InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4 Lai Wei Zihao Jiang Weiran Huang Lichao Sun VLM MLLM 211 73 0 23 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 1.5K 14,494 0 18 Jul 2023
Understanding In-Context Learning via Supportive Pretraining DataAnnual Meeting of the Association for Computational Linguistics (ACL), 2023 Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz Tianlu Wang AIMat 198 45 0 26 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaNeural Information Processing Systems (NeurIPS), 2023 Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 1.7K 6,018 0 09 Jun 2023
Benchmarking Foundation Models with Language-Model-as-an-ExaminerNeural Information Processing Systems (NeurIPS), 2023 Yushi Bai Jiahao Ying Yixin Cao Xin Lv Yuze He ... Yijia Xiao Haozhe Lyu Jiayin Zhang Juanzi Li Lei Hou ALM ELM 198 183 0 07 Jun 2023
Fine-Tuning Language Models with Advantage-Induced Policy Alignment Banghua Zhu Hiteshi Sharma Felipe Vieira Frujeri Shi Dong Chenguang Zhu Michael I. Jordan Jiantao Jiao OSLM 185 46 0 04 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward ModelNeural Information Processing Systems (NeurIPS), 2023 Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 640 6,059 0 29 May 2023
LIMA: Less Is More for AlignmentNeural Information Processing Systems (NeurIPS), 2023 Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun ... Susan Zhang Gargi Ghosh M. Lewis Luke Zettlemoyer Omer Levy ALM 316 1,072 0 18 May 2023
Human-like Summarization Evaluation with ChatGPT Mingqi Gao Jie Ruan Renliang Sun Xunjian Yin Shiping Yang Xiaojun Wan ALM AI4MH 152 161 0 05 Apr 2023
Calibrating Sequence likelihood Improves Conditional Language GenerationInternational Conference on Learning Representations (ICLR), 2022 Yao-Min Zhao Misha Khalman Rishabh Joshi Shashi Narayan Mohammad Saleh Peter J. Liu UQLM 179 153 0 30 Sep 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 657 3,287 0 12 Apr 2022
LoRA: Low-Rank Adaptation of Large Language ModelsInternational Conference on Learning Representations (ICLR), 2021 J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 1.4K 14,105 0 17 Jun 2021
Estimating Training Data Influence by Tracing Gradient DescentNeural Information Processing Systems (NeurIPS), 2020 G. Pruthi Frederick Liu Mukund Sundararajan Satyen Kale TDI 591 511 0 19 Feb 2020
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 998 22,826 0 20 Jul 2017
Deep reinforcement learning from human preferencesNeural Information Processing Systems (NeurIPS), 2017 Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 1.0K 4,125 0 12 Jun 2017
Adam: A Method for Stochastic OptimizationInternational Conference on Learning Representations (ICLR), 2014 Diederik P. Kingma Jimmy Ba ODL 4.2K 158,891 0 22 Dec 2014