Distilling Reasoning Capabilities into Smaller Language Models

1 December 2022

Papers citing "Distilling Reasoning Capabilities into Smaller Language Models"

50 / 113 papers shown

Title
Recursive Decomposition with Dependencies for Generic Divide-and-Conquer Reasoning Sergio Hernández-Gutiérrez Minttu Alakuijala Alexander Nikitin Pekka Marttinen LRM 52 2 0 05 May 2025
Antidistillation Sampling Yash Savani Asher Trockman Zhili Feng Avi Schwarzschild Alexander Robey Marc Finzi J. Zico Kolter 44 0 0 17 Apr 2025
Training Small Reasoning LLMs with Cognitive Preference Alignment Wenrui Cai Chengyu Wang Junbing Yan Jun Huang Xiangzhong Fang LRM 26 0 0 14 Apr 2025
A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions Chengyu Wang Taolin Zhang Richang Hong Jun Huang ReLM LRM 37 1 0 12 Apr 2025
UNDO: Understanding Distillation as Optimization Kushal Kumar Jain Piyushi Goyal Kumar Shridhar 36 0 0 03 Apr 2025
A Survey of Scaling in Large Language Model Reasoning Zihan Chen Song Wang Zhen Tan Xingbo Fu Zhenyu Lei Peng Wang Huan Liu Cong Shen Jundong Li LRM 86 0 0 02 Apr 2025
VITED: Video Temporal Evidence Distillation Yujie Lu Yale Song William Yang Wang Lorenzo Torresani Tushar Nagarajan 115 0 0 17 Mar 2025
Sample-aware Adaptive Structured Pruning for Large Language Models Jun Kong Xinge Ma Jin Wang Xuejie Zhang 45 0 0 08 Mar 2025
Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings Xuanqing Liu Luyang Kong Wei Niu Afshin Khashei Belinda Zeng Steve Johnson Jon Jay Davor Golac Matt Pope 41 0 0 07 Mar 2025
Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? Yudi Zhang Lu Wang Meng Fang Yali Du Chenghua Huang ... Qingwei Lin Mykola Pechenizkiy Dongmei Zhang Saravan Rajmohan Qi Zhang ALM 71 0 0 26 Feb 2025
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning Xinghao Chen Zhijing Sun Wenjin Guo Miaoran Zhang Yanjun Chen ... Hui Su Yijie Pan Dietrich Klakow Wenjie Li Xiaoyu Shen LRM 51 4 0 25 Feb 2025
Pub-Guard-LLM: Detecting Fraudulent Biomedical Articles with Reliable Explanations Lihu Chen Shuojie Fu Gabriel Freedman Cemre Zor Guy Martin James Kinross Uddhav Vaghela Ovidiu Serban Francesca Toni DeLMO 63 0 0 21 Feb 2025
Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification Peipei Wei Dimitris Dimitriadis Yan Xu Mingwei Shen 55 1 0 11 Feb 2025
Who Taught You That? Tracing Teachers in Model Distillation Somin Wadhwa Chantal Shaib Silvio Amir Byron C. Wallace 70 1 0 10 Feb 2025
Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference Jorge García-Carrasco A. Maté Juan Trujillo 71 0 0 20 Dec 2024
Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models Y. Fu Yin Yu Xiaotian Han Runchao Li Xianxuan Long Haotian Yu Pan Li SyDa 57 0 0 25 Nov 2024
SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning Shivam Adarsh Kumar Shridhar Caglar Gulcehre Nicholas Monath Mrinmaya Sachan LRM 29 2 0 24 Oct 2024
CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking Chia-Hsuan Lee Hao Cheng Mari Ostendorf LRM 26 0 0 23 Oct 2024
SleepCoT: A Lightweight Personalized Sleep Health Model via Chain-of-Thought Distillation Huimin Zheng Xiaofeng Xing Xiangmin Xu VLM 43 1 0 22 Oct 2024
Mentor-KD: Making Small Language Models Better Multi-step Reasoners Hojae Lee Junho Kim SangKeun Lee LRM 26 1 0 11 Oct 2024
DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback Zaid Khan Elias Stengel-Eskin Jaemin Cho Mohit Bansal VGen 38 1 0 08 Oct 2024
Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing Yilmazcan Ozyurt Stefan Feuerriegel Mrinmaya Sachan AI4Ed 44 1 0 02 Oct 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 58 23 0 10 Sep 2024
CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Tasks Yongxin Deng Xihe Qiu Xiaoyu Tan Chao Qu Jing Pan Yuan-Chia Cheng Yinghui Xu Wei Chu 34 2 0 05 Sep 2024
Prompt Baking Aman Bhargava Cameron Witkowski Alexander Detkov Matt W. Thomson AI4CE 28 0 0 04 Sep 2024
LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models Lipeng Ma Weidong Yang Sihang Jiang Ben Fei Mingjie Zhou Shuhao Li Bo Xu Bo Xu Yanghua Xiao 51 0 0 03 Sep 2024
Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices Patrícia Schmidtová Saad Mahamood Simone Balloccu Ondřej Dušek Albert Gatt Dimitra Gkatzia David M. Howcroft Ondřej Plátek Adarsa Sivaprasad 43 3 0 17 Aug 2024
FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models Zhongyu Zhao Menghang Dong Rongyu Zhang Wenzhao Zheng Yunpeng Zhang Huanrui Yang Dalong Du Kurt Keutzer Shanghang Zhang 46 0 0 15 Aug 2024
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations Leo Donisch Sigurd Schacht Carsten Lanquillon 22 2 0 06 Aug 2024
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning Tianduo Wang Shichen Li Wei Lu LRM AI4CE 45 14 1 25 Jul 2024
Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model Xunyu Zhu Jian Li Can Ma Weiping Wang LRM 36 0 0 14 Jul 2024
AI Safety in Generative AI Large Language Models: A Survey Jaymari Chua Yun Yvonna Li Shiyi Yang Chen Wang Lina Yao LM&MA 34 12 0 06 Jul 2024
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application Chuanpeng Yang Wang Lu Yao Zhu Yidong Wang Qian Chen Chenlong Gao Bingjie Yan Yiqiang Chen ALM KELM 44 22 0 02 Jul 2024
Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components Phillip Schneider Wessel Poelman Michael Rovatsos Florian Matthes 29 2 0 01 Jul 2024
Investigating Mysteries of CoT-Augmented Distillation Somin Wadhwa Silvio Amir Byron C. Wallace ReLM LRM 27 8 0 20 Jun 2024
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation Yuhang Zhou Jing Zhu Paiheng Xu Xiaoyu Liu Xiyao Wang Danai Koutra Wei Ai Furong Huang 73 4 0 19 Jun 2024
Abstraction-of-Thought Makes Language Models Better Reasoners Ruixin Hong Hongming Zhang Xiaoman Pan Dong Yu Changshui Zhang LRM 43 4 0 18 Jun 2024
Learning from Natural Language Explanations for Generalizable Entity Matching Somin Wadhwa Adit Krishnan Runhui Wang Byron C. Wallace Chris Kong LRM 37 3 0 13 Jun 2024
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios Yuhang Zhou Wei Ai 29 5 0 08 Jun 2024
mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models Huiyuan Lai Malvina Nissim LRM 41 14 0 04 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 41 0 0 03 Jun 2024
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation Chengwei Dai Kun Li Wei Zhou Song Hu LRM 36 5 0 30 May 2024
Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfer Haoyan Yang Yixuan Wang Xingyin Xu Hanyuan Zhang Yirong Bian 38 6 0 27 May 2024
MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoning Shuo Yin Weihao You Zhilong Ji Guoqiang Zhong Jinfeng Bai LRM SyDa 35 9 0 13 May 2024
MathDivide: Improved mathematical reasoning by large language models S. Srivastava Ashutosh Gandhi LRM ReLM 30 0 0 12 May 2024
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models Leonardo Ranaldi André Freitas LRM ReLM 29 8 0 01 May 2024
PatentGPT: A Large Language Model for Intellectual Property Zilong Bai Ruiji Zhang Linqing Chen Qijun Cai Yuan Zhong ... Fu Bian Xiaolong Gu Lisha Zhang Weilei Wang Changyang Tu 41 3 0 28 Apr 2024
Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training Mengzhao Jia Zhihan Zhang W. Yu Fangkai Jiao Meng-Long Jiang VLM ReLM LRM 48 7 0 22 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 82 0 22 Apr 2024
Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following Suyeon Shin Sujin Jeon Junghyun Kim Gi-Cheon Kang Byoung-Tak Zhang LLMAG 34 0 0 21 Apr 2024