Towards Understanding Knowledge Distillation

27 May 2021

Papers citing "Towards Understanding Knowledge Distillation"

50 / 67 papers shown

Title
Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation Reilly Haskins Benjamin Adams 16 0 0 16 May 2025
Importance Analysis for Dynamic Control of Balancing Parameter in a Simple Knowledge Distillation Setting Seongmin Kim Kwanho Kim Minseung Kim Kanghyun Jo 26 0 0 06 May 2025
See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias Junehyoung Kwon Mihyeon Kim Eunju Lee Juhwan Choi Youngbin Kim 60 0 0 18 Mar 2025
Towards Reasoning Ability of Small Language Models Gaurav Srivastava Shuxiang Cao Xuan Wang ReLM LRM 60 7 0 17 Feb 2025
Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identification Jiachen Li Xiaojin Gong DiffM 84 0 0 10 Feb 2025
The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model Kaito Takanami Takashi Takahashi Ayaka Sakata 40 0 0 27 Jan 2025
Provable Weak-to-Strong Generalization via Benign Overfitting David X. Wu A. Sahai 76 6 0 06 Oct 2024
Efficient Low-Resolution Face Recognition via Bridge Distillation Shiming Ge Shengwei Zhao Chenyu Li Yu Zhang Jia Li CVBM 36 58 0 18 Sep 2024
Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting Emmanuel Aboah Boateng Cassiano O. Becker Nabiha Asghar Kabir Walia Ashwin Srinivasan Ehi Nosakhare Victor Dibia Soundar Srinivasan LRM 31 0 0 18 Aug 2024
$$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning$ $\textit{Trans-LoRA}$ : towards data-free Transferable Parameter Efficient Finetuning Runqian Wang Soumya Ghosh David D. Cox Diego Antognini Aude Oliva Rogerio Feris Leonid Karlinsky 42 1 0 27 May 2024
Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System Lei Zheng Ning Li Weinan Zhang Yong Yu AI4TS 41 0 0 24 Apr 2024
Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation Jingxuan Wei Linzhuang Sun Yichong Leng Xu Tan Bihui Yu Ruifeng Guo 51 3 0 23 Apr 2024
CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective Wencheng Zhu Xin Zhou Pengfei Zhu Yu Wang Qinghua Hu VLM 64 1 0 22 Apr 2024
Learning to Maximize Mutual Information for Chain-of-Thought Distillation Xin Chen Hanxian Huang Yanjun Gao Yi Wang Jishen Zhao Ke Ding 35 12 0 05 Mar 2024
Practical Insights into Knowledge Distillation for Pre-Trained Models Norah Alballa Marco Canini 48 2 0 22 Feb 2024
Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation Bharat Srikishan Anika Tabassum S. Allu Ramakrishnan Kannan Nikhil Muralidhar 53 1 0 19 Feb 2024
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning Joonyoung Kim Kangwook Lee Haebin Shin Hurnjoo Lee Sechun Kang Byunguk Choi Dong Shin Joohyung Lee 25 0 0 15 Jul 2023
FCA: Taming Long-tailed Federated Medical Image Classification by Classifier Anchoring Jeffry Wicaksana Zengqiang Yan Kwang-Ting Cheng FedML 40 5 0 01 May 2023
Self-Distillation for Gaussian Process Regression and Classification Kenneth Borup L. Andersen 13 2 0 05 Apr 2023
Knowledge Distillation from Single to Multi Labels: an Empirical Study Youcai Zhang Yuzhuo Qin Heng-Ye Liu Yanhao Zhang Yaqian Li X. Gu VLM 55 2 0 15 Mar 2023
Better Generative Replay for Continual Federated Learning Daiqing Qi Handong Zhao Sheng Li FedML 27 47 0 25 Feb 2023
Practical Knowledge Distillation: Using DNNs to Beat DNNs Chungman Lee Pavlos Anastasios Apostolopulos Igor L. Markov FedML 27 1 0 23 Feb 2023
Knowledge Distillation on Graphs: A Survey Yijun Tian Shichao Pei Xiangliang Zhang Chuxu Zhang Nitesh V. Chawla 23 28 0 01 Feb 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 30 12 0 28 Jan 2023
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 67 2 0 10 Dec 2022
Matching DNN Compression and Cooperative Training with Resources and Data Availability F. Malandrino G. Giacomo Armin Karamzade Marco Levorato C. Chiasserini 45 9 0 02 Dec 2022
Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networks Jiyang Guan Jian Liang Ran He AAML MLAU 50 29 0 21 Oct 2022
Linkless Link Prediction via Relational Distillation Zhichun Guo William Shiao Shichang Zhang Yozen Liu Nitesh V. Chawla Neil Shah Tong Zhao 27 41 0 11 Oct 2022
NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs Yijun Tian Chuxu Zhang Zhichun Guo Xiangliang Zhang Nitesh V. Chawla 47 14 0 22 Aug 2022
Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity Jianyi Yang Shaolei Ren 32 3 0 02 Jul 2022
Prompt-aligned Gradient for Prompt Tuning Beier Zhu Yulei Niu Yucheng Han Yuehua Wu Hanwang Zhang VLM 189 274 0 30 May 2022
Generalized Knowledge Distillation via Relationship Matching Han-Jia Ye Su Lu De-Chuan Zhan FedML 22 20 0 04 May 2022
UniTE: Unified Translation Evaluation Boyi Deng Dayiheng Liu Baosong Yang Haibo Zhang Boxing Chen Derek F. Wong Lidia S. Chao 41 41 0 28 Apr 2022
Universal Representations: A Unified Look at Multiple Task and Domain Learning Wei-Hong Li Xialei Liu Hakan Bilen SSL OOD 30 27 0 06 Apr 2022
PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Models Tai-Yin Chiu Danna Gurari 23 19 0 25 Mar 2022
Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse Martin Pawelczyk Teresa Datta Johannes van-den-Heuvel Gjergji Kasneci Himabindu Lakkaraju 24 38 0 13 Mar 2022
Fortuitous Forgetting in Connectionist Networks Hattie Zhou Ankit Vani Hugo Larochelle Aaron Courville CLL 16 42 0 01 Feb 2022
Being Friends Instead of Adversaries: Deep Networks Learn from Data Simplified by Other Networks Simone Marullo Matteo Tiezzi Marco Gori S. Melacci AAML GAN 27 2 0 18 Dec 2021
Towards Model Agnostic Federated Learning Using Knowledge Distillation A. Afonin Sai Praneeth Karimireddy FedML 30 45 0 28 Oct 2021
Visualizing the embedding space to explain the effect of knowledge distillation Hyun Seung Lee C. Wallraven 17 1 0 09 Oct 2021
Compact representations of convolutional neural networks via weight pruning and quantization Giosuè Cataldo Marinò A. Petrini D. Malchiodi Marco Frasca MQ 21 4 0 28 Aug 2021
Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks Landan Seguin A. Ndirango Neeli Mishra SueYeon Chung Tyler Lee OOD 25 2 0 26 Aug 2021
Follow Your Path: a Progressive Method for Knowledge Distillation Wenxian Shi Yuxuan Song Hao Zhou Bohan Li Lei Li 17 15 0 20 Jul 2021
Class-Incremental Learning for Wireless Device Identification in IoT Yongxin Liu Jian Wang Jianqiang Li Shuteng Niu Haoze Song 31 58 0 08 May 2021
Distilling EEG Representations via Capsules for Affective Computing Guangyi Zhang Ali Etemad 24 16 0 30 Apr 2021
Knowledge Distillation as Semiparametric Inference Tri Dao G. Kamath Vasilis Syrgkanis Lester W. Mackey 40 31 0 20 Apr 2021
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation Kang Min Yoo Dongju Park Jaewook Kang Sang-Woo Lee Woomyeong Park 36 235 0 18 Apr 2021
Universal Representation Learning from Multiple Domains for Few-shot Classification Weihong Li Xialei Liu Hakan Bilen SSL OOD VLM 30 84 0 25 Mar 2021
MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection Abir Rahali M. Akhloufi 32 30 0 05 Mar 2021
Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond Risheng Liu Jiaxin Gao Jin Zhang Deyu Meng Zhouchen Lin AI4CE 59 223 0 27 Jan 2021