Understanding and Improving Knowledge Distillation

10 February 2020

Papers citing "Understanding and Improving Knowledge Distillation"

28 / 28 papers shown

Title
Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data Yuting He Yiqiang Chen Xiaodong Yang H. Yu Yi-Hua Huang Yang Gu FedML 55 20 0 20 Apr 2025
CURing Large Models: Compression via CUR Decomposition Sanghyeon Park Soo-Mook Moon 41 0 0 08 Jan 2025
On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance Jaskirat Singh Bram Adams Ahmed E. Hassan VLM 36 0 0 01 Nov 2024
CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective Wencheng Zhu Xin Zhou Pengfei Zhu Yu Wang Qinghua Hu VLM 56 1 0 22 Apr 2024
Interpretability-Aware Vision Transformer Yao Qiang Chengyin Li Prashant Khanduri D. Zhu ViT 80 7 0 14 Sep 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 22 12 0 28 Jan 2023
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao-Lun Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 61 2 0 10 Dec 2022
Using Knowledge Distillation to improve interpretable models in a retail banking context Maxime Biehler Mohamed Guermazi Célim Starck 49 2 0 30 Sep 2022
Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing? Keshigeyan Chandrasegaran Ngoc-Trung Tran Yunqing Zhao Ngai-man Cheung 83 41 0 29 Jun 2022
Localization Distillation for Object Detection Zhaohui Zheng Rongguang Ye Ping Wang Dongwei Ren Jun Wang W. Zuo Ming-Ming Cheng 21 64 0 12 Apr 2022
Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning Seunghyun Lee B. Song 19 8 0 05 Mar 2022
Better Supervisory Signals by Observing Learning Paths Yi Ren Shangmin Guo Danica J. Sutherland 25 21 0 04 Mar 2022
Fortuitous Forgetting in Connectionist Networks Hattie Zhou Ankit Vani Hugo Larochelle Aaron Courville CLL 6 42 0 01 Feb 2022
Towards Model Agnostic Federated Learning Using Knowledge Distillation A. Afonin Sai Praneeth Karimireddy FedML 30 44 0 28 Oct 2021
Language Modelling via Learning to Rank A. Frydenlund Gagandeep Singh Frank Rudzicz 45 7 0 13 Oct 2021
Knowledge Distillation for Quality Estimation Amit Gajbhiye M. Fomicheva Fernando Alva-Manchego Frédéric Blain A. Obamuyide Nikolaos Aletras Lucia Specia 19 11 0 01 Jul 2021
Diversifying Dialog Generation via Adaptive Label Smoothing Yida Wang Yinhe Zheng Yong-jia Jiang Minlie Huang 22 37 0 30 May 2021
Knowledge Distillation as Semiparametric Inference Tri Dao G. Kamath Vasilis Syrgkanis Lester W. Mackey 22 31 0 20 Apr 2021
Understanding and Improving Lexical Choice in Non-Autoregressive Translation Liang Ding Longyue Wang Xuebo Liu Derek F. Wong Dacheng Tao Zhaopeng Tu 96 77 0 29 Dec 2020
Federated Knowledge Distillation Hyowoon Seo Jihong Park Seungeun Oh M. Bennis Seong-Lyun Kim FedML 15 90 0 04 Nov 2020
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher Guangda Ji Zhanxing Zhu 51 42 0 20 Oct 2020
Anti-Distillation: Improving reproducibility of deep networks G. Shamir Lorenzo Coviello 34 20 0 19 Oct 2020
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks Yoonho Boo Sungho Shin Jungwook Choi Wonyong Sung MQ 16 29 0 30 Sep 2020
Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models T. Wang Selen Pehlivan Jorma T. Laaksonen 22 34 0 18 Aug 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 19 2,835 0 09 Jun 2020
Machine Learning at the Network Edge: A Survey M. G. Sarwar Murshed Chris Murphy Daqing Hou Nazar Khan Ganesh Ananthanarayanan Faraz Hussain 30 378 0 31 Jul 2019
Bag of Tricks for Image Classification with Convolutional Neural Networks Tong He Zhi-Li Zhang Hang Zhang Zhongyue Zhang Junyuan Xie Mu Li 221 1,399 0 04 Dec 2018
Large scale distributed neural network training through online distillation Rohan Anil Gabriel Pereyra Alexandre Passos Róbert Ormándi George E. Dahl Geoffrey E. Hinton FedML 272 404 0 09 Apr 2018