Improved Knowledge Distillation via Teacher Assistant

9 February 2019

Papers citing "Improved Knowledge Distillation via Teacher Assistant"

50 / 151 papers shown

Title
LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks Di Hong Jiangrong Shen Yu Qi Yueming Wang 22 5 0 17 Apr 2023
Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation Feixiang Ren ViT 19 2 0 12 Apr 2023
Grouped Knowledge Distillation for Deep Face Recognition Weisong Zhao Xiangyu Zhu Kaiwen Guo Xiaoyu Zhang Zhen Lei CVBM 15 6 0 10 Apr 2023
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model Xin Yao Ziqing Yang Yiming Cui Shijin Wang 23 3 0 03 Apr 2023
DIME-FM: DIstilling Multimodal and Efficient Foundation Models Ximeng Sun Pengchuan Zhang Peizhao Zhang Hardik Shah Kate Saenko Xide Xia VLM 20 20 0 31 Mar 2023
DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving Ju He Zhi-Qi Cheng Chenyang Li Wangmeng Xiang Binghui Chen Bin Luo Yifeng Geng Xuansong Xie AI4CE 18 20 0 30 Mar 2023
Decoupled Multimodal Distilling for Emotion Recognition Yong Li Yuan-Zheng Wang Zhen Cui 19 71 0 24 Mar 2023
MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation Vitaliy Kinakh M. Drozdova S. Voloshynovskiy 40 1 0 21 Mar 2023
Focus on Your Target: A Dual Teacher-Student Framework for Domain-adaptive Semantic Segmentation Xinyue Huo Lingxi Xie Wen-gang Zhou Houqiang Li Qi Tian 26 8 0 16 Mar 2023
Distillation from Heterogeneous Models for Top-K Recommendation SeongKu Kang Wonbin Kweon Dongha Lee Jianxun Lian Xing Xie Hwanjo Yu VLM 27 21 0 02 Mar 2023
Practical Knowledge Distillation: Using DNNs to Beat DNNs Chungman Lee Pavlos Anastasios Apostolopulos Igor L. Markov FedML 20 1 0 23 Feb 2023
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 27 24 0 19 Feb 2023
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection Chenglong Wang Yi Lu Yongyu Mu Yimin Hu Tong Xiao Jingbo Zhu 32 8 0 01 Feb 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 22 12 0 28 Jan 2023
Tailor: Altering Skip Connections for Resource-Efficient Inference Olivia Weng Gabriel Marcano Vladimir Loncar Alireza Khodamoradi Nojan Sheybani Andres Meza F. Koushanfar K. Denolf Javier Mauricio Duarte Ryan Kastner 33 11 0 18 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 50 121 0 17 Jan 2023
Interaction-Aware Trajectory Planning for Autonomous Vehicles with Analytic Integration of Neural Networks into Model Predictive Control Piyush Gupta David Isele Donggun Lee S. Bae 34 19 0 13 Jan 2023
BD-KD: Balancing the Divergences for Online Knowledge Distillation Ibtihel Amara N. Sepahvand B. Meyer W. Gross J. Clark 24 2 0 25 Dec 2022
CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT Dan DeGenaro Jugal Kalita 27 0 0 22 Dec 2022
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao-Lun Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 67 2 0 10 Dec 2022
Responsible Active Learning via Human-in-the-loop Peer Study Yu Cao Jingya Wang Baosheng Yu Dacheng Tao 25 0 0 24 Nov 2022
Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans? Neeraj Varshney Man Luo Chitta Baral RALM 19 11 0 23 Nov 2022
AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation Hyungmin Kim Sungho Suh Sunghyun Baek Daehwan Kim Daun Jeong Hansang Cho Junmo Kim 22 5 0 20 Nov 2022
Yield Evaluation of Citrus Fruits based on the YoloV5 compressed by Knowledge Distillation Yuqi Li Yuting He Yihang Zhou Zirui Gong Ren-Jun Huang 16 0 0 16 Nov 2022
Teacher-Student Architecture for Knowledge Learning: A Survey Chengming Hu Xuan Li Dan Liu Xi Chen Ju Wang Xue Liu 20 35 0 28 Oct 2022
Improved Feature Distillation via Projector Ensemble Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Zi Huang 31 37 0 27 Oct 2022
Exploiting Features and Logits in Heterogeneous Federated Learning Yun-Hin Chan Edith C. H. Ngai FedML 24 2 0 27 Oct 2022
Respecting Transfer Gap in Knowledge Distillation Yulei Niu Long Chen Chan Zhou Hanwang Zhang 21 23 0 23 Oct 2022
Few-Shot Learning of Compact Models via Task-Specific Meta Distillation Yong Wu Shekhor Chanda M. Hosseinzadeh Zhi Liu Yang Wang VLM 29 7 0 18 Oct 2022
Approximating Continuous Convolutions for Deep Network Compression Theo W. Costain V. Prisacariu 28 0 0 17 Oct 2022
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks Charith Peris Lizhen Tan Thomas Gueudré Turan Gojayev Vivi Wei Gokmen Oz 28 4 0 10 Oct 2022
Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing Peng Ye Shengji Tang Baopu Li Tao Chen Wanli Ouyang 29 13 0 09 Oct 2022
Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation Rahul Mishra Hari Prabhat Gupta 27 8 0 30 Sep 2022
Slimmable Networks for Contrastive Self-supervised Learning Shuai Zhao Xiaohan Wang Linchao Zhu Yi Yang 27 1 0 30 Sep 2022
PROD: Progressive Distillation for Dense Retrieval Zhenghao Lin Yeyun Gong Xiao Liu Hang Zhang Chen Lin ... Jian Jiao Jing Lu Daxin Jiang Rangan Majumder Nan Duan 48 27 0 27 Sep 2022
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching Kunbo Ding Weijie Liu Yuejian Fang Zhe Zhao Qi Ju Xuefeng Yang 23 1 0 13 Sep 2022
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy Jingru Li Sheng Zhou Liangcheng Li Haishuai Wang Zhi Yu Jiajun Bu 28 14 0 29 Aug 2022
Masked Autoencoders Enable Efficient Knowledge Distillers Yutong Bai Zeyu Wang Junfei Xiao Chen Wei Huiyu Wang Alan Yuille Yuyin Zhou Cihang Xie CLL 24 39 0 25 Aug 2022
Task-Balanced Distillation for Object Detection Ruining Tang Zhen-yu Liu Yangguang Li Yiguo Song Hui Liu Qide Wang Jing Shao Guifang Duan Jianrong Tan 26 20 0 05 Aug 2022
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition Chuanguang Yang Zhulin An Helong Zhou Fuzhen Zhuang Yongjun Xu Qian Zhang 36 50 0 23 Jul 2022
Utilizing Excess Resources in Training Neural Networks Amit Henig Raja Giryes 37 0 0 12 Jul 2022
PrUE: Distilling Knowledge from Sparse Teacher Networks Shaopu Wang Xiaojun Chen Mengzhen Kou Jinqiao Shi 8 2 0 03 Jul 2022
Knowledge Distillation of Transformer-based Language Models Revisited Chengqiang Lu Jianwei Zhang Yunfei Chu Zhengyu Chen Jingren Zhou Fei Wu Haiqing Chen Hongxia Yang VLM 25 10 0 29 Jun 2022
Variational Distillation for Multi-View Learning Xudong Tian Zhizhong Zhang Cong Wang Wensheng Zhang Yanyun Qu Lizhuang Ma Zongze Wu Yuan Xie Dacheng Tao 26 5 0 20 Jun 2022
Multi scale Feature Extraction and Fusion for Online Knowledge Distillation Panpan Zou Yinglei Teng Tao Niu 27 3 0 16 Jun 2022
Parameter-Efficient and Student-Friendly Knowledge Distillation Jun Rao Xv Meng Liang Ding Shuhan Qi Dacheng Tao 37 46 0 28 May 2022
A Closer Look at Self-Supervised Lightweight Vision Transformers Shaoru Wang Jin Gao Zeming Li Jian-jun Sun Weiming Hu ViT 67 41 0 28 May 2022
Learning ReLU networks to high uniform accuracy is intractable Julius Berner Philipp Grohs F. Voigtlaender 32 4 0 26 May 2022
Knowledge Distillation from A Stronger Teacher Tao Huang Shan You Fei Wang Chao Qian Chang Xu 17 235 0 21 May 2022
Generalized Knowledge Distillation via Relationship Matching Han-Jia Ye Su Lu De-Chuan Zhan FedML 22 20 0 04 May 2022