Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

17 May 2019

Papers citing "Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation"

43 / 143 papers shown

Title
A Semi-supervised Learning Approach with Two Teachers to Improve Breakdown Identification in Dialogues Qian Lin Hwee Tou Ng 22 4 0 22 Feb 2022
Dynamic Rectification Knowledge Distillation Fahad Rahman Amik Ahnaf Ismat Tasin Silvia Ahmed M. M. L. Elahi Nabeel Mohammed 24 5 0 27 Jan 2022
Class-Incremental Continual Learning into the eXtended DER-verse Matteo Boschini Lorenzo Bonicelli Pietro Buzzega Angelo Porrello Simone Calderara CLL BDL 26 128 0 03 Jan 2022
Multi-label Iterated Learning for Image Classification with Label Ambiguity Sai Rajeswar Pau Rodríguez López Soumye Singhal David Vazquez Aaron C. Courville VLM 23 30 0 23 Nov 2021
MUSE: Feature Self-Distillation with Mutual Information and Self-Information Yunpeng Gong Ye Yu Gaurav Mittal Greg Mori Mei Chen SSL 28 2 0 25 Oct 2021
Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression Usma Niyaz Deepti R. Bathula 18 8 0 21 Oct 2021
Single-Modal Entropy based Active Learning for Visual Question Answering Dong-Jin Kim Jae-Won Cho Jinsoo Choi Yunjae Jung In So Kweon 25 12 0 21 Oct 2021
Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles Bram Wallace Devansh Arpit Huan Wang Caiming Xiong SSL OOD 30 0 0 19 Oct 2021
Mitigating Memorization of Noisy Labels via Regularization between Representations Hao Cheng Zhaowei Zhu Xing Sun Yang Liu NoLa 38 28 0 18 Oct 2021
Unsupervised Representation Learning Meets Pseudo-Label Supervised Self-Distillation: A New Approach to Rare Disease Classification Jinghan Sun Dong Wei Kai Ma Liansheng Wang Yefeng Zheng 27 8 0 09 Oct 2021
Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better Xuanyang Zhang Xinming Zhang Jian Sun 25 1 0 26 Sep 2021
Efficient Medical Image Segmentation Based on Knowledge Distillation Dian Qin Jiajun Bu Zhe Liu Xin Shen Sheng Zhou Jingjun Gu Zhihong Wang Lei Wu Hui-Fen Dai 30 129 0 23 Aug 2021
Joint Multiple Intent Detection and Slot Filling via Self-distillation Lisong Chen Peilin Zhou Yuexian Zou VLM 16 31 0 18 Aug 2021
Semantic Concentration for Domain Adaptation Shuang Li Mixue Xie Fangrui Lv Chi Harold Liu Jian Liang C. Qin Wei Li 52 87 0 12 Aug 2021
Linking Common Vulnerabilities and Exposures to the MITRE ATT&CK Framework: A Self-Distillation Approach Benjamin Ampel Sagar Samtani Steven Ullman Hsinchun Chen 25 35 0 03 Aug 2021
Feature Mining: A Novel Training Strategy for Convolutional Neural Network Tianshu Xie Xuan Cheng Xiaomin Wang Minghui Liu Jiali Deng Ming Liu 36 5 0 18 Jul 2021
Meta-learning Amidst Heterogeneity and Ambiguity Kyeongryeol Go Seyoung Yun 24 1 0 05 Jul 2021
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption Dara Bahri Heinrich Jiang Yi Tay Donald Metzler SSL 19 163 0 29 Jun 2021
R-Drop: Regularized Dropout for Neural Networks Xiaobo Liang Lijun Wu Juntao Li Yue Wang Qi Meng Tao Qin Wei Chen M. Zhang Tie-Yan Liu 47 424 0 28 Jun 2021
Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers Andreas Kirsch Tom Rainforth Y. Gal OOD TTA 33 22 0 22 Jun 2021
Training Graph Neural Networks with 1000 Layers Guohao Li Matthias Muller Guohao Li V. Koltun GNN AI4CE 45 235 0 14 Jun 2021
Generate, Annotate, and Learn: NLP with Synthetic Text Xuanli He Islam Nassar J. Kiros Gholamreza Haffari Mohammad Norouzi 36 51 0 11 Jun 2021
Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead Arian Bakhtiarnia Qi Zhang Alexandros Iosifidis 27 35 0 19 May 2021
Distilling a Powerful Student Model via Online Knowledge Distillation Shaojie Li Mingbao Lin Yan Wang Yongjian Wu Yonghong Tian Ling Shao Rongrong Ji FedML 27 46 0 26 Mar 2021
Student Network Learning via Evolutionary Knowledge Distillation Kangkai Zhang Chunhui Zhang Shikun Li Dan Zeng Shiming Ge 16 83 0 23 Mar 2021
Compacting Deep Neural Networks for Internet of Things: Methods and Applications Ke Zhang Hanbo Ying Hongning Dai Lin Li Yuangyuang Peng Keyi Guo Hongfang Yu 16 38 0 20 Mar 2021
Knowledge Evolution in Neural Networks Ahmed Taha Abhinav Shrivastava L. Davis 45 21 0 09 Mar 2021
Localization Distillation for Dense Object Detection Zhaohui Zheng Rongguang Ye Ping Wang Dongwei Ren W. Zuo Qibin Hou Ming-Ming Cheng ObjD 98 115 0 24 Feb 2021
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning Zeyuan Allen-Zhu Yuanzhi Li FedML 41 355 0 17 Dec 2020
Data-Free Model Extraction Jean-Baptiste Truong Pratyush Maini R. Walls Nicolas Papernot MIACV 15 181 0 30 Nov 2020
End-to-End Object Detection with Adaptive Clustering Transformer Minghang Zheng Peng Gao Renrui Zhang Kunchang Li Xiaogang Wang Hongsheng Li Hao Dong ViT 6 193 0 18 Nov 2020
Fast Video Salient Object Detection via Spatiotemporal Knowledge Distillation Tang Yi Li Yuan Wenbin Zou 21 4 0 20 Oct 2020
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks Yoonho Boo Sungho Shin Jungwook Choi Wonyong Sung MQ 22 29 0 30 Sep 2020
Densely Guided Knowledge Distillation using Multiple Teacher Assistants Wonchul Son Jaemin Na Junyong Choi Wonjun Hwang 20 110 0 18 Sep 2020
Rethinking Graph Regularization for Graph Neural Networks Han Yang Kaili Ma James Cheng AI4CE 27 72 0 04 Sep 2020
Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models T. Wang Selen Pehlivan Jorma T. Laaksonen 29 34 0 18 Aug 2020
SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud Stefanos Laskaridis Stylianos I. Venieris Mario Almeida Ilias Leontiadis Nicholas D. Lane 28 265 0 14 Aug 2020
HAPI: Hardware-Aware Progressive Inference Stefanos Laskaridis Stylianos I. Venieris Hyeji Kim Nicholas D. Lane 17 45 0 10 Aug 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 19 2,837 0 09 Jun 2020
Regularizing Class-wise Predictions via Self-knowledge Distillation Sukmin Yun Jongjin Park Kimin Lee Jinwoo Shin 27 274 0 31 Mar 2020
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 253 656 0 23 Mar 2020
Preparing Lessons: Improve Knowledge Distillation with Better Supervision Tiancheng Wen Shenqi Lai Xueming Qian 25 67 0 18 Nov 2019
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 284 2,889 0 15 Sep 2016