FitNets: Hints for Thin Deep Nets

19 December 2014

Adriana Romero

Nicolas Ballas

Samira Ebrahimi Kahou

Papers citing "FitNets: Hints for Thin Deep Nets"

50 / 667 papers shown

Title
Debiased Distillation by Transplanting the Last Layer Jiwoon Lee Jaeho Lee 23 3 0 22 Feb 2023
Two-in-one Knowledge Distillation for Efficient Facial Forgery Detection Chu Zhou Jiajun Huang Daochang Liu Chengbin Du Siqi Ma Surya Nepal Chang Xu 30 0 0 21 Feb 2023
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 27 24 0 19 Feb 2023
Take a Prior from Other Tasks for Severe Blur Removal Pei Wang Danna Xue Yu Zhu Jinqiu Sun Qingsen Yan Sung-eui Yoon Yanning Zhang 26 1 0 14 Feb 2023
Audio Representation Learning by Distilling Video as Privileged Information Amirhossein Hajavi Ali Etemad 21 4 0 06 Feb 2023
Rethinking Soft Label in Label Distribution Learning Perspective Seungbum Hong Jihun Yoon Bogyu Park Min-Kook Choi 31 0 0 31 Jan 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 30 12 0 28 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 53 121 0 17 Jan 2023
CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials Kishalay Das Bidisha Samanta Pawan Goyal Seung-Cheol Lee S. Bhattacharjee Niloy Ganguly 20 17 0 14 Jan 2023
InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers Leonid Boytsov Preksha Patel Vivek Sourabh Riddhi Nisar Sayan Kundu R. Ramanathan Eric Nyberg 29 19 0 08 Jan 2023
Explainability and Robustness of Deep Visual Classification Models Jindong Gu AAML 44 2 0 03 Jan 2023
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models Sucheng Ren Fangyun Wei Zheng-Wei Zhang Han Hu 40 34 0 03 Jan 2023
BD-KD: Balancing the Divergences for Online Knowledge Distillation Ibtihel Amara N. Sepahvand B. Meyer W. Gross J. Clark 29 2 0 25 Dec 2022
Exploring Content Relationships for Distilling Efficient GANs Lizhou You Mingbao Lin Tie Hu Rongrong Ji Rongrong Ji 43 3 0 21 Dec 2022
Gait Recognition Using 3-D Human Body Shape Inference Haidong Zhu Zhao-Heng Zheng Ramkant Nevatia CVBM 3DH 28 23 0 18 Dec 2022
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images Yuan Yao Yuanhan Zhang Zhen-fei Yin Jiebo Luo Wanli Ouyang Xiaoshui Huang 3DPC 29 10 0 17 Dec 2022
Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning? Runpei Dong Zekun Qi Linfeng Zhang Junbo Zhang Jian‐Yuan Sun Zheng Ge Li Yi Kaisheng Ma ViT 3DPC 29 84 0 16 Dec 2022
Multi-scale Feature Imitation for Unsupervised Anomaly Localization Chao Hu Shengxin Lai 19 0 0 12 Dec 2022
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 67 2 0 10 Dec 2022
Deep Incubation: Training Large Models by Divide-and-Conquering Zanlin Ni Yulin Wang Jiangwei Yu Haojun Jiang Yu Cao Gao Huang VLM 18 11 0 08 Dec 2022
BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection Jianing Li Ming Lu Jiaming Liu Yandong Guo Li Du Shanghang Zhang 40 6 0 01 Dec 2022
Hint-dynamic Knowledge Distillation Yiyang Liu Chenxin Li Xiaotong Tu Xinghao Ding Yue Huang 14 1 0 30 Nov 2022
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection Zizhang Wu Yunzhe Wu Jian Pu Xianzhi Li Xiaoquan Wang 30 14 0 30 Nov 2022
Curriculum Temperature for Knowledge Distillation Zheng Li Xiang Li Lingfeng Yang Borui Zhao Renjie Song Lei Luo Jun Yu Li Jian Yang 33 133 0 29 Nov 2022
Rethinking Implicit Neural Representations for Vision Learners Yiran Song Qianyu Zhou Lizhuang Ma 18 7 0 22 Nov 2022
Accelerating Diffusion Sampling with Classifier-based Feature Distillation Wujie Sun Defang Chen Can Wang Deshi Ye Yan Feng Chun-Yen Chen 35 16 0 22 Nov 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 40 6 0 17 Nov 2022
D $^3$ ETR: Decoder Distillation for Detection Transformer Xiaokang Chen Jiahui Chen Yong-Jin Liu Gang Zeng 42 16 0 17 Nov 2022
Yield Evaluation of Citrus Fruits based on the YoloV5 compressed by Knowledge Distillation Yuqi Li Yuting He Yihang Zhou Zirui Gong Ren-Jun Huang 21 0 0 16 Nov 2022
Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection Linfeng Zhang Yukang Shi Hung-Shuo Tai Zhipeng Zhang Yuan He Ke Wang Kaisheng Ma 23 2 0 14 Nov 2022
Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection Yu Hong Hang Dai Yong Ding 3DPC 30 71 0 14 Nov 2022
Partial Binarization of Neural Networks for Budget-Aware Efficient Learning Udbhav Bamba Neeraj Anand Saksham Aggarwal Dilip K Prasad D. K. Gupta MQ 26 0 0 12 Nov 2022
Multimodal Dyadic Impression Recognition via Listener Adaptive Cross-Domain Fusion Yuanchao Li P. Bell Catherine Lai 28 3 0 09 Nov 2022
Hilbert Distillation for Cross-Dimensionality Networks Dian Qin Haishuai Wang Zhe Liu Hongjia Xu Sheng Zhou Jiajun Bu 23 4 0 08 Nov 2022
Distilling Representations from GAN Generator via Squeeze and Span Yu Yang Xiaotian Cheng Chang-rui Liu Hakan Bilen Xiang Ji GAN 31 0 0 06 Nov 2022
Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation Yanyang Li Jianqiao Zhao M. Lyu Liwei Wang 24 15 0 03 Nov 2022
MPCFormer: fast, performant and private Transformer inference with MPC Dacheng Li Rulin Shao Hongyi Wang Han Guo Eric P. Xing Haotong Zhang 13 79 0 02 Nov 2022
Gradient Knowledge Distillation for Pre-trained Language Models Lean Wang Lei Li Xu Sun VLM 23 5 0 02 Nov 2022
Maximum Likelihood Distillation for Robust Modulation Classification Javier Maroto Gérôme Bovet P. Frossard AAML 15 5 0 01 Nov 2022
SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization Masud An Nur Islam Fahim Jani Boutellier 40 0 0 01 Nov 2022
ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation Tomokatsu Takahashi Masanori Yamada Yuuki Yamanaka Tomoya Yamashita 20 0 0 01 Nov 2022
Pixel-Wise Contrastive Distillation Junqiang Huang Zichao Guo 42 4 0 01 Nov 2022
Teacher-Student Architecture for Knowledge Learning: A Survey Chengming Hu Xuan Li Dan Liu Xi Chen Ju Wang Xue Liu 20 35 0 28 Oct 2022
Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition Yujin Wang Changli Tang Ziyang Ma Zhisheng Zheng Xie Chen Weiqiang Zhang 46 1 0 27 Oct 2022
Multimodal Transformer Distillation for Audio-Visual Synchronization Xuan-Bo Chen Haibin Wu Chung-Che Wang Hung-yi Lee J. Jang 26 3 0 27 Oct 2022
Improved Feature Distillation via Projector Ensemble Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Zi Huang 37 37 0 27 Oct 2022
Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision Jiongyu Guo Defang Chen Can Wang 22 3 0 25 Oct 2022
Respecting Transfer Gap in Knowledge Distillation Yulei Niu Long Chen Chan Zhou Hanwang Zhang 26 23 0 23 Oct 2022
Few-Shot Learning of Compact Models via Task-Specific Meta Distillation Yong Wu Shekhor Chanda M. Hosseinzadeh Zhi Liu Yang Wang VLM 29 7 0 18 Oct 2022
Approximating Continuous Convolutions for Deep Network Compression Theo W. Costain V. Prisacariu 36 0 0 17 Oct 2022