Knowledge Distillation as Efficient Pre-training: Faster Convergence,
Higher Data-efficiency, and Better Transferability

Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability

10 March 2022

Shuyang Sun

Xiaojuan Qi

Papers citing "Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability"

18 / 18 papers shown

Title
Reinforced Model Merging J. N. Han Jingwen Ye Shunyu Liu Haofei Zhang Jie Song Zunlei Feng Mingli Song MoMe 55 0 0 27 Mar 2025
Keeping Representation Similarity in Finetuning for Medical Image Analysis Wenqiang Zu Shenghao Xie Hao Chen Yiming Liang Lei Ma MedIm OOD 41 0 0 10 Mar 2025
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning Amin Karimi Monsefi Mengxi Zhou Nastaran Karimi Monsefi Ser-Nam Lim Wei-Lun Chao R. Ramnath 36 1 0 16 Sep 2024
Lightweight Model Pre-training via Language Guided Knowledge Distillation Mingsheng Li Lin Zhang Mingzhen Zhu Zilong Huang Gang Yu Jiayuan Fan Tao Chen 34 0 0 17 Jun 2024
Acceleration Algorithms in GNNs: A Survey Lu Ma Zeang Sheng Xunkai Li Xin Gao Zhezheng Hao Ling Yang Wentao Zhang Bin Cui GNN 34 3 0 07 May 2024
Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data Niclas Popp J. H. Metzen Matthias Hein VLM 40 1 0 25 Apr 2024
SeD: Semantic-Aware Discriminator for Image Super-Resolution Bingchen Li Xin Li Hanxin Zhu Yeying Jin Ruoyu Feng Zhizheng Zhang Zhibo Chen SupR 35 22 0 29 Feb 2024
A Deep Hierarchical Feature Sparse Framework for Occluded Person Re-Identification Yihu Song Shuaishi Liu 23 1 0 15 Jan 2024
Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments Calvin Tanama Kunyu Peng Zdravko Marinov Rainer Stiefelhagen Alina Roitberg 12 1 0 10 Nov 2023
Online Speculative Decoding Xiaoxuan Liu Lanxiang Hu Peter Bailis Alvin Cheung Zhijie Deng Ion Stoica Hao Zhang 23 50 0 11 Oct 2023
Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval Yi Xie Huaidong Zhang Xuemiao Xu Jianqing Zhu Shengfeng He VLM 13 13 0 16 Mar 2023
TAKT: Target-Aware Knowledge Transfer for Whole Slide Image Classification Conghao Xiong Yi-Mou Lin Hao Chen Hao Zheng Dong Wei Yefeng Zheng Joseph J. Y. Sung Irwin King 18 3 0 10 Mar 2023
Knowledge Distillation on Graphs: A Survey Yijun Tian Shichao Pei Xiangliang Zhang Chuxu Zhang Nitesh V. Chawla 10 28 0 01 Feb 2023
Long-Range Zero-Shot Generative Deep Network Quantization Yan Luo Yangcheng Gao Zhao Zhang Haijun Zhang Mingliang Xu Meng Wang MQ 17 9 0 13 Nov 2022
Attention Distillation: self-supervised vision transformer students need more guidance Kai Wang Fei Yang Joost van de Weijer ViT 17 16 0 03 Oct 2022
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 166 684 0 22 Apr 2021
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 245 655 0 23 Mar 2020
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 249 1,821 0 18 Aug 2016