A Comprehensive Overhaul of Feature Distillation

3 April 2019

Papers citing "A Comprehensive Overhaul of Feature Distillation"

50 / 98 papers shown

Title
Swapped Logit Distillation via Bi-level Teacher Alignment Stephen Ekaputra Limantoro Jhe-Hao Lin Chih-Yu Wang Yi-Lung Tsai Hong-Han Shuai Ching-Chun Huang Wen-Huang Cheng 54 0 0 27 Apr 2025
Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data Yuting He Yiqiang Chen Xiaodong Yang H. Yu Yi-Hua Huang Yang Gu FedML 55 20 0 20 Apr 2025
GranQ: Granular Zero-Shot Quantization with Unified Layer-Channel Awareness Inpyo Hong Youngwan Jo Hyojeong Lee Sunghyun Ahn Sanghyun Park MQ 60 0 0 24 Mar 2025
Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models Y. Cai Ziqi Zhang Ding Li Yao Guo Xiangqun Chen 50 0 0 13 Mar 2025
FEDS: Feature and Entropy-Based Distillation Strategy for Efficient Learned Image Compression H. Fu Jie Liang Zhenman Fang Jingning Han 45 0 0 09 Mar 2025
VRM: Knowledge Distillation via Virtual Relation Matching W. Zhang Fei Xie Weidong Cai Chao Ma 76 0 0 28 Feb 2025
Contrastive Representation Distillation via Multi-Scale Feature Decoupling Cuipeng Wang Tieyuan Chen Haipeng Wang 49 0 0 09 Feb 2025
Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer Hu Hu Sabato Marco Siniscalchi Chao-Han Huck Yang Chin-Hui Lee 72 0 0 28 Jan 2025
Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective Jinjing Zhu Songze Li Lin Wang 47 0 0 13 Jan 2025
Knowledge Distillation with Adapted Weight Sirong Wu Xi Luo Junjie Liu Yuhui Deng 40 0 0 06 Jan 2025
GazeGen: Gaze-Driven User Interaction for Visual Content Generation He-Yen Hsieh Ziyun Li Sai Qian Zhang W. Ting Kao-Den Chang B. D. Salvo Chiao Liu H. T. Kung VGen 35 0 0 07 Nov 2024
Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning Yuxiang Lu Shengcao Cao Yu-xiong Wang 49 1 0 18 Oct 2024
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation Mike Ranzinger Jon Barker Greg Heinrich Pavlo Molchanov Bryan Catanzaro Andrew Tao 37 5 0 02 Oct 2024
Harmonizing knowledge Transfer in Neural Network with Unified Distillation Yaomin Huang Zaomin Yan Chaomin Shen Faming Fang Guixu Zhang 29 0 0 27 Sep 2024
Towards Model-Agnostic Dataset Condensation by Heterogeneous Models Jun-Yeong Moon Jung Uk Kim Gyeong-Moon Park DD 33 1 0 22 Sep 2024
Revisiting Prompt Pretraining of Vision-Language Models Zhenyuan Chen Lingfeng Yang Shuo Chen Zhaowei Chen Jiajun Liang Xiang Li MLLM VPVLM VLM 43 1 0 10 Sep 2024
Relational Representation Distillation Nikolaos Giakoumoglou Tania Stathaki 37 0 0 16 Jul 2024
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification Omar S. El-Assiouti Ghada Hamed Dina Khattab H. M. Ebied 37 1 0 10 Jul 2024
FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning Seongyoon Kim Minchan Jeong Sungnyun Kim Sungwoo Cho Sumyeong Ahn Se-Young Yun FedML 47 0 0 04 Jun 2024
OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All Yuanhuiyi Lyu Xueye Zheng Dahun Kim Lin Wang 51 11 0 25 May 2024
CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective Wencheng Zhu Xin Zhou Pengfei Zhu Yu Wang Qinghua Hu VLM 58 1 0 22 Apr 2024
On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models Sean Farhat Deming Chen 42 0 0 04 Apr 2024
Attention-guided Feature Distillation for Semantic Segmentation Amir M. Mansourian Arya Jalali Rozhan Ahmadi S. Kasaei 28 0 0 08 Mar 2024
Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport Haseeb Aslam Muhammad Osama Zeeshan Soufiane Belharbi M. Pedersoli A. L. Koerich Simon L Bacon Eric Granger 20 9 0 27 Jan 2024
TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation Taeyang Yun Hyunkuk Lim Jeong-Hoon Lee Min Song 32 11 0 16 Jan 2024
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One Michael Ranzinger Greg Heinrich Jan Kautz Pavlo Molchanov VLM 41 42 0 10 Dec 2023
Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization Jixuan Leng Yijiang Li Haohan Wang VLM 31 0 0 26 Nov 2023
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning Seonghak Kim Gyeongdo Ham Yucheol Cho Daeshik Kim 22 2 0 23 Nov 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 26 0 0 26 Oct 2023
Review helps learn better: Temporal Supervised Knowledge Distillation Dongwei Wang Zhi-Long Han Yanmei Wang Xi’ai Chen Baicheng Liu Yandong Tang 57 1 0 03 Jul 2023
CrossKD: Cross-Head Knowledge Distillation for Object Detection Jiabao Wang Yuming Chen Zhaohui Zheng Xiang Li Ming-Ming Cheng Qibin Hou 40 32 0 20 Jun 2023
Improving Knowledge Distillation via Regularizing Feature Norm and Direction Yuzhu Wang Lechao Cheng Manni Duan Yongheng Wang Zunlei Feng Shu Kong 29 19 0 26 May 2023
Knowledge Diffusion for Distillation Tao Huang Yuan Zhang Mingkai Zheng Shan You Fei Wang Chao Qian Chang Xu 37 50 0 25 May 2023
Decoupled Kullback-Leibler Divergence Loss Jiequan Cui Zhuotao Tian Zhisheng Zhong Xiaojuan Qi Bei Yu Hanwang Zhang 39 38 0 23 May 2023
Student-friendly Knowledge Distillation Mengyang Yuan Bo Lang Fengnan Quan 20 17 0 18 May 2023
Function-Consistent Feature Distillation Dongyang Liu Meina Kan Shiguang Shan Xilin Chen 44 18 0 24 Apr 2023
DIME-FM: DIstilling Multimodal and Efficient Foundation Models Ximeng Sun Pengchuan Zhang Peizhao Zhang Hardik Shah Kate Saenko Xide Xia VLM 25 20 0 31 Mar 2023
Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection Pilhyeon Lee Taeoh Kim Minho Shim Dongyoon Wee H. Byun 30 11 0 30 Mar 2023
Decoupled Multimodal Distilling for Emotion Recognition Yong Li Yuan-Zheng Wang Zhen Cui 19 71 0 24 Mar 2023
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li 26 72 0 23 Mar 2023
Audio Representation Learning by Distilling Video as Privileged Information Amirhossein Hajavi Ali Etemad 15 4 0 06 Feb 2023
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models Sucheng Ren Fangyun Wei Zheng-Wei Zhang Han Hu 40 34 0 03 Jan 2023
Exploring Content Relationships for Distilling Efficient GANs Lizhou You Mingbao Lin Tie Hu Fei Chao Rongrong Ji 30 3 0 21 Dec 2022
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images Yuan Yao Yuanhan Zhang Zhen-fei Yin Jiebo Luo Wanli Ouyang Xiaoshui Huang 3DPC 29 10 0 17 Dec 2022
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection Zizhang Wu Yunzhe Wu Jian Pu Xianzhi Li Xiaoquan Wang 30 14 0 30 Nov 2022
Rethinking Implicit Neural Representations for Vision Learners Yiran Song Qianyu Zhou Lizhuang Ma 16 7 0 22 Nov 2022
Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling Yu Wang Xin Li Shengzhao Wen Fu-En Yang Wanping Zhang Gang Zhang Haocheng Feng Junyu Han Errui Ding 42 5 0 15 Nov 2022
Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection Linfeng Zhang Yukang Shi Hung-Shuo Tai Zhipeng Zhang Yuan He Ke Wang Kaisheng Ma 18 2 0 14 Nov 2022
Pixel-Wise Contrastive Distillation Junqiang Huang Zichao Guo 37 4 0 01 Nov 2022
Improved Feature Distillation via Projector Ensemble Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Zi Huang 31 37 0 27 Oct 2022