Knowledge distillation: A good teacher is patient and consistent

9 June 2021

Papers citing "Knowledge distillation: A good teacher is patient and consistent"

50 / 203 papers shown

Title
Adversarial Robustness of Distilled and Pruned Deep Learning-based Wireless Classifiers N. Baishya B. R. Manoj AAML 23 0 0 11 Apr 2024
CodecLM: Aligning Language Models with Tailored Synthetic Data Zifeng Wang Chun-Liang Li Vincent Perot Long T. Le Jin Miao Zizhao Zhang Chen-Yu Lee Tomas Pfister SyDa ALM 23 17 0 08 Apr 2024
Tiny Machine Learning: Progress and Futures Ji Lin Ligeng Zhu Wei-Ming Chen Wei-Chen Wang Song Han 36 51 0 28 Mar 2024
Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure Fan Wan Xingyu Miao Haoran Duan Jingjing Deng Rui Gao Yang Long VLM 37 4 0 14 Mar 2024
$V_kD:$ Improving Knowledge Distillation using Orthogonal Projections Roy Miles Ismail Elezi Jiankang Deng 44 10 0 10 Mar 2024
On the Effectiveness of Distillation in Mitigating Backdoors in Pre-trained Encoder Tingxu Han Shenghan Huang Ziqi Ding Weisong Sun Yebo Feng ... Hanwei Qian Cong Wu Quanjun Zhang Yang Liu Zhenyu Chen 21 8 0 06 Mar 2024
On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models Juliette Marrie Michael Arbel Julien Mairal Diane Larlus VLM MQ 40 1 0 17 Feb 2024
Good Teachers Explain: Explanation-Enhanced Knowledge Distillation Amin Parchami-Araghi Moritz Bohle Sukrut Rao Bernt Schiele FAtt 8 3 0 05 Feb 2024
Contextualization Distillation from Large Language Model for Knowledge Graph Completion Dawei Li Zhen Tan Tianlong Chen Huan Liu KELM 25 12 0 28 Jan 2024
Model Compression Techniques in Biometrics Applications: A Survey Eduarda Caldeira Pedro C. Neto Marco Huber Naser Damer Ana F. Sequeira 32 11 0 18 Jan 2024
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information Linfeng Ye Shayan Mohajer Hamidi Renhao Tan En-Hui Yang VLM 37 12 0 16 Jan 2024
An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation Md Arafat Sultan Aashka Trivedi Parul Awasthy Avirup Sil 30 0 0 12 Jan 2024
Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data Jinjin Zhu Yucheng Chen Lin Wang 25 2 0 10 Jan 2024
Data-Centric Foundation Models in Computational Healthcare: A Survey Yunkun Zhang Jin Gao Zheling Tan Lingfeng Zhou Kexin Ding Mu Zhou Shaoting Zhang Dequan Wang AI4CE 23 22 0 04 Jan 2024
Multi-modal Learning with Missing Modality in Predicting Axillary Lymph Node Metastasis Shichuan Zhang Sunyi Zheng Zhongyi Shui Honglin Li Lin Yang 21 5 0 03 Jan 2024
Compressing Deep Image Super-resolution Models Yuxuan Jiang Jakub Nawala Fan Zhang David Bull 42 6 0 31 Dec 2023
FerKD: Surgical Label Adaptation for Efficient Distillation Zhiqiang Shen 21 3 0 29 Dec 2023
Adaptive Depth Networks with Skippable Sub-Paths Woochul Kang 28 1 0 27 Dec 2023
StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation Shiu-hong Kao Jierun Chen S.-H. Gary Chan 17 0 0 20 Dec 2023
On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width Satoki Ishikawa Ryo Karakida 24 2 0 19 Dec 2023
Decoupled Knowledge with Ensemble Learning for Online Distillation Baitan Shao Ying Chen 18 0 0 18 Dec 2023
Mixed Distillation Helps Smaller Language Model Better Reasoning Chenglin Li Qianglong Chen Liangyue Li Wang Caiyu Yicheng Li Zhang Yin Yin Zhang LRM 30 11 0 17 Dec 2023
Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval Zhe Ma Jianfeng Dong Shouling Ji Zhenguang Liu Xuhong Zhang Zonghui Wang Sifeng He Feng Qian Xiaobo Zhang Lei Yang 33 6 0 15 Dec 2023
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision Collin Burns Pavel Izmailov Jan Hendrik Kirchner Bowen Baker Leo Gao ... Adrien Ecoffet Manas Joglekar Jan Leike Ilya Sutskever Jeff Wu ELM 39 258 0 14 Dec 2023
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction Yi Guo Yiqian He Xiaoyang Li Haotong Qin Van Tung Pham Yang Zhang Shouda Liu 43 1 0 14 Dec 2023
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One Michael Ranzinger Greg Heinrich Jan Kautz Pavlo Molchanov VLM 31 42 0 10 Dec 2023
Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models Hongzhan Lin Ziyang Luo Jing Ma Long Chen 27 9 0 09 Dec 2023
Bootstrapping SparseFormers from Vision Foundation Models Ziteng Gao Zhan Tong K. Lin Joya Chen Mike Zheng Shou 33 0 0 04 Dec 2023
MoEC: Mixture of Experts Implicit Neural Compression Jianchen Zhao Cheng-Ching Tseng Ming Lu Ruichuan An Xiaobao Wei He Sun Shanghang Zhang 16 3 0 03 Dec 2023
Initializing Models with Larger Ones Zhiqiu Xu Yanjie Chen Kirill Vishniakov Yida Yin Zhiqiang Shen Trevor Darrell Lingjie Liu Zhuang Liu 30 17 0 30 Nov 2023
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training Pavan Kumar Anasosalu Vasu Hadi Pouransari Fartash Faghri Raviteja Vemulapalli Oncel Tuzel CLIP VLM 29 43 0 28 Nov 2023
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning Seonghak Kim Gyeongdo Ham Yucheol Cho Daeshik Kim 22 2 0 23 Nov 2023
MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters Chau Pham Piotr Teterwak Soren Nelson Bryan A. Plummer 9 3 0 07 Nov 2023
Comparative Knowledge Distillation Alex Wilf Alex Tianyi Xu Paul Pu Liang A. Obolenskiy Daniel Fried Louis-Philippe Morency VLM 18 1 0 03 Nov 2023
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model Karsten Roth Lukas Thede Almut Sophia Koepke Oriol Vinyals Olivier J. Hénaff Zeynep Akata AAML 17 11 0 26 Oct 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 26 0 0 26 Oct 2023
Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models Florian Schmid Khaled Koutini Gerhard Widmer 16 11 0 24 Oct 2023
Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images Logan Frank Jim Davis 25 1 0 20 Oct 2023
Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models Ruida Wang Wangchunshu Zhou Mrinmaya Sachan 19 32 0 20 Oct 2023
How a student becomes a teacher: learning and forgetting through Spectral methods Lorenzo Giambagli L. Buffoni Lorenzo Chicchi Duccio Fanelli 19 7 0 19 Oct 2023
What do larger image classifiers memorise? Michal Lukasik Vaishnavh Nagarajan A. S. Rawat A. Menon Sanjiv Kumar 30 5 0 09 Oct 2023
Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication Zhe Zhao Qingyun Liu Huan Gui Bang An Lichan Hong Ed H. Chi 20 1 0 04 Oct 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 21 1 0 20 Sep 2023
ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer Arkadiy Saakyan Smaranda Muresan 18 3 0 15 Sep 2023
Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection Hoàng-Ân Lê Minh-Tan Pham 32 2 0 12 Sep 2023
Towards Comparable Knowledge Distillation in Semantic Image Segmentation Onno Niemann Christopher Vox Thorben Werner VLM 17 1 0 07 Sep 2023
EgoBlur: Responsible Innovation in Aria Nikhil Raina Guruprasad Somasundaram Kang Zheng Sagar Miglani Steve Saarinen ... Prince Gupta Mingfei Yan Richard A. Newcombe C. Ren Omkar M. Parkhi CVBM 11 8 0 24 Aug 2023
DLIP: Distilling Language-Image Pre-training Huafeng Kuang Jie Wu Xiawu Zheng Ming Li Xuefeng Xiao Rui Wang Min Zheng Rongrong Ji VLM 36 4 0 24 Aug 2023
Don't Look into the Sun: Adversarial Solarization Attacks on Image Classifiers Paul Gavrikov J. Keuper AAML 28 0 0 24 Aug 2023
CED: Consistent ensemble distillation for audio tagging Heinrich Dinkel Yongqing Wang Zhiyong Yan Junbo Zhang Yujun Wang 20 17 0 23 Aug 2023