On student-teacher deviations in distillation: does it pay to disobey?

On student-teacher deviations in distillation: does it pay to disobey?

30 January 2023

Vaishnavh Nagarajan

Srinadh Bhojanapalli

Papers citing "On student-teacher deviations in distillation: does it pay to disobey?"

6 / 6 papers shown

Title
Contrastive Representation Distillation via Multi-Scale Feature Decoupling Cuipeng Wang Tieyuan Chen Haipeng Wang 49 0 0 09 Feb 2025
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws M. E. Ildiz Halil Alperen Gozeten Ege Onur Taga Marco Mondelli Samet Oymak 54 2 0 24 Oct 2024
Provable Weak-to-Strong Generalization via Benign Overfitting David X. Wu A. Sahai 58 6 0 06 Oct 2024
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher Guangda Ji Zhanxing Zhu 51 42 0 20 Oct 2020
Large scale distributed neural network training through online distillation Rohan Anil Gabriel Pereyra Alexandre Passos Róbert Ormándi George E. Dahl Geoffrey E. Hinton FedML 267 404 0 09 Apr 2018
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 282 39,190 0 01 Sep 2014