Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication

4 October 2023

Papers citing "Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication"

8 / 8 papers shown

Title
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Lokesh Nagalapatti Chun-Liang Li Chih-Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen-Yu Lee Tomas Pfister ALM 211 499 0 03 May 2023
Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts Tao Zhong Zhixiang Chi Li Gu Yang Wang Yuanhao Yu Jingshan Tang OOD 66 29 0 08 Oct 2022
ViTKD: Practical Guidelines for ViT feature knowledge distillation Zhendong Yang Zhe Li Ailing Zeng Zexian Li Chun Yuan Yu Li 86 42 0 06 Sep 2022
Cross-Task Knowledge Distillation in Multi-Task Recommendation Chenxiao Yang Junwei Pan Xiaofeng Gao Tingyu Jiang Dapeng Liu Guihai Chen 34 44 0 20 Feb 2022
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 173 686 0 22 Apr 2021
Pruning and Quantization for Deep Neural Network Acceleration: A Survey Tailin Liang C. Glossner Lei Wang Shaobo Shi Xiaotong Zhang MQ 124 671 0 24 Jan 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 228 4,460 0 23 Jan 2020
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,194 0 01 Sep 2014