XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

12 April 2020

Papers citing "XtremeDistil: Multi-stage Distillation for Massive Multilingual Models"

12 / 12 papers shown

Title
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training Nitay Calderon Subhabrata Mukherjee Roi Reichart Amir Kantor 31 17 0 03 May 2023
Distillation of encoder-decoder transformers for sequence labelling M. Farina D. Pappadopulo Anant Gupta Leslie Huang Ozan Irsoy Thamar Solorio VLM 103 3 0 10 Feb 2023
In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models Yukun Huang Yanda Chen Zhou Yu Kathleen McKeown 18 30 0 20 Dec 2022
HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crisis Response Selim Fekih Nicolò Tamagnone Benjamin Minixhofer R. Shrestha Ximena Contla Ewan Oglethorpe Navid Rekabsaz 11 6 0 10 Oct 2022
Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems Yoshitomo Matsubara Luca Soldaini Eric Lind Alessandro Moschitti 21 6 0 15 Jan 2022
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation Subhabrata Mukherjee Ahmed Hassan Awadallah Jianfeng Gao 17 22 0 08 Jun 2021
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers Wenhui Wang Hangbo Bao Shaohan Huang Li Dong Furu Wei MQ 19 255 0 31 Dec 2020
Rethinking embedding coupling in pre-trained language models Hyung Won Chung Thibault Févry Henry Tsai Melvin Johnson Sebastian Ruder 93 142 0 24 Oct 2020
Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor Xinyu Wang Yong-jia Jiang Zhaohui Yan Zixia Jia Nguyen Bach Tao Wang Zhongqiang Huang Fei Huang Kewei Tu 26 10 0 10 Oct 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers Wenhui Wang Furu Wei Li Dong Hangbo Bao Nan Yang Ming Zhou VLM 45 1,198 0 25 Feb 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,817 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,956 0 20 Apr 2018