Patient Knowledge Distillation for BERT Model Compression

25 August 2019

Papers citing "Patient Knowledge Distillation for BERT Model Compression"

50 / 491 papers shown

Title
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression Chenhe Dong Yaliang Li Ying Shen Minghui Qiu VLM 30 7 0 16 Oct 2021
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora Xisen Jin Dejiao Zhang Henghui Zhu Wei Xiao Shang-Wen Li Xiaokai Wei Andrew O. Arnold Xiang Ren KELM CLL 23 110 0 16 Oct 2021
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher Mehdi Rezagholizadeh A. Jafari Puneeth Salad Pranav Sharma Ali Saheb Pasand A. Ghodsi 71 17 0 16 Oct 2021
A Short Study on Compressing Decoder-Based Language Models Tianda Li Yassir El Mesbahi I. Kobyzev Ahmad Rashid A. Mahmud Nithin Anchuri Habib Hajimolahoseini Yang Liu Mehdi Rezagholizadeh 86 25 0 16 Oct 2021
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding Mengnan Du Subhabrata Mukherjee Yu Cheng Milad Shokouhi Xia Hu Ahmed Hassan Awadallah 44 13 0 16 Oct 2021
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm Shaoyi Huang Dongkuan Xu Ian En-Hsu Yen Yijue Wang Sung-En Chang ... Shiyang Chen Mimi Xie Sanguthevar Rajasekaran Hang Liu Caiwen Ding CLL VLM 21 29 0 15 Oct 2021
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline Xiangyang Liu Tianxiang Sun Junliang He Jiawen Wu Lingling Wu Xinyu Zhang Hao Jiang Zhao Cao Xuanjing Huang Xipeng Qiu ELM 23 46 0 13 Oct 2021
ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training Hui-Po Wang Sebastian U. Stich Yang He Mario Fritz FedML AI4CE 28 46 0 11 Oct 2021
SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions Vinod Ganesan Gowtham Ramesh Pratyush Kumar 31 9 0 10 Oct 2021
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 19 117 0 05 Oct 2021
Towards Efficient Post-training Quantization of Pre-trained Language Models Haoli Bai Lu Hou Lifeng Shang Xin Jiang Irwin King M. Lyu MQ 73 47 0 30 Sep 2021
Improving Question Answering Performance Using Knowledge Distillation and Active Learning Yasaman Boreshban Seyed Morteza Mirbostani Gholamreza Ghassem-Sani Seyed Abolghasem Mirroshandel Shahin Amiriparian 24 15 0 26 Sep 2021
Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better Xuanyang Zhang X. Zhang Jian-jun Sun 23 1 0 26 Sep 2021
DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference Cristobal Eyzaguirre Felipe del-Rio Vladimir Araujo Alvaro Soto 8 7 0 24 Sep 2021
Dynamic Knowledge Distillation for Pre-trained Language Models Lei Li Yankai Lin Shuhuai Ren Peng Li Jie Zhou Xu Sun 18 49 0 23 Sep 2021
Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing Haoyu He Xingjian Shi Jonas W. Mueller Zha Sheng Mu Li George Karypis 8 9 0 23 Sep 2021
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation Md. Akmal Haidar Nithin Anchuri Mehdi Rezagholizadeh Abbas Ghaddar Philippe Langlais Pascal Poupart 31 22 0 21 Sep 2021
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications Shuo Sun Ahmed El-Kishky Vishrav Chaudhary James Cross Francisco Guzmán Lucia Specia 21 1 0 17 Sep 2021
General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings Lukas Galke Isabelle Cuber Christophe Meyer Henrik Ferdinand Nolscher Angelina Sonderecker A. Scherp 28 2 0 17 Sep 2021
Distilling Linguistic Context for Language Model Compression Geondo Park Gyeongman Kim Eunho Yang 45 37 0 17 Sep 2021
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation Chenhe Dong Guangrun Wang Hang Xu Jiefeng Peng Xiaozhe Ren Xiaodan Liang 16 28 0 15 Sep 2021
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation Marzieh S. Tahaei Ella Charlaix V. Nia A. Ghodsi Mehdi Rezagholizadeh 41 22 0 13 Sep 2021
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding Tianda Li Ahmad Rashid A. Jafari Pranav Sharma A. Ghodsi Mehdi Rezagholizadeh AAML 25 5 0 13 Sep 2021
FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks Chen Liu Mengchao Zhang Liang Pang J. Guo Xueqi Cheng CLIP 21 19 0 12 Sep 2021
Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification Varsha Suresh Desmond C. Ong VLM 63 78 0 12 Sep 2021
Block Pruning For Faster Transformers François Lagunas Ella Charlaix Victor Sanh Alexander M. Rush VLM 16 218 0 10 Sep 2021
Learning to Teach with Student Feedback Yitao Liu Tianxiang Sun Xipeng Qiu Xuanjing Huang VLM 13 6 0 10 Sep 2021
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression Canwen Xu Wangchunshu Zhou Tao Ge Kelvin J. Xu Julian McAuley Furu Wei 10 41 0 07 Sep 2021
FedKD: Communication Efficient Federated Learning via Knowledge Distillation Chuhan Wu Fangzhao Wu Lingjuan Lyu Yongfeng Huang Xing Xie FedML 15 370 0 30 Aug 2021
Analyzing and Mitigating Interference in Neural Architecture Search Jin Xu Xu Tan Kaitao Song Renqian Luo Yichong Leng Tao Qin Tie-Yan Liu Jian Li MoMe 26 29 0 29 Aug 2021
Design and Scaffolded Training of an Efficient DNN Operator for Computer Vision on the Edge Vinod Ganesan Pratyush Kumar 34 2 0 25 Aug 2021
YANMTT: Yet Another Neural Machine Translation Toolkit Raj Dabre Eiichiro Sumita 31 13 0 25 Aug 2021
Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenches Leonard Dahlmann Tomer Lancewicki MQ 25 0 0 23 Aug 2021
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing Katikapalli Subramanyam Kalyan A. Rajasekharan S. Sangeetha VLM LM&MA 26 258 0 12 Aug 2021
Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data Kuluhan Binici N. Pham T. Mitra K. Leman 17 40 0 11 Aug 2021
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models Yichun Yin Cheng Chen Lifeng Shang Xin Jiang Xiao Chen Qun Liu VLM 17 50 0 29 Jul 2021
Go Wider Instead of Deeper Fuzhao Xue Ziji Shi Futao Wei Yuxuan Lou Yong Liu Yang You ViT MoE 17 80 0 25 Jul 2021
Follow Your Path: a Progressive Method for Knowledge Distillation Wenxian Shi Yuxuan Song Hao Zhou Bohan Li Lei Li 17 14 0 20 Jul 2021
Federated Action Recognition on Heterogeneous Embedded Devices Pranjali Jain Shreyas Goenka S. Bagchi Biplab Banerjee Somali Chaterji FedML 43 7 0 18 Jul 2021
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search Lei-tai Chen Fajie Yuan Jiaxi Yang Min Yang Chengming Li 11 3 0 15 Jul 2021
Trustworthy AI: A Computational Perspective Haochen Liu Yiqi Wang Wenqi Fan Xiaorui Liu Yaxin Li Shaili Jain Yunhao Liu Anil K. Jain Jiliang Tang FaML 98 196 0 12 Jul 2021
A Flexible Multi-Task Model for BERT Serving Tianwen Wei Jianwei Qi Shenghuang He 26 7 0 12 Jul 2021
Investigation of Practical Aspects of Single Channel Speech Separation for ASR Jian Wu Zhuo Chen Sanyuan Chen Yu-Huan Wu Takuya Yoshioka Naoyuki Kanda Shujie Liu Jinyu Li 22 17 0 05 Jul 2021
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation Zhiwei Hao Jianyuan Guo Ding Jia Kai Han Yehui Tang Chao Zhang Dacheng Tao Yunhe Wang ViT 33 68 0 03 Jul 2021
Learned Token Pruning for Transformers Sehoon Kim Sheng Shen D. Thorsley A. Gholami Woosuk Kwon Joseph Hassoun Kurt Keutzer 9 145 0 02 Jul 2021
Knowledge Distillation for Quality Estimation Amit Gajbhiye M. Fomicheva Fernando Alva-Manchego Frédéric Blain A. Obamuyide Nikolaos Aletras Lucia Specia 19 11 0 01 Jul 2021
Elbert: Fast Albert with Confidence-Window Based Early Exit Keli Xie Siyuan Lu Meiqi Wang Zhongfeng Wang 14 20 0 01 Jul 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 37 813 0 14 Jun 2021
Why Can You Lay Off Heads? Investigating How BERT Heads Transfer Ting-Rui Chiang Yun-Nung Chen 26 0 0 14 Jun 2021
Generate, Annotate, and Learn: NLP with Synthetic Text Xuanli He Islam Nassar J. Kiros Gholamreza Haffari Mohammad Norouzi 31 51 0 11 Jun 2021