MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

6 April 2020

Papers citing "MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices"

50 / 145 papers shown

Title
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method Mihyeon Kim Juhyoung Park Youngbin Kim 34 0 0 11 May 2025
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Zhilin Wang Shi Wang Qianqian Xu Q. Huang 42 0 0 07 May 2025
FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation Chaitali Bhattacharyya Yeseong Kim 45 0 0 01 May 2025
The Rise of Small Language Models in Healthcare: A Comprehensive Survey Muskan Garg Shaina Raza Shebuti Rayana Xingyi Liu Sunghwan Sohn LM&MA AILaw 92 0 0 23 Apr 2025
FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Framework S M Sarwar AI4MH 46 0 0 27 Feb 2025
A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images N. Shvetsov T. Kilvaer M. Tafavvoghi Anders Sildnes Kajsa Møllersen Lill-ToveRasmussen Busund L. A. Bongo VLM 71 1 0 26 Feb 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 75 14 0 28 Jan 2025
HadamRNN: Binary and Sparse Ternary Orthogonal RNNs Armand Foucault Franck Mamalet François Malgouyres MQ 74 0 0 28 Jan 2025
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance Chu Myaet Thwal Ye Lin Tun Minh N. H. Nguyen Eui-nam Huh Choong Seon Hong VLM 74 0 0 05 Dec 2024
SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments Syed Abdul Gaffar Shakhadri Kruthika KR Rakshit Aralimatti VLM 20 1 0 15 Oct 2024
On Importance of Pruning and Distillation for Efficient Low Resource NLP Aishwarya Mirashi Purva Lingayat Srushti Sonavane Tejas Padhiyar Raviraj Joshi Geetanjali Kale 31 1 0 21 Sep 2024
NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radar Runwei Guan Jianan Liu Liye Jia Haocheng Zhao Shanliang Yao Xiaohui Zhu Ka Lok Man Eng Gee Lim Jeremy S. Smith Yutao Yue 63 5 0 30 Aug 2024
Accelerating Large Language Model Inference with Self-Supervised Early Exits Florian Valade LRM 44 1 0 30 Jul 2024
LPViT: Low-Power Semi-structured Pruning for Vision Transformers Kaixin Xu Zhe Wang Chunyun Chen Xue Geng Jie Lin Xulei Yang Min-man Wu Min Wu Xiaoli Li Weisi Lin ViT VLM 51 7 0 02 Jul 2024
Factual Dialogue Summarization via Learning from Large Language Models Rongxin Zhu Jey Han Lau Jianzhong Qi HILM 52 1 0 20 Jun 2024
Fast Vocabulary Transfer for Language Model Compression Leonidas Gee Andrea Zugarini Leonardo Rigutini Paolo Torroni 35 26 0 15 Feb 2024
DE $^3$ -BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks Jianing He Qi Zhang Weiping Ding Duoqian Miao Jun Zhao Liang Hu LongBing Cao 38 3 0 03 Feb 2024
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality Huy Q. Le Chu Myaet Thwal Yu Qiao Ye Lin Tun Minh N. H. Nguyen Choong Seon Hong Choong Seon Hong 65 4 0 25 Jan 2024
DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization Rahul Chand Yashoteja Prabhu Pratyush Kumar 20 3 0 20 Dec 2023
A Comparative Analysis of Pretrained Language Models for Text-to-Speech M. G. Moya Panagiota Karanasou S. Karlapati Bastian Schnell Nicole Peinelt Alexis Moinet Thomas Drugman 37 3 0 04 Sep 2023
Mobile Foundation Model as Firmware Jinliang Yuan Chenchen Yang Dongqi Cai Shihe Wang Xin Yuan ... Di Zhang Hanzi Mei Xianqing Jia Shangguang Wang Mengwei Xu 40 19 0 28 Aug 2023
Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models Seungcheol Park Ho-Jin Choi U. Kang VLM 40 5 0 07 Aug 2023
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing Yelysei Bondarenko Markus Nagel Tijmen Blankevoort MQ 15 88 0 22 Jun 2023
Breaking On-device Training Memory Wall: A Systematic Survey Shitian Li Chunlin Tian Kahou Tam Ruirui Ma Li Li 21 2 0 17 Jun 2023
FedMultimodal: A Benchmark For Multimodal Federated Learning Tiantian Feng Digbalay Bose Tuo Zhang Rajat Hebbar Anil Ramakrishna Rahul Gupta Mi Zhang Salman Avestimehr Shrikanth Narayanan 32 48 0 15 Jun 2023
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model Shicheng Tan Weng Lam Tam Yuanchun Wang Wenwen Gong Yang Yang ... Jiahao Liu Jingang Wang Shuo Zhao Peng-Zhen Zhang Jie Tang ALM MoE 30 11 0 11 Jun 2023
Just CHOP: Embarrassingly Simple LLM Compression A. Jha Tom Sherborne Evan Pete Walsh Dirk Groeneveld Emma Strubell Iz Beltagy 28 3 0 24 May 2023
Can Public Large Language Models Help Private Cross-device Federated Learning? Wei Ping Yibo Jacky Zhang Yuan Cao Bo-wen Li H. B. McMahan Sewoong Oh Zheng Xu Manzil Zaheer FedML 29 37 0 20 May 2023
Lifting the Curse of Capacity Gap in Distilling Language Models Chen Zhang Yang Yang Jiahao Liu Jingang Wang Yunsen Xian Benyou Wang Dawei Song MoE 32 19 0 20 May 2023
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation Yuxin Ren Zi-Qi Zhong Xingjian Shi Yi Zhu Chun Yuan Mu Li 24 7 0 16 May 2023
Word Sense Induction with Knowledge Distillation from BERT Anik Saha Alex Gittens B. Yener 12 1 0 20 Apr 2023
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model Xin Yao Ziqing Yang Yiming Cui Shijin Wang 23 3 0 03 Apr 2023
BERTino: an Italian DistilBERT model Matteo Muffo E. Bertino VLM 16 14 0 31 Mar 2023
oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes Daniel Fernando Campos Alexandre Marques Mark Kurtz Chengxiang Zhai VLM AAML 13 2 0 30 Mar 2023
EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms Shikhar Tuli N. Jha 36 3 0 24 Mar 2023
Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings Ulf A. Hamster Ji-Ung Lee Alexander Geyken Iryna Gurevych 21 0 0 13 Mar 2023
Greener yet Powerful: Taming Large Code Generation Models with Quantization Xiaokai Wei Sujan Kumar Gonugondla W. Ahmad Shiqi Wang Baishakhi Ray ... Ben Athiwaratkun Mingyue Shang M. K. Ramanathan Parminder Bhatia Bing Xiang MQ 28 6 0 09 Mar 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 34 20 0 07 Mar 2023
DTW-SiameseNet: Dynamic Time Warped Siamese Network for Mispronunciation Detection and Correction R. Anantha Kriti Bhasin Daniela Aguilar Prabal Vashisht Becci Williamson Srinivas Chappidi 19 0 0 01 Mar 2023
Full Stack Optimization of Transformer Inference: a Survey Sehoon Kim Coleman Hooper Thanakul Wattanawong Minwoo Kang Ruohan Yan ... Qijing Huang Kurt Keutzer Michael W. Mahoney Y. Shao A. Gholami MQ 36 101 0 27 Feb 2023
MUX-PLMs: Data Multiplexing for High-throughput Language Models Vishvak Murahari A. Deshpande Carlos E. Jimenez Izhak Shafran Mingqiu Wang Yuan Cao Karthik Narasimhan MoE 23 5 0 24 Feb 2023
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 27 24 0 19 Feb 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 33 10 0 13 Feb 2023
Lightweight Transformers for Clinical Natural Language Processing Omid Rohanian Mohammadmahdi Nouriborji Hannah Jauncey Samaneh Kouchaki Isaric Clinical Characterisation Group Lei A. Clifton L. Merson David A. Clifton MedIm LM&MA 16 12 0 09 Feb 2023
Bioformer: an efficient transformer language model for biomedical text mining Li Fang Qingyu Chen Chih-Hsuan Wei Zhiyong Lu Kai Wang MedIm AI4CE 32 18 0 03 Feb 2023
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective Jongwoo Ko Seungjoon Park Minchan Jeong S. Hong Euijai Ahn Duhyeuk Chang Se-Young Yun 23 6 0 03 Feb 2023
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection Chenglong Wang Yi Lu Yongyu Mu Yimin Hu Tong Xiao Jingbo Zhu 32 8 0 01 Feb 2023
Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Models Rui Zhu Di Tang Siyuan Tang Xiaofeng Wang Haixu Tang AAML FedML 34 13 0 09 Dec 2022
SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers A. Deshpande Md Arafat Sultan Anthony Ferritto A. Kalyan Karthik Narasimhan Avirup Sil MoE 33 1 0 29 Nov 2022
A Survey for Efficient Open Domain Question Answering Qin Zhang Shan Chen Dongkuan Xu Qingqing Cao Xiaojun Chen Trevor Cohn Meng Fang 28 33 0 15 Nov 2022