Mask More and Mask Later: Efficient Pre-training of Masked Language
Models by Disentangling the [MASK] Token

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token

9 November 2022

Baohao Liao

Sanjika Hewavitharana

Christof Monz

Papers citing "Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token"

12 / 12 papers shown

Title
Emerging Property of Masked Token for Effective Pre-training Hyesong Choi Hunsang Lee Seyoung Joung Hyejin Park Jiyeong Kim Dongbo Min 20 9 0 12 Apr 2024
ApiQ: Finetuning of 2-Bit Quantized Large Language Model Baohao Liao Christian Herold Shahram Khadivi Christof Monz CLL MQ 39 12 0 07 Feb 2024
Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning Baohao Liao Shaomu Tan Christof Monz KELM 14 28 0 01 Jun 2023
Parameter-Efficient Fine-Tuning without Introducing New Latency Baohao Liao Yan Meng Christof Monz 13 48 0 26 May 2023
Representation Deficiency in Masked Language Modeling Yu Meng Jitin Krishnan Sinong Wang Qifan Wang Yuning Mao Han Fang Marjan Ghazvininejad Jiawei Han Luke Zettlemoyer 50 7 0 04 Feb 2023
Should You Mask 15% in Masked Language Modeling? Alexander Wettig Tianyu Gao Zexuan Zhong Danqi Chen CVBM 17 160 0 16 Feb 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 258 7,412 0 11 Nov 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 239 642 0 21 Apr 2021
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 221 197 0 07 Feb 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,815 0 17 Sep 2019
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives Elena Voita Rico Sennrich Ivan Titov 188 181 0 03 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018