Learning Better Masking for Better Language Model Pre-training

23 August 2022

Papers citing "Learning Better Masking for Better Language Model Pre-training"

10 / 10 papers shown

Title
EuroBERT: Scaling Multilingual Encoders for European Languages Nicolas Boizard Hippolyte Gisserot-Boukhlef Duarte M. Alves André F. T. Martins Ayoub Hammal ... Maxime Peyrard Nuno M. Guerreiro Patrick Fernandes Ricardo Rei Pierre Colombo 96 1 0 07 Mar 2025
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text Andrei Jarca Florinel-Alin Croitoru Radu Tudor Ionescu 48 0 0 18 Feb 2025
KidLM: Advancing Language Models for Children -- Early Insights and Future Directions Mir Tafseer Nayeem Davood Rafiei ALM 31 3 0 04 Oct 2024
Unlocking Efficiency: Adaptive Masking for Gene Transformer Models Soumyadeep Roy S. Sural Niloy Ganguly MedIm 30 0 0 13 Aug 2024
Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics Anas Belfathi Ygor Gallina Nicolas Hernandez Richard Dufour Laura Monceaux 16 1 0 19 Feb 2024
Reformulating NLP tasks to Capture Longitudinal Manifestation of Language Disorders in People with Dementia Dimitris Gkoumas Matthew Purver M. Liakata 8 1 0 15 Oct 2023
GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning Soumyadeep Roy Jonas Wallat Sowmya S. Sundaram Wolfgang Nejdl Niloy Ganguly 20 3 0 29 Jul 2023
Dynamic Masking Rate Schedules for MLM Pretraining Zachary Ankner Naomi Saphra Davis W. Blalock Jonathan Frankle Matthew L. Leavitt 19 5 0 24 May 2023
Dict-BERT: Enhancing Language Model Pre-training with Dictionary W. Yu Chenguang Zhu Yuwei Fang Donghan Yu Shuohang Wang Yichong Xu Michael Zeng Meng-Long Jiang 45 64 0 13 Oct 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018