Trained on 100 million words and still in shape: BERT meets British National Corpus

17 March 2023

Papers citing "Trained on 100 million words and still in shape: BERT meets British National Corpus"

26 / 26 papers shown

Title
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora Alex Warstadt Aaron Mueller Leshem Choshen E. Wilcox Chengxu Zhuang ... Rafael Mosquera Bhargavi Paranjape Adina Williams Tal Linzen Ryan Cotterell 38 106 0 10 Apr 2025
Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance Nirvan Patil Malhar Abhay Inamdar Agnivo Gosai Guruprasad Pathak Anish Joshi Aryan Sagavekar Anish Joshirao Raj Abhijit Dandekar Rajat Dandekar Sreedath Panat 41 0 0 07 Apr 2025
Named Entity Recognition in Context Colin Brisson Ayoub Kahfy Marc Bui Frédéric Constant 54 0 0 26 Mar 2025
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies Laurie Burchell Ona de Gibert Nikolay Arefyev Mikko Aulamo Marta Bañón ... Pavel Stepachev and Jörg Tiedemann Dušan Variš Tereza Vojtěchová Jaume Zaragoza-Bernabeu 43 1 0 13 Mar 2025
BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training Nikitas Theodoropoulos Giorgos Filandrianos Vassilis Lyberatos Maria Lymperaiou Giorgos Stamou SyDa 52 1 0 24 Feb 2025
BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context Alexis Matzopoulos Charl Hendriks Hishaam Mahomed Francois Meyer 28 0 0 08 Jan 2025
GPT or BERT: why not both? Lucas Georges Gabriel Charpentier David Samuel 47 5 0 31 Dec 2024
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora Michael Y. Hu Aaron Mueller Candace Ross Adina Williams Tal Linzen Chengxu Zhuang Ryan Cotterell Leshem Choshen Alex Warstadt Ethan Gotlieb Wilcox 91 7 0 06 Dec 2024
AntLM: Bridging Causal and Masked Language Models Xinru Yu Bin Guo Shiwei Luo J. Wang Tao Ji Yuanbin Wu CLL 77 1 0 04 Dec 2024
When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets? Srikrishna Iyer FedML 64 0 0 25 Nov 2024
From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes Zébulon Goriely Richard Diehl Martinez Andrew Caines Lisa Beinborn P. Buttery CLL 42 5 0 30 Oct 2024
BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data J. Tastet I. Timiryasov 18 4 0 25 Sep 2024
A comparison of correspondence analysis with PMI-based word embedding methods Qianqian Qi D. Hessen Peter G. M. van der Heijden 18 0 0 31 May 2024
More Room for Language: Investigating the Effect of Retrieval on Language Models David Samuel Lucas Georges Gabriel Charpentier Sondre Wold LRM RALM KELM 28 1 0 16 Apr 2024
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus Leshem Choshen Ryan Cotterell Michael Y. Hu Tal Linzen Aaron Mueller Candace Ross Alex Warstadt Ethan Gotlieb Wilcox Adina Williams Chengxu Zhuang 26 22 0 09 Apr 2024
Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations Wolf Nuyts Ruben Cartuyvels Marie-Francine Moens 34 1 0 25 Jan 2024
CLIMB: Curriculum Learning for Infant-inspired Model Building Richard Diehl Martinez Zébulon Goriely Hope McGovern Christopher Davis Andrew Caines P. Buttery Lisa Beinborn 17 10 0 15 Nov 2023
Not all layers are equally as important: Every Layer Counts BERT Lucas Georges Gabriel Charpentier David Samuel 13 15 0 03 Nov 2023
Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings David Samuel 11 2 0 30 Oct 2023
Tokenization with Factorized Subword Encoding David Samuel Lilja Øvrelid 25 1 0 13 Jun 2023
NorBench -- A Benchmark for Norwegian Language Models David Samuel Andrey Kutuzov Samia Touileb Erik Velldal Lilja Ovrelid Egil Rønningstad Elina Sigdel Anna Palatkina 21 23 0 06 May 2023
Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics Prajjwal Bhargava Aleksandr Drozd Anna Rogers 87 101 0 04 Oct 2021
Probing Classifiers: Promises, Shortcomings, and Advances Yonatan Belinkov 224 404 0 24 Feb 2021
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? Tal Linzen 218 188 0 03 May 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,943 0 20 Apr 2018
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,740 0 26 Sep 2016