How Good is Your Tokenizer? On the Monolingual Performance of
Multilingual Language Models

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

31 December 2020

Sebastian Ruder

Papers citing "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models"

17 / 17 papers shown

Title
Token-free Models for Sarcasm Detection Sumit Mamtani Maitreya Sonawane Kanika Agarwal Nishanth Sanjeev 34 0 0 02 May 2025
Modes of Sequence Models and Learning Coefficients Zhongtian Chen Daniel Murfet 60 1 0 25 Apr 2025
Large Language Models in Numberland: A Quick Test of Their Numerical Reasoning Abilities Roussel Rahman ReLM ELM LRM 46 0 0 31 Mar 2025
Cross-Tokenizer Distillation via Approximate Likelihood Matching Benjamin Minixhofer Ivan Vulić E. Ponti 44 0 0 25 Mar 2025
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation Langlin Huang Mengyu Bu Yang Feng 18 0 0 03 Nov 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 31 0 0 07 Oct 2024
Smirk: An Atomically Complete Tokenizer for Molecular Foundation Models Alexius Wadell Anoushka Bhutani Venkatasubramanian Viswanathan 18 0 0 19 Sep 2024
A Principled Framework for Evaluating on Typologically Diverse Languages Esther Ploeger Wessel Poelman Andreas Holck Høeg-Petersen Anders Schlichtkrull Miryam de Lhoneux Johannes Bjerva 23 1 0 06 Jul 2024
From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences Prashant Kodali Anmol Goel Likhith Asapu Vamshi Bonagiri Anirudh Govil Monojit Choudhury Manish Shrivastava Ponnurangam Kumaraguru 34 0 0 09 May 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 35 17 0 28 Feb 2024
CroissantLLM: A Truly Bilingual French-English Language Model Manuel Faysse Patrick Fernandes Nuno M. Guerreiro António Loison Duarte M. Alves ... François Yvon André F.T. Martins Gautier Viaud C´eline Hudelot Pierre Colombo 29 33 0 01 Feb 2024
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP Sabrina J. Mielke Zaid Alyafeai Elizabeth Salesky Colin Raffel Manan Dey ... Arun Raja Chenglei Si Wilson Y. Lee Benoît Sagot Samson Tan 10 137 0 20 Dec 2021
Improving Multilingual Models with Language-Clustered Vocabularies Hyung Won Chung Dan Garrette Kiat Chuan Tan Jason Riesa VLM 50 56 0 24 Oct 2020
What the [MASK]? Making Sense of Language-Specific BERT Models Debora Nozza Federico Bianchi Dirk Hovy 63 94 0 05 Mar 2020
SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis Pavel Efimov Andrey Chertok Leonid Boytsov Pavel Braslavski 50 56 0 20 Dec 2019
MLQA: Evaluating Cross-lingual Extractive Question Answering Patrick Lewis Barlas Oğuz Ruty Rinott Sebastian Riedel Holger Schwenk ELM 239 489 0 16 Oct 2019
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,435 0 26 Sep 2016