UniMax: Fairer and more Effective Language Sampling for Large-Scale
Multilingual Pretraining

UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

18 April 2023

Hyung Won Chung

Sharan Narang

Papers citing "UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining"

12 / 12 papers shown

Title
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models Xinlin Zhuang Jiahui Peng Ren Ma Y. Wang Tianyi Bai Xingjian Wei Jiantao Qiu Chi Zhang Ying Qian Conghui He 39 0 0 19 Apr 2025
SkyReels-V2: Infinite-length Film Generative Model Guibin Chen D. Lin Jiangping Yang Chunze Lin J. Zhu ... Di Qiu Debang Li Zhengcong Fei Yang Li Yahui Zhou DiffM VGen 47 1 0 17 Apr 2025
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets Tianjian Li Haoran Xu Weiting Tan Kenton Murray Daniel Khashabi 35 1 0 06 Oct 2024
YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation Sungkyun Chang Emmanouil Benetos Holger Kirchhoff Simon Dixon 24 2 0 05 Jul 2024
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences David C. Uthus Santiago Ontañón Joshua Ainslie Mandy Guo VLM 14 10 0 18 May 2023
Systematic Inequalities in Language Technology Performance across the World's Languages Damián E. Blasi Antonios Anastasopoulos Graham Neubig 103 130 0 13 Oct 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 237 588 0 14 Jul 2021
Larger-Scale Transformers for Multilingual Masked Language Modeling Naman Goyal Jingfei Du Myle Ott Giridhar Anantharaman Alexis Conneau 88 98 0 02 May 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 267 1,798 0 14 Dec 2020
Improving Multilingual Models with Language-Clustered Vocabularies Hyung Won Chung Dan Garrette Kiat Chuan Tan Jason Riesa VLM 58 65 0 24 Oct 2020
MLQA: Evaluating Cross-lingual Extractive Question Answering Patrick Lewis Barlas Oğuz Ruty Rinott Sebastian Riedel Holger Schwenk ELM 239 489 0 16 Oct 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 396 2,576 0 03 Sep 2019