Lifting the Curse of Multilinguality by Pre-training Modular Transformers

12 May 2022

Xian Li

Papers citing "Lifting the Curse of Multilinguality by Pre-training Modular Transformers"

50 / 116 papers shown

Title
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models Divyanshu Aggarwal Ashutosh Sathe Ishaan Watts Sunayana Sitaram 14 1 0 15 Jan 2024
The Obscure Limitation of Modular Multilingual Language Models Muhammad Farid Adilazuarda Samuel Cahyawijaya Ayu Purwarianti 11 6 0 21 Nov 2023
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning Clifton A. Poth Hannah Sterz Indraneil Paul Sukannya Purkayastha Leon Arne Engländer Timo Imhof Ivan Vulić Sebastian Ruder Iryna Gurevych Jonas Pfeiffer 17 45 0 18 Nov 2023
To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages Benedikt Ebing Goran Glavas 19 1 0 15 Nov 2023
Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks Rochelle Choenni Ekaterina Shutova Daniel H Garrette 14 8 0 14 Nov 2023
Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures Julius Steuer Marius Mosbach Dietrich Klakow 17 10 0 08 Nov 2023
Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts Thomas Palmeira Ferraz Marcely Zanon Boito Caroline Brun Vassilina Nikoulina 13 12 0 02 Nov 2023
CUNI Submission to MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrieval Jindvrich Helcl Jindvrich Libovický LRM 11 0 0 25 Oct 2023
Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks Sunit Bhattacharya Ondrej Bojar 14 7 0 24 Oct 2023
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging Joel Jang Seungone Kim Bill Yuchen Lin Yizhong Wang Jack Hessel Luke Zettlemoyer Hannaneh Hajishirzi Yejin Choi Prithviraj Ammanabrolu MoMe 21 130 0 17 Oct 2023
SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects David Ifeoluwa Adelani Hannah Liu Xiaoyu Shen Nikita Vassilyev Jesujoba Oluwadara Alabi Yanke Mao Haonan Gao Annie En-Shiun Lee ELM 14 59 0 14 Sep 2023
Multilingual Text Representation Fahim Faisal 16 0 0 02 Sep 2023
Task-Based MoE for Multitask Multilingual Machine Translation Hai Pham Young Jin Kim Subhabrata Mukherjee David P. Woodruff Barnabás Póczós Hany Awadalla MoE 16 4 0 30 Aug 2023
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models Sara Babakniya A. Elkordy Yahya H. Ezzeldin Qingfeng Liu Kee-Bong Song Mostafa El-Khamy Salman Avestimehr 11 31 0 12 Aug 2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore Sewon Min Suchin Gururangan Eric Wallace Hannaneh Hajishirzi Noah A. Smith Luke Zettlemoyer AILaw 14 63 0 08 Aug 2023
Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing Tom Sherborne Tom Hosking Mirella Lapata OT 11 4 0 09 Jul 2023
Improving Language Plasticity via Pretraining with Active Forgetting Yihong Chen Kelly Marchisio Roberta Raileanu David Ifeoluwa Adelani Pontus Stenetorp Sebastian Riedel Mikel Artetx KELM AI4CE CLL 15 23 0 03 Jul 2023
Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability Jiacheng Ye Xijia Tao Lingpeng Kong LRM 20 22 0 11 Jun 2023
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification Inigo Jauregi Unanue Gholamreza Haffari Massimo Piccardi VLM 13 8 0 08 Jun 2023
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations Yusen Zhang J. Wang Zhiguo Wang Rui Zhang VLM 19 9 0 07 Jun 2023
Soft Merging of Experts with Adaptive Routing Mohammed Muqeeth Haokun Liu Colin Raffel MoMe MoE 9 44 0 06 Jun 2023
Distilling Efficient Language-Specific Models for Cross-Lingual Transfer Alan Ansell E. Ponti Anna Korhonen Ivan Vulić 17 4 0 02 Jun 2023
Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation Benjamin Minixhofer Jonas Pfeiffer Ivan Vulić 13 16 0 30 May 2023
Emergent Modularity in Pre-trained Transformers Zhengyan Zhang Zhiyuan Zeng Yankai Lin Chaojun Xiao Xiaozhi Wang Xu Han Zhiyuan Liu Ruobing Xie Maosong Sun Jie Zhou MoE 29 23 0 28 May 2023
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation Marius Mosbach Tiago Pimentel Shauli Ravfogel Dietrich Klakow Yanai Elazar 43 119 0 26 May 2023
An Efficient Multilingual Language Model Compression through Vocabulary Trimming Asahi Ushio Yi Zhou Jose Camacho-Collados 19 7 0 24 May 2023
LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages M. Agarwal Md Mahfuz Ibn Alam Antonios Anastasopoulos 23 5 0 23 May 2023
mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations Jonas Pfeiffer Francesco Piccinno Massimo Nicosia Xinyi Wang Machel Reid Sebastian Ruder VLM LRM 21 27 0 23 May 2023
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models Aitor Ormazabal Mikel Artetxe Eneko Agirre 17 19 0 23 May 2023
mPMR: A Multilingual Pre-trained Machine Reader at Scale Weiwen Xu Xin Li Wai Lam Lidong Bing LRM 15 5 0 23 May 2023
Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis Seraphina Goldfarb-Tarrant Bjorn Ross Adam Lopez 16 7 0 22 May 2023
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages Ayyoob Imani Peiqin Lin Amir Hossein Kargaran Silvia Severini Masoud Jalili Sabet ... Chunlan Ma Helmut Schmid André F. T. Martins François Yvon Hinrich Schütze ALM LRM 23 95 0 20 May 2023
Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks Anas Himmi Ekhine Irurozki Nathan Noiry Stéphan Clémençon Pierre Colombo 6 5 0 17 May 2023
Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Models Shangbin Feng Weijia Shi Yuyang Bai Vidhisha Balachandran Tianxing He Yulia Tsvetkov KELM 45 28 0 17 May 2023
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining Hyung Won Chung Noah Constant Xavier Garcia Adam Roberts Yi Tay Sharan Narang Orhan Firat 8 49 0 18 Apr 2023
Romanization-based Large-scale Adaptation of Multilingual Language Models Sukannya Purkayastha Sebastian Ruder Jonas Pfeiffer Iryna Gurevych Ivan Vulić 10 10 0 18 Apr 2023
Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese Vésteinn Snaebjarnarson A. Simonsen Goran Glavavs Ivan Vulić 19 19 0 18 Apr 2023
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Stella Biderman Hailey Schoelkopf Quentin G. Anthony Herbie Bradley Kyle O'Brien ... USVSN Sai Prashanth Edward Raff Aviya Skowron Lintang Sutawika Oskar van der Wal 13 1,160 0 03 Apr 2023
Scaling Expert Language Models with Unsupervised Domain Discovery Suchin Gururangan Margaret Li M. Lewis Weijia Shi Tim Althoff Noah A. Smith Luke Zettlemoyer MoE 8 46 0 24 Mar 2023
SwissBERT: The Multilingual Language Model for Switzerland Jannis Vamvas Johannes Graen Rico Sennrich 20 6 0 23 Mar 2023
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer Shanu Kumar Abbaraju Soujanya Sandipan Dandapat Sunayana Sitaram Monojit Choudhury VLM 17 1 0 04 Mar 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić E. Ponti MoMe OOD 13 73 0 22 Feb 2023
Exploring the Benefits of Training Expert Language Models over Instruction Tuning Joel Jang Seungone Kim Seonghyeon Ye Doyoung Kim Lajanugen Logeswaran Moontae Lee Kyungjae Lee Minjoon Seo LRM ALM 11 79 0 07 Feb 2023
AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning Han Zhou Xingchen Wan Ivan Vulić Anna Korhonen 11 45 0 28 Jan 2023
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training Kelly Marchisio Patrick Lewis Yihong Chen Mikel Artetxe 14 16 0 20 Dec 2022
NusaCrowd: Open Source Initiative for Indonesian NLP Resources Samuel Cahyawijaya Holy Lovenia Alham Fikri Aji Genta Indra Winata Bryan Wilie ... Timothy Baldwin Sebastian Ruder Herry Sujaini S. Sakti Ayu Purwarianti 13 47 0 19 Dec 2022
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting Zheng-Xin Yong Hailey Schoelkopf Niklas Muennighoff Alham Fikri Aji David Ifeoluwa Adelani ... Genta Indra Winata Stella Biderman Edward Raff Dragomir R. Radev Vassilina Nikoulina CLL VLM AI4CE LRM 27 81 0 19 Dec 2022
Domain Adaptation of Transformer-Based Models using Unlabeled Data for Relevance and Polarity Classification of German Customer Feedback Ahmad Idrissi-Yaghir Henning Schafer Nadja Bauer Christoph M. Friedrich 17 6 0 12 Dec 2022
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models Isabel Papadimitriou Kezia Lopez Daniel Jurafsky 14 0 0 11 Oct 2022
Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation Alexandra Chronopoulou Dario Stojanovski Alexander M. Fraser 13 17 0 30 Sep 2022