XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

24 March 2020

Graham Neubig

Papers citing "XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization"

50 / 659 papers shown

Title
The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks Benedikt Ebing Goran Glavas 19 0 0 15 May 2025
What Causes Knowledge Loss in Multilingual Language Models? Maria Khelli Samuel Cahyawijaya Ayu Purwarianti Genta Indra Winata CLL 56 0 0 29 Apr 2025
Long-context Non-factoid Question Answering in Indic Languages Ritwik Mishra R. Shah Ponnurangam Kumaraguru 26 0 0 18 Apr 2025
Bias Beyond English: Evaluating Social Bias and Debiasing Methods in a Low-Resource Setting Ej Zhou Weiming Lu 26 0 0 15 Apr 2025
Cross-Document Cross-Lingual Natural Language Inference via RST-enhanced Graph Fusion and Interpretability Prediction Mengying Yuan Wangzi Xuan Fei Li 26 0 0 11 Apr 2025
Enhancing NER Performance in Low-Resource Pakistani Languages using Cross-Lingual Data Augmentation Toqeer Ehsan Thamar Solorio 101 0 0 07 Apr 2025
Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer Soumen Kumar Mondal Sayambhu Sen Abhishek Singhania P. Jyothi 31 0 0 21 Mar 2025
Fragile Mastery: Are Domain-Specific Trade-Offs Undermining On-Device Language Models? Basab Jha Firoj Paudel 37 0 0 16 Mar 2025
TLUE: A Tibetan Language Understanding Evaluation Benchmark Fan Gao Cheng Huang Nyima Tashi Xiangxiang Wang Thupten Tsering ... Gadeng Luosang Rinchen Dongrub Dorje Tashi Xiao Feng Yongbin Yu ELM 74 2 0 15 Mar 2025
LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama Naome A. Etori Kevin Lu Randu Karisa Arturs Kanepajs LRM ELM 107 0 0 14 Mar 2025
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models Catherine Arnett Tyler A. Chang J. Michaelov Benjamin Bergen 41 0 0 05 Mar 2025
MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages Chen Zhang Mingxu Tao Zhiyuan Liao Yansong Feng 36 0 0 03 Mar 2025
ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer Omer Goldman Uri Shaham Dan Malkin Sivan Eiger Avinatan Hassidim ... Shruti Rijhwani Laura Rimell Idan Szpektor Reut Tsarfaty Matan Eyal 42 3 0 28 Feb 2025
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs Linyang He Ercong Nie Sukru Samet Dindar Arsalan Firoozi Adrian Nicolas Florea ... Haotian Ye Jonathan R. Brennan Helmut Schmid Hinrich Schütze Nima Mesgarani 54 1 0 27 Feb 2025
Where Are We? Evaluating LLM Performance on African Languages Ife Adebara Hawau Olamide Toyin Nahom Tesfu Ghebremichael AbdelRahim Elmadany Muhammad Abdul-Mageed 52 0 0 26 Feb 2025
NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts Muhammad Farid Adilazuarda M. Wijanarko Lucky Susanto Khumaisa Nuráini Derry Wijaya Alham Fikri Aji 52 0 0 25 Feb 2025
Language Models' Factuality Depends on the Language of Inquiry Tushar Aggarwal Kumar Tanmay Ayush Agrawal Kumar Ayush Hamid Palangi Paul Pu Liang HILM KELM 71 0 0 25 Feb 2025
Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models Anirudh Sundar Sinead Williamson Katherine Metcalf B. Theobald Skyler Seto Masha Fedzechkina LLMSV 77 0 0 24 Feb 2025
Batayan: A Filipino NLP benchmark for evaluating Large Language Models Jann Railey Montalan Jimson Paulo Layacan David Demitri Africa Richell Isaiah Flores Michael T. Lopez II Theresa Denise Magsajo Anjanette Cayabyab William-Chandra Tjhi 39 0 0 19 Feb 2025
Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning Yilei Tu Andrew Xue Freda Shi 49 0 0 17 Feb 2025
A Large-Scale Benchmark for Vietnamese Sentence Paraphrases Sang Quang Nguyen Kiet Van Nguyen 60 0 0 11 Feb 2025
Beyond Literal Token Overlap: Token Alignability for Multilinguality Katharina Hämmerl Tomasz Limisiewicz Jindrich Libovický Alexander M. Fraser 43 0 0 10 Feb 2025
Revisiting Projection-based Data Transfer for Cross-Lingual Named Entity Recognition in Low-Resource Languages Andrei Politov Oleh Shkalikov René Jäkel Michael Färber 59 0 0 30 Jan 2025
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding Sankalp KJ Ashutosh Kumar Laxmaan Balaji Nikunj Kotecha Vinija Jain Aman Chadha S. Bhaduri ELM 100 1 0 27 Jan 2025
Can linguists better understand DNA? Wang Liang 82 1 0 20 Jan 2025
Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond Weiyu Chen Xiaoyuan Zhang Baijiong Lin Xi Victoria Lin Han Zhao Qingfu Zhang James T. Kwok 73 2 0 19 Jan 2025
LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning Hongbin Zhang K. Chen Xuefeng Bai Yang Xiang Min Zhang 79 0 0 17 Dec 2024
PolyIPA -- Multilingual Phoneme-to-Grapheme Conversion Model Davor Lauc 72 0 0 12 Dec 2024
SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languages Jia Guo Longxu Dou Guangtao Zeng Stanley Kok Wei Lu Qian Liu ELM LRM 70 1 0 02 Dec 2024
ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain Ali Shiraee Kasmaee Mohammad Khodadad Mohammad Arshi Saloot Nick Sherck Stephen Dokas H. Mahyar Soheila Samiee ELM 133 0 0 30 Nov 2024
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge Angelika Romanou Negar Foroutan Anna Sotnikova Zeming Chen Sree Harsha Nelaturu ... Mike Zhang Imanol Schlag Marzieh Fadaee Sara Hooker Antoine Bosselut ELM 105 6 0 29 Nov 2024
Dynamic Strategy Planning for Efficient Question Answering with Large Language Models Tanmay Parekh Pradyot Prakash Alexander Radovic Akshay Shekher Denis Savenkov LRM 51 1 0 30 Oct 2024
Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense Samuel Cahyawijaya Ruochen Zhang Holy Lovenia Jan Christian Blaise Cruz Elisa Gilbert Hiroki Nomoto Alham Fikri Aji LRM 30 0 0 28 Oct 2024
SandboxAQ's submission to MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval Isidora Chara Tourni Sayontan Ghosh Brenda Miao Constantijn van der Poel LRM 28 0 0 28 Oct 2024
Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code Jipeng Zhang Jianshu Zhang Yuanzhe Li Renjie Pi Rui Pan Runtao Liu Ziqiang Zheng Tong Zhang 36 0 0 24 Oct 2024
VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks Shailaja Keyur Sampat Mutsumi Nakamura Shankar Kailas Kartik Aggarwal Mandy Zhou Yezhou Yang Chitta Baral MLLM CoGe ReLM VLM LRM 29 0 0 17 Oct 2024
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs Sumanth Doddapaneni Mohammed Safi Ur Rahman Khan Dilip Venkatesh Raj Dabre Anoop Kunchukuttan Mitesh M. Khapra ELM 35 1 0 17 Oct 2024
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment Amir Hossein Kargaran Ali Modarressi Nafiseh Nikeghbal Jana Diesner François Yvon Hinrich Schütze ELM 44 3 0 08 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 31 2 0 07 Oct 2024
How Transliterations Improve Crosslingual Alignment Yihong Liu Mingyang Wang Amir Hossein Kargaran Ayyoob Imani Orgest Xhelili Haotian Ye Chunlan Ma François Yvon Hinrich Schütze 34 2 0 25 Sep 2024
Exposing Assumptions in AI Benchmarks through Cognitive Modelling Jonathan H. Rystrøm Kenneth C. Enevoldsen 32 0 0 25 Sep 2024
IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios Hai Lin Shaoxiong Zhan Junyou Su Haitao Zheng Hui Wang RALM 21 1 0 24 Sep 2024
XTRUST: On the Multilingual Trustworthiness of Large Language Models Yahan Li Yi Wang Yi-Ju Chang Yuan Wu HILM LRM 24 0 0 24 Sep 2024
Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint Dayeon Ki Cheonbok Park H. Kim FedML 23 0 0 24 Sep 2024
Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination Eva Sánchez Salido Roser Morante Julio Gonzalo Guillermo Marco Jorge Carrillo-de-Albornoz ... Enrique Amigó Andrés Fernández Alejandro Benito-Santos Adrián Ghajari Espinosa Victor Fresno ELM 39 0 0 19 Sep 2024
MEXMA: Token-level objectives improve sentence representations Joao Maria Janeiro Benjamin Piwowarski Patrick Gallinari Loïc Barrault 26 1 0 19 Sep 2024
AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs Basel Mousi Nadir Durrani Fatema Ahmad Md. Arid Hasan Maram Hasanain Tameem Kabbani Fahim Dalvi Shammur A. Chowdhury Firoj Alam 43 8 0 17 Sep 2024
Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement Vivi Nastase Chunyang Jiang Giuseppe Samo Paola Merlo 30 1 0 10 Sep 2024
CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset Mashael Al-Duwais H. Al-Khalifa Abdulmalik Al-Salman 37 0 0 22 Aug 2024
Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks Yiyi Chen Russa Biswas Heather Lent Johannes Bjerva AAML 38 4 0 21 Aug 2024