ORCA: A Challenging Benchmark for Arabic Language Understanding

21 December 2022

AbdelRahim Elmadany

El Moatez Billah Nagoudi

Muhammad Abdul-Mageed

ELM

ArXiv PDF HTML

Papers citing "ORCA: A Challenging Benchmark for Arabic Language Understanding"

36 / 36 papers shown

Title
Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs Fakhraddin Alwajih Abdellah El Mekki Samar Magdy AbdelRahim Elmadany Omer Nacar ... Anis Koubaa Ismail Berrada Mustafa Jarrar Shady Shehata Muhammad Abdul-Mageed 31 1 0 28 Feb 2025
Where Are We? Evaluating LLM Performance on African Languages Ife Adebara Hawau Olamide Toyin Nahom Tesfu Ghebremichael AbdelRahim Elmadany Muhammad Abdul-Mageed 52 0 0 26 Feb 2025
Arabic Dataset for LLM Safeguard Evaluation Yasser Ashraf Yuxia Wang Bin Gu Preslav Nakov Timothy Baldwin 21 0 0 22 Oct 2024
RoQLlama: A Lightweight Romanian Adapted Language Model George-Andrei Dima Andrei-Marius Avram Cristian-George Crăciun Dumitru-Clementin Cercel 26 2 0 05 Oct 2024
CoDi: Conversational Distillation for Grounded Question Answering Patrick Huber Arash Einolghozati Rylan Conway Kanika Narang Matt Smith Waqar Nayyar Adithya Sagar Ahmed Aly Akshat Shrivastava 18 0 0 20 Aug 2024
EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora Faisal Qarah 32 3 0 07 Aug 2024
NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task Muhammad Abdul-Mageed Amr Keleg AbdelRahim Elmadany Chiyu Zhang Injy Hamed Walid Magdy Houda Bouamor Nizar Habash 30 17 0 06 Jul 2024
Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art Chen Cecilia Liu Iryna Gurevych Anna Korhonen 33 5 0 06 Jun 2024
SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora Faisal Qarah 36 4 0 10 May 2024
DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages Fahim Faisal Orevaoghene Ahia Aarohi Srivastava Kabir Ahuja David Chiang Yulia Tsvetkov Antonios Anastasopoulos 56 27 0 16 Mar 2024
AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic Emad A. Alghamdi Reem I. Masoud Deema Alnuhait Afnan Y. Alomairi Ahmed Ashraf Mohamed Zaytoon 40 4 0 14 Mar 2024
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic Fajri Koto Haonan Li Sara Shatnawi Jad Doughman Abdelrahman Boda Sadallah ... Neha Sengupta Shady Shehata Nizar Habash Preslav Nakov Timothy Baldwin ELM LRM 72 30 0 20 Feb 2024
On the importance of Data Scale in Pretraining Arabic Language Models Abbas Ghaddar Philippe Langlais Mehdi Rezagholizadeh Boxing Chen 16 0 0 15 Jan 2024
Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction S. Kwon Gagan Bhatia El Moatez Billah Nagoudi Muhammad Abdul-Mageed 47 17 0 13 Dec 2023
Toxic language detection: a systematic review of Arabic datasets Imene Bensalem Paolo Rosso Hanane Zitouni 25 4 0 12 Dec 2023
Arabic Fine-Grained Entity Recognition Haneen Liqreina Mustafa Jarrar Mohammed Khalilia Ahmed Oumar El-Shangiti Muhammad AbdulMageed 20 13 0 26 Oct 2023
NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task Muhammad Abdul-Mageed AbdelRahim Elmadany Chiyu Zhang El Moatez Billah Nagoudi Houda Bouamor Nizar Habash 25 35 0 24 Oct 2023
Prediction of Arabic Legal Rulings using Large Language Models Adel Ammar Anis Koubaa Bilel Benjdira Omar Najar Serry Sibaee AILaw ELM 17 7 0 16 Oct 2023
ChatGPT for Arabic Grammatical Error Correction S. Kwon Gagan Bhatia El Moatez Billah Nagoudi Muhammad Abdul-Mageed 26 7 0 08 Aug 2023
TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties Karima Kadaoui Samar Magdy Abdul Waheed Md. Tawkat Islam Khondaker Ahmed Oumar El-Shangiti El Moatez Billah Nagoudi Muhammad Abdul-Mageed 27 20 0 06 Aug 2023
Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models Zaid Alyafeai Maged S. Alshaibani Badr AlKhamissi H. Luqman Ebrahim Alareqi A. Fadel ELM LM&MA AI4MH 9 17 0 28 Jun 2023
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing Asaad Alghamdi Xinyu Duan Wei Jiang Zhenhai Wang Yimeng Wu ... Yifei Zheng Mehdi Rezagholizadeh Baoxing Huai Peilun Cheng Abbas Ghaddar VLM 14 8 0 11 Jun 2023
Dolphin: A Challenging and Diverse Benchmark for Arabic NLG El Moatez Billah Nagoudi AbdelRahim Elmadany Ahmed Oumar El-Shangiti Muhammad Abdul-Mageed LM&MA 30 17 0 24 May 2023
LAraBench: Benchmarking Arabic AI with Large Language Models Ahmed Abdelali Hamdy Mubarak Shammur A. Chowdhury Maram Hasanain Basel Mousi ... Yousseif Elshahawy Ahmed M. Ali Nadir Durrani Natasa Milic-Frayling Firoj Alam ELM LM&MA 16 19 0 24 May 2023
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP Md. Tawkat Islam Khondaker Abdul Waheed El Moatez Billah Nagoudi Muhammad Abdul-Mageed ELM LM&MA 19 62 0 24 May 2023
Having Beer after Prayer? Measuring Cultural Bias in Large Language Models Tarek Naous Michael Joseph Ryan Alan Ritter Wei-ping Xu 24 85 0 23 May 2023
DUMB: A Benchmark for Smart Evaluation of Dutch Models Wietse de Vries Martijn B. Wieling Malvina Nissim ELM ALM MoE 26 6 0 22 May 2023
JASMINE: Arabic GPT Models for Few-Shot Learning El Moatez Billah Nagoudi Muhammad Abdul-Mageed AbdelRahim Elmadany Alcides Alcoba Inciarte Md. Tawkat Islam Khondaker 25 7 0 21 Dec 2022
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation Kaustubh D. Dhole Varun Gangal Sebastian Gehrmann Aadesh Gupta Zhenhao Li ... Tianbao Xie Usama Yaseen Michael A. Yee Jing Zhang Yue Zhang 169 86 0 06 Dec 2021
AraT5: Text-to-Text Transformers for Arabic Language Generation El Moatez Billah Nagoudi AbdelRahim Elmadany Muhammad Abdul-Mageed 84 118 0 31 Aug 2021
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy ... Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu Shujie Liu ELM 196 853 0 09 Feb 2021
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann Tosin P. Adewumi Karmanya Aggarwal Pawan Sasanka Ammanamanchi Aremu Anuoluwapo ... Nishant Subramani Wei-ping Xu Diyi Yang Akhila Yerukola Jiawei Zhou VLM 246 283 0 02 Feb 2021
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic Muhammad Abdul-Mageed AbdelRahim Elmadany El Moatez Billah Nagoudi VLM 60 447 0 27 Dec 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 251 2,012 0 28 Jul 2020
MLQA: Evaluating Cross-lingual Extractive Question Answering Patrick Lewis Barlas Oğuz Ruty Rinott Sebastian Riedel Holger Schwenk ELM 244 491 0 16 Oct 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,950 0 20 Apr 2018