We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. By comparing base, instruction-tuned, and knowledge-distilled models, we find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) Instruction tuning improves performance in concept understanding but does not enhance internal competence; knowledge distillation can enhance internal competence in conceptual understanding for low-resource languages with limited gains in explicit task performance. 4) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.
View on arXiv@article{he2025_2502.19737,
title={ XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs },
author={ Linyang He and Ercong Nie and Sukru Samet Dindar and Arsalan Firoozi and Adrian Florea and Van Nguyen and Corentin Puffay and Riki Shimizu and Haotian Ye and Jonathan Brennan and Helmut Schmid and Hinrich Schütze and Nima Mesgarani },
journal={arXiv preprint arXiv:2502.19737},
year={ 2025 }
}