SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes

We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs). Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The large number of submissions underscores the interest of the community in hallucination detection. We present the results of the participating systems and conduct an empirical analysis to identify key factors contributing to strong performance in this task. We also emphasize relevant current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.

@article{vázquez2025_2504.11975,
  title={ SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes },
  author={ Raúl Vázquez and Timothee Mickus and Elaine Zosa and Teemu Vahtola and Jörg Tiedemann and Aman Sinha and Vincent Segonne and Fernando Sánchez-Vega and Alessandro Raganato and Jindřich Libovický and Jussi Karlgren and Shaoxiong Ji and Jindřich Helcl and Liane Guillou and Ona de Gibert and Jaione Bengoetxea and Joseph Attieh and Marianna Apidianaki },
  journal={arXiv preprint arXiv:2504.11975},
  year={ 2025 }
}