ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz). These games are designed to evaluate a range of capabilities such as strategic reasoning, planning, knowledge application, safety, and adaptability. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework for easily implementing games and leverages DSPy to provide a better abstraction for LLM player strategies.

@article{alyahya2025_2503.10673,
  title={ ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition },
  author={ Hisham A. Alyahya and Haidar Khan and Yazeed Alnumay and M Saiful Bari and Bülent Yener },
  journal={arXiv preprint arXiv:2503.10673},
  year={ 2025 }
}