Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark

This paper explores the feasibility of using text-to-image models in a zero-shot setup to generate images for taxonomy concepts. While text-based methods for taxonomy enrichment are well-established, the potential of the visual dimension remains unexplored. To address this, we propose a comprehensive benchmark for Taxonomy Image Generation that assesses models' abilities to understand taxonomy concepts and generate relevant, high-quality images. The benchmark includes common-sense and randomly sampled WordNet concepts, alongside the LLM generated predictions. The 12 models are evaluated using 9 novel taxonomy-related text-to-image metrics and human feedback. Moreover, we pioneer the use of pairwise evaluation with GPT-4 feedback for image generation. Experimental results show that the ranking of models differs significantly from standard T2I tasks. Playground-v2 and FLUX consistently outperform across metrics and subsets and the retrieval-based approach performs poorly. These findings highlight the potential for automating the curation of structured data resources.

@article{moskvoretskii2025_2503.10357,
  title={ Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark },
  author={ Viktor Moskvoretskii and Alina Lobanova and Ekaterina Neminova and Chris Biemann and Alexander Panchenko and Irina Nikishina },
  journal={arXiv preprint arXiv:2503.10357},
  year={ 2025 }
}