GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 1.9k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate the GDI-Bench on various open-source and closed-source models, conducting decoupled analyses in the visual and reasoning domains. For instance, the GPT-4o model excels in reasoning tasks but exhibits limitations in visual capabilities. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI Model that mitigates the issue of catastrophic forgetting during the supervised fine-tuning (SFT) process through a intelligence-preserving training strategy. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and model will be open source.

@article{li2025_2505.00063,
  title={ GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling },
  author={ Siqi Li and Yufan Shen and Xiangnan Chen and Jiayi Chen and Hengwei Ju and Haodong Duan and Song Mao and Hongbin Zhou and Bo Zhang and Pinlong Cai and Licheng Wen and Botian Shi and Yong Liu and Xinyu Cai and Yu Qiao },
  journal={arXiv preprint arXiv:2505.00063},
  year={ 2025 }
}