Tabular data contains rich structural semantics and plays a crucial role in organizing and manipulating information. To better capture these structural semantics, this paper introduces the HybrId-modal Preference oPtimizatiOn (HIPPO) model, which represents tables using both text and image, and optimizes MLLMs to effectively learn more comprehensive table information from these multiple modalities. Specifically, HIPPO samples model responses from hybrid-modal table representations and designs a modality-consistent sampling strategy to enhance response diversity and mitigate modality bias during DPO training. Experimental results on table question answering and table fact verification tasks demonstrate the effectiveness of HIPPO, achieving a 4% improvement over various table reasoning models. Further analysis reveals that HIPPO not only enhances reasoning abilities based on unimodal table representations but also facilitates the extraction of crucial and distinct semantics from different modal representations. All data and codes are available atthis https URL.
View on arXiv@article{liu2025_2502.17315,
title={ HIPPO: Enhancing the Table Understanding Capability of Large Language Models through Hybrid-Modal Preference Optimization },
author={ Zhenghao Liu and Haolan Wang and Xinze Li and Qiushi Xiong and Xiaocui Yang and Yu Gu and Yukun Yan and Qi Shi and Fangfang Li and Ge Yu and Maosong Sun },
journal={arXiv preprint arXiv:2502.17315},
year={ 2025 }
}