AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), often produce out-of-distribution or noisy inputs, leading to misalignment between the modalities. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where scanned document images must be accurately mapped to their textual content. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods. We provide further analysis demonstrating improved vision-text feature alignment and robustness to noise.
View on arXiv@article{masry2025_2502.01341, title={ AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding }, author={ Ahmed Masry and Juan A. Rodriguez and Tianyu Zhang and Suyuchen Wang and Chao Wang and Aarash Feizi and Akshay Kalkunte Suresh and Abhay Puri and Xiangru Jian and Pierre-André Noël and Sathwik Tejaswi Madhusudhan and Marco Pedersoli and Bang Liu and Nicolas Chapados and Yoshua Bengio and Enamul Hoque and Christopher Pal and Issam H. Laradji and David Vazquez and Perouz Taslakian and Spandana Gella and Sai Rajeswar }, journal={arXiv preprint arXiv:2502.01341}, year={ 2025 } }