On the Limitations of Vision-Language Models in Understanding Image Transforms

Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations.

@article{anis2025_2503.09837,
  title={ On the Limitations of Vision-Language Models in Understanding Image Transforms },
  author={ Ahmad Mustafa Anis and Hasnain Ali and Saquib Sarfraz },
  journal={arXiv preprint arXiv:2503.09837},
  year={ 2025 }
}