Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection, and make fine-grained alignments between the extracted features and texts. We argue that object detection may not be necessary for vision language pre-training. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key of learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.