Efficient Large-Scale Visual Representation Learning And Evaluation

In this article, we present our approach to single-modality visual representation learning. Understanding visual representations of items is vital for fashion recommendations in e-commerce. We detail and contrast techniques used to finetune large-scale visual representation learning models in an efficient manner under low-resource settings, including several pretrained backbone architectures, both in the convolutional neural network as well as the vision transformer family. We describe the challenges for e-commerce applications at-scale and highlight the efforts to more efficiently train, evaluate, and serve visual representations. We present ablation studies evaluating the representation offline performance for several downstream tasks, including visually similar ad recommendations on mobile devices. To this end, we present a novel multilingual text-to-image generative offline evaluation method for visually similar recommendation systems. Finally, we include online results from deployed machine learning systems in production at Etsy.