Quantifying and Exploring Heterogeneity in Domain Generalization through Contrastive Analysis

Domain generalization (DG) is a commonly encountered issue in real-world applications. Its objective is to train models that can generalize well to unseen target domains by utilizing multiple source domains. In most DG algorithms, domain labels, which indicate the domain from which each data point is sampled, are treated as a form of supervision to enhance generalization performance. However, using the original domain labels as the supervision signal may not be optimal due to a lack of diversity among domains, known as heterogeneity. This lack of heterogeneity can lead to the original labels being noisy and disrupting the generalization learning process. Some methods attempt to address this by re-dividing the domains and applying a new dividing pattern. However, the chosen pattern may not capture the maximum heterogeneity since there is no metric available to quantify it accurately. In this paper, we propose that domain heterogeneity primarily lies in variant features within the invariant learning framework. We introduce a novel approach which utilizes contrastive learning to guide the metric for domain heterogeneity. By promoting the learning of variant features, we develop a metric that captures models' learning potential for data heterogeneity. We also emphasize the distinction between seeking variance-based heterogeneity and training an invariance-based generalizable model. In the first stage, we generate the most heterogeneous dividing pattern using our contrastive metric. In the second stage, we employ contrastive learning focused on invariance by constructing pairs based on the stable relationships indicated by domains and classes. This approach effectively utilizes the generated domain labels for generalization. Extensive experiments demonstrate that our method successfully uncovers heterogeneity and achieves remarkable generalization performance.