PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data

We propose PyViT-FUSE, a foundation model for earth observation data explicitly designed to handle multi-modal imagery by learning to fuse an arbitrary number of mixed-resolution input bands into a single representation through an attention mechanism. The learned patch tokens are further processed by a stack of vision transformers with a novel pyramidal structure. We train the model on a globally sampled dataset in a self-supervised manner, leveraging core concepts of the SwAV algorithm. We show the interpretability of the fusion mechanism by visualization of the attention scores and the models applicability to downstream tasks.

@article{weber2025_2504.18770,
  title={ PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data },
  author={ Manuel Weber and Carly Beneke },
  journal={arXiv preprint arXiv:2504.18770},
  year={ 2025 }
}