Steerable Transformers

In this work we introduce Steerable Transformers, an extension of the Vision Transformer mechanism that maintains equivariance to the special Euclidean group $\mathrm{SE}(d)$. We propose an equivariant attention mechanism that operates on features extracted by steerable convolutions. Operating in Fourier space, our network utilizes Fourier space non-linearities. Our experiments in both two and three dimensions show that adding steerable transformer layers to steerable convolutional networks enhances performance.

@article{kundu2025_2405.15932,
  title={ Steerable Transformers },
  author={ Soumyabrata Kundu and Risi Kondor },
  journal={arXiv preprint arXiv:2405.15932},
  year={ 2025 }
}