Recent foundational models for tabular data, such as TabPFN, have demonstrated remarkable effectiveness in adapting to new tasks through in-context learning. However, these models overlook a crucial equivariance property: the arbitrary ordering of target dimensions should not influence model predictions. In this study, we identify this oversight as a source of incompressible error, termed the equivariance gap, which introduces instability in predictions. To mitigate these issues, we propose a novel model designed to preserve equivariance across output dimensions. Our experimental results indicate that our proposed model not only addresses these pitfalls effectively but also achieves competitive benchmark performance.
View on arXiv@article{arbel2025_2502.06684,
title={ EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks },
author={ Michael Arbel and David Salinas and Frank Hutter },
journal={arXiv preprint arXiv:2502.06684},
year={ 2025 }
}