Fast Single-Class Classification and the Principle of Logit Separation

We consider neural network training, in applications in which there are many possible classes, but at test time the task is to identify whether the example belongs to one specific class, e.g., when searching for photos of a specific person. We focus on reducing the computational burden at test-time in such applications. We define the Single Logit Classification (SLC) task: training the network so that at test time, it would be possible to accurately identify if the example belongs to a given class, based only on the output logit of the trained model for this class. We propose a natural principle, the Principle of Logit Separation (PoLS), as a guideline for choosing and designing losses suitable for the SLC. We study previously suggested losses and their alignment with the PoLS. We further derive new batch versions of known losses, and show that unlike the standard versions, these new versions satisfy the PoLS. Our experiments show that the losses aligned with the PoLS overwhelmingly outperform the other losses on the SLC task. Tensorflow code for optimizing the new batch losses is publicly available in https://github.com/cruvadom/Logit_Separation.