Emergence of Latent Binary Encoding in Deep Neural Network Classifiers

We investigate the emergence of binary encoding within the latent space of deep-neural-network classifiers. Such binary encoding is induced by the integration of a linear penultimate layer, which employs during training a loss function specifically designed to compress the latent representations. As a result of a trade-off between compression and information retention, the network learns to assume only one of two possible values for each dimension in the latent space. The binary encoding is provoked by the collapse of all representations of the same class to the same point, which corresponds to the vertex of a hypercube, thereby creating the encoding. We demonstrate that the emergence of binary encoding significantly enhances robustness, reliability and accuracy of the network.