Neural Simpletrons - Minimalistic Directed Generative Networks for Learning with Few Labels

Deep learning is intensively studied using the perspectives of unsupervised and supervised learning. Comparisons of deep directed generative models and deep discriminative networks is difficult, however, because of: (A) the different semantics of their graphical descriptions; (B) different parameter optimization methods; (C) different benchmarking objectives, and (D) different scalability. Here, we investigate a deep directed model in a form and setting as similar to standard deep neural networks as possible. Based on normalized Poisson mixtures, we derive a minimalistic deep neural network with local activation and learning rules. The network can learn in a semi-supervised setting and can be scaled using standard deep learning tools. Benchmarks with partly labeled data provide the canonical domain for comparison with deep discriminative networks. Empirical evaluations show, that: (A) Performance of the network is competitive with recent deep networks (and other systems). (B) The network can be applied down to the limit of very few labeled data points. (C) The network is the best performing monolithic (i.e., non-hybrid) system for few labels. Our results provide a baseline for more expressive deep directed models, they highlight the performance vs. complexity tradeoff for deep learning, and show that already minimalistic deep directed models are competitive if they can be scaled.