Stochastic Neural Networks with Monotonic Activation Functions

We propose a Laplace approximation that closely links any smooth monotonic activation function to its stochastic counterpart using Gaussian noise. We investigate the application of this approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family -- exponential family RBM (Exp-RBM) -- is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearities for neurons. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBMs can learn useful representations using novel stochastic units.