Discrete Restricted Boltzmann Machines

We describe discrete restricted Boltzmann machines: probabilistic graphical models with bipartite interactions between discrete visible and hidden variables. These models generalize standard binary restricted Boltzmann machines and discrete na\"ive Bayes models. For a given number of visible variables and cardinalities of their state spaces, we bound the number of hidden variables, depending on the cardinalities of their state spaces, for which the model is a universal approximator of probability distributions. More generally, we describe exponential subfamilies and use them to bound the Kullback-Leibler approximation errors of these models from above. We use coding theory and algebraic methods to study the geometry of these models, and show that in many cases they have the dimension expected from counting parameters, but in some cases they do not. We discuss inference functions, mixtures of product distributions with shared parameters, and patterns of strong modes of probability distributions represented by discrete restricted Boltzmann machines in terms of configurations of projected products of simplices in normal fans of products of simplices.