Mixture Decompositions using a Decomposition of the Sample Space

We present a scheme for decomposing joint probability distributions of $N$ binary random variables as mixtures of other distributions: $p(x_1,...,x_N)=\sum_{i=1}^m \alpha_i f_i(x_1,...,x_N)$, where $\alpha_i \geq 0$, $\sum_{i=1}^m \alpha_i =1$, and the $f_i$ belong to some exponential family. We characterize subsets of the sample space for which any distribution with support therein can be used as mixture component $f_i$ from an exponential family. This allows us to derive bounds for the minimal number $m$ of mixture components from a hierarchy of exponential families which is sufficient to represent any distribution, and bounds for the number of mixture components necessary to represent distributions with arbitrary correlations up to a given order. We show in particular that every distribution $p$ on $\{0,1\}^N$ can be written as a mixture of $m$ independent distributions whenever $m\geq 2^{N-1}$, and furthermore, that there are distributions which cannot be written as a mixture of less than $2^{N-1}$ independent distributions. We find also that a number $\sim 2^{N-(k+1)}$ of mixture components from the exponential family with interaction order $k$ is sufficient to represent any distribution.