98
5

Deformed Statistics Formulation of the Information Bottleneck Method

Abstract

A candidate variational principle for the information bottleneck (IB) method is formulated within the ambit of the generalized nonadditive statistics of Tsallis. Given a nonadditivity parameter q q , the role of the \textit{additive duality} of nonadditive statistics (q=2q q^*=2-q ) in relating Tsallis entropies for ranges of the nonadditivity parameter q<1 q < 1 and q>1 q > 1 is described. Defining X X , X~ \tilde X , and Y Y to be the source alphabet, the compressed reproduction alphabet, and, the \textit{relevance variable} respectively, it is demonstrated that minimization of a generalized IB Lagrangian defined in terms of the nonadditivity parameter q q^* self-consistently yields the \textit{nonadditive effective distortion measure} to be the \textit{q q -deformed} generalized Kullback-Leibler divergence: DKLq[p(YX)p(YX~)] D_{K-L}^{q}[p(Y|X)||p(Y|\tilde X)] . This result is achieved without enforcing any \textit{a-priori} assumptions. Finally, it is proven that the nonadditive free energy of the system in q q^* space is non-negative and convex. These results generalize critical features of the IB method to the case of Tsallis statistics.

View on arXiv
Comments on this paper