ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2007.03814
69
33
v1v2v3v4 (latest)

Variational Representations and Neural Network Estimation of Rényi Divergences

7 July 2020
Jeremiah Birrell
P. Dupuis
Markos A. Katsoulakis
Luc Rey-Bellet
Jie Wang
ArXiv (abs)PDFHTML
Abstract

We derive a new variational formula for the R\ényi family of divergences, Rα(Q∥P)R_\alpha(Q\|P)Rα​(Q∥P), between probability measures QQQ and PPP. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this R\ényi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for R\ényi divergence estimators. By applying this theory to neural-network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding R\ényi divergence estimator is consistent. In contrast to density-estimator based methods, our estimators involve only expectations under QQQ and PPP and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.

View on arXiv
Comments on this paper