ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2210.00189
13
4

Pitfalls of Gaussians as a noise distribution in NCE

1 October 2022
Holden Lee
Chirag Pabbaraju
A. Sevekari
Andrej Risteski
ArXivPDFHTML
Abstract

Noise Contrastive Estimation (NCE) is a popular approach for learning probability density functions parameterized up to a constant of proportionality. The main idea is to design a classification problem for distinguishing training data from samples from an easy-to-sample noise distribution qqq, in a manner that avoids having to calculate a partition function. It is well-known that the choice of qqq can severely impact the computational and statistical efficiency of NCE. In practice, a common choice for qqq is a Gaussian which matches the mean and covariance of the data. In this paper, we show that such a choice can result in an exponentially bad (in the ambient dimension) conditioning of the Hessian of the loss, even for very simple data distributions. As a consequence, both the statistical and algorithmic complexity for such a choice of qqq will be problematic in practice, suggesting that more complex noise distributions are essential to the success of NCE.

View on arXiv
Comments on this paper