Quantifying Exposure Bias for Neural Language Generation

The exposure bias problem refers to the training-testing discrepancy caused by teacher forcing in maximum likelihood estimation (MLE) training, for auto-regressive neural network language models (LM). It has been regarded as a central problem for natural language generation (NLG) model training. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore to alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we first identify and analyze the self-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias. We then develop a precise, quantifiable definition for exposure bias. Based on that definition, we design experiments to measure the seriousness of exposure bias. Surprisingly, we find that removing the training-testing discrepancy only brings very little performance gain, in both real and synthetic settings. With these results, we conclude that on the contrary to popular belief, the exposure bias problem is only a minor problem for MLE-based LM training.