Quantifying Exposure Bias for Neural Language Generation

The exposure bias problem refers to the training-generation discrepancy, caused by teacher forcing, in maximum likelihood estimation (MLE) training for auto-regressive neural network language models (LM). It has been regarded as a central problem for neural language generation (NLG) model training. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore` to alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we first identify the self-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias. We then propose sequence-level (EB-bleu) and word-level (EB-C) metrics to quantify the impact of exposure bias. We conduct experiments for the LSTM/transformer model, in both real and synthetic settings. In addition to the unconditional NLG task, we also include results for a seq2seq machine translation task. Surprisingly, all our measurements indicate that removing the training-generation discrepancy only brings very little performance gain. In our analysis, we hypothesise that although there exist a mismatch between the model distribution and the data distribution, the mismatch is still in the model's "comfortable zone", and is not big enough to induce significant performance loss.