Broad Context Language Modeling as Reading Comprehension

Progress in text understanding has been driven by the availability of large datasets that test particular capabilities, like recent datasets for assessing reading comprehension. We focus here on the LAMBADA dataset, a word prediction task requiring broader context than the immediate sentence. We view the LAMBADA task as a reading comprehension problem and apply off-the-shelf comprehension models based on neural networks. Though these models are constrained to choose a word from the context, they improve the state of the art on LAMBADA from 7.3% to 45.4%. We analyze 100 instances, finding that neural network readers perform well in cases that involve selecting a name from the context based on dialogue or discourse cues but struggle when coreference resolution or external knowledge is needed.