Learning Sentence Representation with Guidance of Human Attention

The most existing sentence representation models typically treat each word in sentences equally. However, extensive studies have proven that human read sentences by making a sequence of fixation and saccades (Rayner 1998), which is extremely efficient. In this paper, we propose two novel approaches, using significant predictors of human reading time, e.g., surprisal and word classes, implemented as attention models to improve representation capability of sentence embeddings. One approach utilizes surprisal directly as the attention weight over baseline models. The other one builds attention model with the help of POS tag and CCG supertag vectors which are trained together with word embeddings in the process of sentence representation learning. In experiments, we have evaluated our models on 24 textual semantic similarity datasets and the results demonstrate that the proposed models significantly outperform the state-of-the-art sentence representation models.