Robustness to Modification with Shared Words in Paraphrase Identification

Studying the robustness issues of natural language processing models and improving their robustness is important to their performance under difficult situations. In this paper, we study the robustness of paraphrase identification models from a new perspective -- via modification with shared words. For an example consisting of a pair of sentences, we either replace some words shared by both sentences or introduce new shared words. We aim to construct a new example such that the model makes a wrong prediction. We try to find a solution with beam search constrained by heuristic rules and a language model. Experiments show that the performance of the models has a dramatic drop on our modified examples, revealing a robustness issue. We also mitigate the issue with adversarial training.