A Survey of Knowledge Enhanced Pre-trained Models

Pre-trained models learn informative representations on large-scale training data through a self-supervised or supervised learning method, which has achieved promising performance in natural language processing (NLP), computer vision (CV), and cross-modal fields after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. Pre-trained models with knowledge injection, which we call knowledge enhanced pre-trained models (KEPTMs), possess deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPTMs in NLP and CV. We first introduce the progress of pre-trained models and knowledge representation learning. Then we systematically categorize existing KEPTMs from three different perspectives. Finally, we outline some potential directions of KEPTMs for future research.