Small-footprint Highway Deep Neural Networks for Speech Recognition

Morden state-of-the-art speech recognition systems usually employ neural networks for acoustic modeling. However, compared to the conventional Gaussian mixture models, deep neural network (DNN) based acoustic models usually have much larger number of model parameters, making it challenging for their applications in resource constrained platforms such as mobile devices. In this paper, we study the application of the recently proposed highway deep neural network (HDNN) for training small-footprint acoustic models. HDNN is a type of depth-gated feedforward neural network, which introduces two type of gate functions to facilitate the information flow through different layers. Our study demonstrates that HDNNs are more compact than plain DNNs for acoustic modeling, i.e., they can achieve comparable recognition accuracy with much less model parameters than plain DNN-based acoustic models. Furthermore, HDNNs are more controllable than plain DNNs. The gate functions of a HDNN largely control the behavior of the whole network with very small number of model parameters. And finally, HDNNs are more adaptable than plain DNNs. For example, simply updating the gate functions using the adaptation data can result in considerable gains. We demonstrate these aspect by experiments using the publicly available AMI meeting speech transcription corpus, which has around 80 hours of training data. Moreover, we also investigate the knowledge distillation technique to further improve the small-footprint HDNN acoustic models.