Multi-Shot Mining Semantic Part Concepts in CNNs

This paper proposes a new learning strategy that incrementally embeds new object-part concepts into a pre-trained convolutional neural network (CNN), in order to 1) explore explicit semantics for the CNN units and 2) gradually transfer the pre-trained CNN into a "white-box" model for hierarchical object understanding. Given part annotations on a very small number (e.g. 3--12) of objects, our method mines certain units from the pre-trained CNN and associate them with different part concepts. We use a four-layer And-Or graph to organize the CNN units, which clarifies their internal semantic hierarchy. Our method is guided by a small number of part annotations, and it achieves superior part-localization performance (about 28%--107% improvement in part center prediction).