Deep Manifold Traversal: Changing Labels with Convolutional Features

Machine learning is increasingly used in high impact applications such as prediction of hospital re-admission, cancer screening or bio-medical research applications. As predictions become increasingly accurate, practitioners may be interested in identifying actionable changes to inputs in order to alter their class membership. For example, a doctor might want to know what changes to a patient's status would predict him/her to not be re-admitted to the hospital soon. Szegedy et al. (2013b) demonstrated that identifying such changes can be very hard in image classification tasks. In fact, tiny, imperceptible changes can result in completely different predictions without any change to the true class label of the input. In this paper we ask the question if we can make small but meaningful changes in order to truly alter the class membership of images from a source class to a target class. To this end we propose deep manifold traversal, a method that learns the manifold of natural images and provides an effective mechanism to move images from one area (dominated by the source class) to another (dominated by the target class).The resulting algorithm is surprisingly effective and versatile. It allows unrestricted movements along the image manifold and only requires few images from source and target to identify meaningful changes. We demonstrate that the exact same procedure can be used to change an individual's appearance of age, facial expressions or even recolor black and white images.