From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion

We consider the problem of Multi-view 3D Face Reconstruction (MVR) with weakly supervised learning that leverages a limited number of 2D face images (e.g. 3) to generate a high-quality 3D face model with very light annotation. Despite their encouraging performance, present MVR methods simply concatenate multi-view image features and pay less attention to critical areas (e.g. eye, brow, nose, and mouth). To this end, we propose a novel model called Deep Fusion MVR (DF-MVR) and design a multi-view encoding to a single decoding framework with skip connections, able to extract, integrate, and compensate deep features with attention from multi-view images. In addition, we develop a multi-view face parse network to learn, identify, and emphasize the critical common face area. Finally, though our model is trained with a few 2D images, it can reconstruct an accurate 3D model even if one single 2D image is input. We conduct extensive experiments to evaluate various multi-view 3D face reconstruction methods. Experiments on Pixel-Face and Bosphorus datasets indicate the superiority of our model. Without 3D landmarks annotation, DF-MVR achieves 5.2% and 3.0% RMSE improvements over the existing best weakly supervised MVRs respectively on Pixel-Face and Bosphorus datasets; with 3D landmarks annotation, DF-MVR attains superior performance particularly on Pixel-Face dataset, leading to 13.4% RMSE improvement over the best weakly supervised MVR model.