Learning 3D Articulation and Deformation using 2D Images

With the rise of Augmented Reality, Virtual Reality and 3D printing, methods for acquiring 3D models from the real world are more important then ever. One approach to generate 3D models is by modifying an existing template 3D mesh to fit the pose and shape of similar objects in images. To model the pose of an highly articulated and deformable object, it is essential to understand how an object class can articulate and deform. In this paper we propose to learn a class model of articulation and deformation from a set of annotated Internet images. To do so, we incorporate the idea of local stiffness, which specifies the amount of distortion allowed for a local region. Our system jointly learns the stiffness as it deforms a template 3D mesh to fit the pose of the objects in images. We show that this seemingly complex task can be solved with a sequence of convex optimization programs. We demonstrate our approach on two highly articulated and deformable animals, cats and horses. Our approach obtains significantly more realistic deformations compared to other related approaches.