Coupled Depth Learning

In this paper we propose a method for estimating depth from a single image using a coarse to fine approach. We argue that modeling the fine depth details is easier after a coarse depth map is computed. We express the global depth map of an image as a linear combination of a depth basis learned from examples. The depth basis captures spatial and statistical regularities and reduces the problem of coarse depth estimation to the task of predicting the input-specific coefficients in the linear combination, which are much fewer than the number of pixels. This is formulated as a regression problem from a holistic representation of the image. Crucially, the depth basis and the regression function are {\bf coupled} and jointly optimized by our learning scheme. We demonstrate that this results in a significant improvement in accuracy compared to direct regression of depth pixel values or approaches learning the depth basis disjointly from the regression function. This global estimation is then used as a guidance by a local refinement method that introduces depth details that could not be captured at the coarse level. Experiments on the NYUv2 and KITTI datasets show that our method outperforms the existing state-of-the-art at a considerably lower computational cost for both training and testing.