We learn to compute optical flow by combining a classical coarse-to-fine flow approach with deep learning. Specifically we adopt a spatial-pyramid formulation to deal with large motions. According to standard practice, we warp one image of a pair at each pyramid level by the current flow estimate and compute an update to the flow field. Instead of the standard minimization of an objective function at each pyramid level, we train a deep neural network at each level to compute the flow update. Unlike the recent FlowNet approach, the networks do not need to deal with large motions; these are dealt with by the pyramid. This has several advantages. First the network is much simpler and much smaller; our Spatial Pyramid Network (SPyNet) is 96% smaller than FlowNet in terms of model parameters. Because it is so small, the method could be made to run on a cell phone or other embedded system. Second, since the flow is assumed to be small at each level ($< 1$ pixel), a convolutional approach applied to pairs of warped images is appropriate. Third, unlike FlowNet, the filters that we learn appear similar to classical spatio-temporal filters, possibly giving insight into how to improve the method further. Our results are more accurate than FlowNet in most cases and suggest a new direction of combining classical flow methods with deep learning.