This paper presents a method to estimate optical flow under rainy scenes. Optical flow estimation in the rainy scenes is considered challenging due to background degradation introduced by rain streaks and rain accumulation effects in the scene. Rain accumulation effect refers to poor visibility of remote object due to the intense rainfall. Most existing optical flow methods are erroneous when applied to rain sequences because the conventional brightness constancy constraint (BCC) and gradient constancy constraint (GCC) generally break down in this situation. In this paper, our method considers the rain streaks and rain accumulation separately. Based on the fact that the RGB color channels receive raindrop radiance equally, we introduce a residue channel as a new data constraint to significantly reduce rain streaks. In the case of rain accumulation, our method proposes to separate the image into a piecewise-smooth background layer and a high-frequency detail layer and enforce BCC on the background layer only. Results on both synthetic dataset and real images show that our algorithm outperforms existing methods on different types of rain sequences. To our knowledge, this is the first optical flow method dealing with rain.