Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training

Transmission latency significantly affects users' quality of experience in real-time interaction and actuation. As latency is principally inevitable, video prediction can be utilized to mitigate the latency and ultimately enable zero-latency transmission. However, most of the existing video prediction methods are computationally expensive and impractical for real-time applications. In this work, we therefore propose real-time video prediction towards the zero-latency interaction over networks, called IFRVP (Intermediate Feature Refinement Video Prediction). Firstly, we propose three training methods for video prediction that extend frame interpolation models, where we utilize a simple convolution-only frame interpolation network based on IFRNet. Secondly, we introduce ELAN-based residual blocks into the prediction models to improve both inference speed and accuracy. Our evaluations show that our proposed models perform efficiently and achieve the best trade-off between prediction accuracy and computational speed among the existing video prediction methods. A demonstration movie is also provided atthis http URL. The code will be released atthis https URL.

@article{hirose2025_2503.23185,
  title={ Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training },
  author={ Shota Hirose and Kazuki Kotoyori and Kasidis Arunruangsirilert and Fangzheng Lin and Heming Sun and Jiro Katto },
  journal={arXiv preprint arXiv:2503.23185},
  year={ 2025 }
}