Efficient Motion-Aware Video MLLM

Most current video MLLMs rely on uniform frame sampling and image-level encoders, resulting in inefficient data processing and limited motion awareness. To address these challenges, we introduce EMA, an Efficient Motion-Aware video MLLM that utilizes compressed video structures as inputs. We propose a motion-aware GOP (Group of Pictures) encoder that fuses spatial and motion information within a GOP unit in the compressed video stream, generating compact, informative visual tokens. By integrating fewer but denser RGB frames with more but sparser motion vectors in this native slow-fast input architecture, our approach reduces redundancy and enhances motion representation. Additionally, we introduce MotionBench, a benchmark for evaluating motion understanding across four motion types: linear, curved, rotational, and contact-based. Experimental results show that EMA achieves state-of-the-art performance on both MotionBench and popular video question answering benchmarks, while reducing inference costs. Moreover, EMA demonstrates strong scalability, as evidenced by its competitive performance on long video understanding benchmarks.

@article{zhao2025_2503.13016,
  title={ Efficient Motion-Aware Video MLLM },
  author={ Zijia Zhao and Yuqi Huo and Tongtian Yue and Longteng Guo and Haoyu Lu and Bingning Wang and Weipeng Chen and Jing Liu },
  journal={arXiv preprint arXiv:2503.13016},
  year={ 2025 }
}