Beyond standard benchmarks: Parameterizing performance evaluation in visual object tracking

Benchmarks have played an important role in advancing the field of visual object tracking. Due to weakly defined and sometimes biased attribute specification, existing benchmarks do not allow fine-grained tracker analysis with respect to specific attributes. Apart from illumination changes and occlusions, the tracking performance is most strongly affected by the object motion. In this paper, we propose a novel approach for tracker evaluation with respect to the motion-related attributes. Our approach utilizes 360 degree videos to generate realistic annotated short-term tracking scenarios with exact specification of the object motion type and extent. A fully annotated dataset of 360 degree videos was constructed and fine-grained performance of 17 state-of-the-art trackers is reported. The proposed approach offers unique tracking insights, is complementary to existing benchmarks, and will be made publicly available. The evaluation system was implemented within a state-of-the-art performance evaluation toolkit and supports straight-forward extension with third-party trackers.