Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters

In this paper, we newly introduce the concept of temporal attention filters, and describe how they can be used for human activity recognition from videos. Many high-level activities are often composed of multiple temporal parts (i.e., sub-events) with different duration/speed, and our objective is to make the model explicitly learn such latent sub-events and their structure using multiple temporal filters. Our attention filters are designed to be fully differentiable, allowing end-of-end training of the temporal filters together with the underlying frame-based or segment-based convolutional neural network architectures. This paper presents an approach of learning optimal static temporal attention filters to be shared across different videos, and extends this approach to dynamically adjust attention filters per testing video using recurrent long short-term memory networks (LSTMs). This allows our model to learn latent sub-events specific to each activity. We experimentally confirm that the proposed concept of temporal attention filter benefits the activity recognition, and we visualize the learned latent sub-events.