Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization

Dense-localization Audio-Visual Events (DAVE) aims to identify time boundaries and corresponding categories for events that can be heard and seen concurrently in an untrimmed video. Existing DAVE solutions extract audio and visual features through modality-specific encoders and fuse them via dense cross-attention. The independent processing of each modality neglects their complementarity, resulting in modality-specific noise, while dense attention fails to account for local temporal continuity of events, causing irrelevant signal distractions. In this paper, we present LoCo, a Locality-aware cross-modal Correspondence learning framework for DAVE. The core idea is to explore local temporal continuity nature of audio-visual events, which serves as informative yet free supervision signals to guide the filtering of irrelevant information and inspire the extraction of complementary multimodal information during both unimodal and cross-modal learning stages. i) Specifically, LoCo applies Locality-aware Correspondence Correction (LCC) to unimodal features via leveraging cross-modal local-correlated properties without any extra annotations. This enforces unimodal encoders to highlight similar semantics shared by audio and visual features. ii) To better aggregate such audio and visual features, we further customize Cross-modal Dynamic Perception layer (CDP) in cross-modal feature pyramid to understand local temporal patterns of audio-visual events by imposing local consistency within multimodal features in a data-driven manner. By incorporating LCC and CDP, LoCo provides solid performance gains and outperforms existing DAVE methods.