Lifelong Compositional Feature Replays Beat Image Replays in Stream Learning

Our brains extract durable, generalizable knowledge from transient experiences of the world. Artificial neural networks come nowhere close: when tasked with learning to classify objects by training on non-repeating video frames in temporal order (online stream learning), models that learn well from shuffled datasets catastrophically forget old knowledge upon learning new stimuli. We propose a new continual learning algorithm, Compositional Replay Using Memory Blocks (CRUMB), which mitigates forgetting by replaying feature maps reconstructed by recombining generic parts. Just as crumbs together form a loaf of bread, we concatenate trainable and re-usable "memory block" vectors to compositionally reconstruct feature map tensors in convolutional neural networks. CRUMB stores the indices of memory blocks used to reconstruct new stimuli, enabling replay of specific memories during later tasks. CRUMB's memory blocks are tuned to enhance replay: a single feature map stored, reconstructed, and replayed by CRUMB mitigates forgetting during video stream learning more effectively than an entire image while occupying only 3.6% of the memory. We stress-tested CRUMB alongside 13 competing methods on 5 challenging datasets, including 3 video stream datasets containing 10, 12, and 14 classes, and 2 image datasets containing 100 and 1000 classes. To address the limited number of existing online stream learning datasets, we introduce 2 new benchmarks by adapting existing datasets for stream learning. With about 4% of the memory and 20% of the runtime, CRUMB mitigates catastrophic forgetting more effectively than the prior state of the art with an average top-1 accuracy margin of 4.4%, achieving the highest accuracy on 6 out of 8 benchmarks. Our code is available at https://tinyurl.com/4zed3ske.