Families In Wild Multimedia (FIW-MM): A Multi-Modal Database for Recognizing Kinship

Kinship is a soft biometric detectable in media with an abundance of practical applications. Despite the difficulty of detecting kinship, annual data challenges using still images have consistently resulted in improved performances and attracted new researchers. Now, systems are reaching performance levels unforeseeable a decade ago, closing in on performances acceptable for real-world use. Similar to other biometric tasks, we expect that systems can benefit from additional modalities. We hypothesize that adding modalities to FIW, which contains only still images, will improve performance. Thus, to narrow the gap between research-and-reality and enhance the power of kinship recognition systems, we extend FIW with multimedia (MM) data (i.e., video, audio, and text captions). Specifically, we introduce the first publicly available multi-task MM kinship dataset. To build FIW-MM, machinery was developed to collect, annotate, and prepare the data automatically, requiring minimal human input and no financial cost. The proposed MM corpus allows us to formulate problems following more realistic template-based protocols. We show significant improvements in all benchmarks with the added modalities. The results are analyzed by highlighting edge cases to inspire future research with different areas of improvement. FIW-MM provides the data required to increase the potential of systems built to automatically detect kinship in MM. It also allows experts from diverse fields to collaborate in new ways.