Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual
Speech Recognition

v1v2 (latest)

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition

24 February 2022

Xinbing Wang

ArXiv (abs)PDF HTML

Papers citing "Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition"

12 / 12 papers shown

Title
CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization Detao Bai Zhiheng Ma Xihan Wei Liefeng Bo 449 0 0 06 May 2025
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim Sungwoo Cho Sangmin Bae Kangwook Jang Se-Young Yun SSL 163 1 0 23 Jan 2025
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs A. Haliassos Rodrigo Mira Honglie Chen Zoe Landgraf Stavros Petridis Maja Pantic SSL 81 7 0 04 Nov 2024
LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition Sreyan Ghosh Sonal Kumar Ashish Seth Purva Chiniya Utkarsh Tyagi R. Duraiswami Dinesh Manocha 94 0 0 06 Jun 2024
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition Yusheng Dai Hang Chen Jun Du Ruoyu Wang Shihao Chen Jie Ma Haotian Wang Chin-Hui Lee 96 4 0 07 Mar 2024
UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Models Sen Fang Bowen Gao Yangjian Wu T. Teoh DiffM 63 1 0 29 Jul 2023
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition Yuchen Hu Chen Chen Ruizhe Li Heqing Zou Chng Eng Siong GAN 113 9 0 18 Jun 2023
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Cheng Chen Chengwei Qin Qiu-shi Zhu Eng Siong Chng 112 5 0 18 Jun 2023
Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognition Wangyou Zhang Y. Qian 87 11 0 25 May 2023
Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Chen Chen Heqing Zou Qiu-shi Zhu Eng Siong Chng 91 8 0 16 May 2023
Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models Xiangming Gu Weizhen Zeng Jianan Zhang Longshen Ou Ye Wang 95 6 0 24 Apr 2023
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning Qiu-shi Zhu Long Zhou Zi-Hua Zhang Shujie Liu Binxing Jiao Jie Zhang Lirong Dai Daxin Jiang Jinyu Li Furu Wei 108 38 0 21 Nov 2022