Robust Self-Supervised Audio-Visual Speech Recognition

5 January 2022

Papers citing "Robust Self-Supervised Audio-Visual Speech Recognition"

50 / 63 papers shown

Title
SyncDiff: Diffusion-based Talking Head Synthesis with Bottlenecked Temporal Visual Prior for Improved Synchronization Xulin Fan Heting Gao Ziyi Chen Peng Chang Mei Han Mark Hasegawa-Johnson DiffM 45 0 0 17 Mar 2025
Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models Jing-Xuan Zhang Genshun Wan Jianqing Gao Zhen-Hua Ling 47 0 0 09 Feb 2025
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models Christopher Simic K. Riedhammer Tobias Bocklet 88 0 0 03 Feb 2025
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Andrew Rouditchenko Saurabhchand Bhati Samuel Thomas Hilde Kuehne Rogerio Feris 90 1 0 03 Feb 2025
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim Sungwoo Cho Sangmin Bae Kangwook Jang Se-Young Yun SSL 68 1 0 23 Jan 2025
Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition Rui Liu Hongyu Yuan H. Li 38 0 0 03 Jan 2025
Transferable Adversarial Attacks against ASR Xiaoxue Gao Zexin Li Yiming Chen Cong Liu H. Li AAML 21 1 0 14 Nov 2024
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective Chen Chen Xiaolou Li Zehua Liu Lantian Li D. Wang 18 0 0 29 Sep 2024
Large Language Models are Strong Audio-Visual Speech Recognition Learners Umberto Cappellazzo Minsu Kim Honglie Chen Pingchuan Ma Stavros Petridis Daniele Falavigna Alessio Brutti Maja Pantic 31 9 0 18 Sep 2024
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy? Yiwen Guan V. Trinh Vivek Voleti Jacob Whitehill 32 1 0 13 Sep 2024
DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module Xinyu Wang Qian Wang Haolin Huang Yu Fang Mengjie Xu Qian Wang 21 0 0 31 Aug 2024
Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition Sungnyun Kim Kangwook Jang Sangmin Bae Hoirin Kim Se-Young Yun 29 3 0 04 Jul 2024
OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance Shuheng Ge Haoyu Xing Li Zhang Xiangqian Wu 16 0 0 23 May 2024
ecVoice: Audio Text Extraction and Optimization of Video Based on Idioms Similarity Replacement Jinwei Lin 29 0 0 20 May 2024
Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation Dogucan Yaman Fevziye Irem Eyiokur Leonard Barmann Seymanur Akti H. K. Ekenel Alexander H. Waibel EGVM 15 9 0 07 May 2024
Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction Zhaoxi Mu Xinyu Yang 24 5 0 19 Apr 2024
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer Maxime Burchi Krishna C. Puvvada Jagadeesh Balam Boris Ginsburg Radu Timofte 25 7 0 14 Mar 2024
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang 24 19 0 08 Feb 2024
Robust Dual-Modal Speech Keyword Spotting for XR Headsets Zhuojiang Cai Yuhan Ma Feng Lu 17 0 0 26 Jan 2024
Audio-visual fine-tuning of audio-only ASR models Avner May Dmitriy Serdyuk Ankit Parag Shah Otavio Braga Olivier Siohan 16 3 0 14 Dec 2023
Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism Georgios Milis P. Filntisis A. Roussos Petros Maragos CVBM 19 2 0 11 Dec 2023
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation J. Choi Se Jin Park Minsu Kim Y. Ro 9 12 0 05 Dec 2023
A-JEPA: Joint-Embedding Predictive Architecture Can Listen Zhengcong Fei Mingyuan Fan Junshi Huang 18 17 0 27 Nov 2023
Do VSR Models Generalize Beyond LRS3? Y. A. D. Djilali Sanath Narayan Eustache Le Bihan Haithem Boussaid Ebtesam Almazrouei Merouane Debbah 19 4 0 23 Nov 2023
AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection Sahibzada Adil Shahzad Ammarah Hashmi Yan-Tsung Peng Yu Tsao Hsin-Min Wang 8 5 0 05 Nov 2023
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch Jeff Hwang Moto Hira Caroline Chen Xiaohui Zhang Zhaoheng Ni ... Yumeng Tao Robin Scheibler Samuele Cornell Sean Kim Stavros Petridis 25 22 0 27 Oct 2023
CAD -- Contextual Multi-modal Alignment for Dynamic AVQA Asmar Nadeem Adrian Hilton R. Dawes Graham A. Thomas A. Mustafa 6 9 0 25 Oct 2023
Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading Songtao Luo Shuang Yang Shiguang Shan Xilin Chen 19 1 0 08 Oct 2023
AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition Andrew Rouditchenko R. Collobert Tatiana Likhomanenko VLM 16 3 0 29 Sep 2023
AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement Ju-Chieh Chou Chung-Ming Chien Karen Livescu DiffM 11 4 0 14 Sep 2023
Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signals Louis Blankemeier Sebastien Baur Wei-Hung Weng Jake Garrison Yossi Matias Shruthi Prabhakara Diego Ardila Zaid Nabulsi 24 0 0 11 Sep 2023
SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus Haoxu Wang Fan Yu Xian Shi Yuezhang Wang Shiliang Zhang Ming Li 14 11 0 11 Sep 2023
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge Minsu Kim Jeong Hun Yeo J. Choi Y. Ro 19 16 0 18 Aug 2023
Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping Y. A. D. Djilali Sanath Narayan Haithem Boussaid Ebtesam Almazrouei Merouane Debbah 16 4 0 11 Aug 2023
Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition Guinan Li Jiajun Deng Mengzhe Geng Zengrui Jin Tianzi Wang Shujie Hu Mingyu Cui Helen M. Meng Xunying Liu 24 10 0 06 Jul 2023
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition Yuchen Hu Chen Chen Ruizhe Li Heqing Zou Chng Eng Siong GAN 34 9 0 18 Jun 2023
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Cheng Chen Chengwei Qin Qiu-shi Zhu E. Chng 16 5 0 18 Jun 2023
VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition Ziyi Ni Minglun Han Feilong Chen Linghui Meng Jing Shi Shuang Xu Bo Xu 18 0 0 31 May 2023
Intelligible Lip-to-Speech Synthesis with Speech Units J. Choi Minsu Kim Y. Ro 14 23 0 31 May 2023
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser Yun-hsuan Lai Yen-Chun Chen Y. Wang 6 8 0 27 May 2023
Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Chen Chen Heqing Zou Qiu-shi Zhu E. Chng 23 4 0 16 May 2023
Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models Xiangming Gu Weizhen Zeng Jianan Zhang Longshen Ou Ye Wang 32 6 0 24 Apr 2023
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels Pingchuan Ma A. Haliassos Adriana Fernandez-Lopez Honglie Chen Stavros Petridis M. Pantic 19 104 0 25 Mar 2023
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition Xize Cheng Lin Li Tao Jin Rongjie Huang Wang Lin Zehan Wang Huangdai Liu Yejin Wang Aoxiong Yin Zhou Zhao 13 24 0 09 Mar 2023
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation Mohamed Anwar Bowen Shi Vedanuj Goswami Wei-Ning Hsu J. Pino Changhan Wang 31 35 0 01 Mar 2023
Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English Xiaoming Ren Chao Li Shenjian Wang Biao Li 17 0 0 28 Feb 2023
AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations Jiachen Lian Alexei Baevski Wei-Ning Hsu Michael Auli SSL 24 32 0 10 Feb 2023
Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning Chen Chen Yuchen Hu Qiang Zhang Heqing Zou Beier Zhu E. Chng 12 26 0 10 Dec 2022
Learning to Dub Movies via Hierarchical Prosody Models Gaoxiang Cong Liang Li Yuankai Qi Zhengjun Zha Qi Wu Wen-yu Wang Bin Jiang Ming Yang Qin Huang 52 23 0 08 Dec 2022
Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation Jing-Xuan Zhang Genshun Wan Zhenhua Ling Jia-Yu Pan Jianqing Gao Cong Liu SSL 11 13 0 06 Dec 2022