Neural Dubber: Dubbing for Videos According to Scripts

15 October 2021

Yuxuan Wang

Hang Zhao

DiffM

VGen

ArXiv PDF HTML

Papers citing "Neural Dubber: Dubbing for Videos According to Scripts"

30 / 30 papers shown

Title
FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing Gaoxiang Cong Liang-Sheng Li Jiadong Pan Zhedong Zhang Amin Beheshti A. Hengel Yuankai Qi Qingming Huang 44 0 0 02 May 2025
Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarks Chaoyi Wang Junjie Zheng Zihao Chen Shiyu Xia Chaofan Ding Xiaohao Zhang Xi Tao Xiaoming He Xinhan Di AuLLM 39 0 0 30 Apr 2025
AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation J. Choi Ji-Hoon Kim Kim Sung-Bin Tae-Hyun Oh Joon Son Chung DiffM 48 0 0 29 Apr 2025
VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models Kim Sung-Bin Jeongsoo Choi Puyuan Peng Joon Son Chung Tae-Hyun Oh David F. Harwath VGen 45 1 0 03 Apr 2025
DeepDubber-V1: Towards High Quality and Dialogue, Narration, Monologue Adaptive Movie Dubbing Via Multi-Modal Chain-of-Thoughts Reasoning Guidance Junjie Zheng Zihao Chen Chaofan Ding Xinhan Di VGen 62 1 0 31 Mar 2025
EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversion Ashishkumar Gudmalwar Ishan D. Biyani Nirmesh J. Shah Pankaj Wasnik R. Shah DiffM 21 0 0 31 Dec 2024
HELPNet: Hierarchical Perturbations Consistency and Entropy-guided Ensemble for Scribble Supervised Medical Image Segmentation Xiao Zhang Shaoxuan Wu Peilin Zhang Zhuo Jin Xiaosong Xiong Qirong Bu Jingkun Chen Jun Feng 84 2 0 25 Dec 2024
Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation Lucas Goncalves Prashant Mathur Xing Niu Brady Houston Chandrashekhar Lavania Srikanth Vishnubhotla Lijia Sun Anthony Ferritto 59 0 0 21 Dec 2024
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing Gaoxiang Cong Jiadong Pan Liang-Sheng Li Yuankai Qi Yuxin Peng A. Hengel Jian Yang Qingming Huang 90 6 0 12 Dec 2024
Self-Supervised Audio-Visual Soundscape Stylization Tingle Li Renhao Wang Po-Yao Huang Andrew Owens Gopala Anumanchipalli DiffM SSL 22 4 0 22 Sep 2024
FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications Hao-Han Guo Kun Liu Fei-Yu Shen Yi-Chen Wu Xu Tang Kun Xie Kai-Tuo Xu Kun Xie Kai-Tuo Xu 30 19 0 05 Sep 2024
Synchronous Multi-modal Semantic Communication System with Packet-level Coding Yun Tian Jingkai Ying Zhijin Qin Ye Jin Xiaoming Tao 20 3 0 08 Aug 2024
The Tug-of-War Between Deepfake Generation and Detection Hannah Lee Changyeon Lee Kevin Farhat Lin Qiu Steve Geluso Aerin Kim O. Etzioni 28 1 0 08 Jul 2024
DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing Neha Sahipjohn Ashishkumar Gudmalwar Nirmesh Shah Pankaj Wasnik R. Shah 24 5 0 13 Jun 2024
Towards Accurate Lip-to-Speech Synthesis in-the-Wild Sindhu B. Hegde Rudrabha Mukhopadhyay C. V. Jawahar Vinay P. Namboodiri 27 4 0 02 Mar 2024
StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing Gaoxiang Cong Yuankai Qi Liang-Sheng Li Amin Beheshti Zhedong Zhang A. Hengel Ming-Hsuan Yang Chenggang Yan Qingming Huang 30 12 0 20 Feb 2024
DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin Tao Li Chenxu Hu Jian Cong Xinfa Zhu Jingbei Li Qiao Tian Yuping Wang Linfu Xie DiffM 17 8 0 02 Sep 2023
High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units Junchen Lu Berrak Sisman Mingyang Zhang Haizhou Li 11 4 0 29 Jun 2023
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models Simian Luo Chuanhao Yan Chenxu Hu Hang Zhao DiffM 11 79 0 29 Jun 2023
Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters Proyag Pal Brian Thompson Yogesh Virkar Prashant Mathur Alexandra Chronopoulou Marcello Federico 25 4 0 22 May 2023
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition Xize Cheng Lin Li Tao Jin Rongjie Huang Wang Lin Zehan Wang Huangdai Liu Yejin Wang Aoxiong Yin Zhou Zhao 13 24 0 09 Mar 2023
Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing Alexandra Chronopoulou Brian Thompson Prashant Mathur Yogesh Virkar Surafel Melaku Lakew Marcello Federico 11 7 0 25 Feb 2023
Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing William Brannon Yogesh Virkar Brian Thompson 31 21 0 23 Dec 2022
Learning to Dub Movies via Hierarchical Prosody Models Gaoxiang Cong Liang Li Yuankai Qi Zhengjun Zha Qi Wu Wen-yu Wang Bin Jiang Ming Yang Qin Huang 52 23 0 08 Dec 2022
Controllable and Lossless Non-Autoregressive End-to-End Text-to-Speech Zhengxi Liu Qiao Tian Chenxu Hu Xudong Liu Meng-Che Wu Yuping Wang Hang Zhao Yuxuan Wang 18 10 0 13 Jul 2022
Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcasts Debjoy Saha Shravan Nayak Timo Baumann 9 3 0 24 May 2022
Learning Visual Styles from Audio-Visual Associations Tingle Li Yichen Liu Andrew Owens Hang Zhao DiffM 17 20 0 10 May 2022
More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech Michael Hassid Michelle Tadmor Ramanovich Brendan Shillingford Miaosen Wang Ye Jia Tal Remez DiffM 9 16 0 19 Nov 2021
Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis Ye Jia Yu Zhang Ron J. Weiss Quan Wang Jonathan Shen ... Z. Chen Patrick Nguyen Ruoming Pang Ignacio López Moreno Yonghui Wu 199 817 0 12 Jun 2018
Lip Reading Sentences in the Wild Joon Son Chung A. Senior Oriol Vinyals Andrew Zisserman 162 782 0 16 Nov 2016