DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation

10 January 2023

Wenliang Zhao

Jie Zhou

Papers citing "DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation"

50 / 79 papers shown

Title
SynSHRP2: A Synthetic Multimodal Benchmark for Driving Safety-critical Events Derived from Real-world Driving Data Liang Shi Boyu Jiang Zhenyuan Yuan Miguel A. Perez Feng Guo 29 0 0 06 May 2025
KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution Antoni Bigata Rodrigo Mira Stella Bounareli Michał Stypułkowski Konstantinos Vougioukas Stavros Petridis Maja Pantic 52 0 0 01 May 2025
Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation Weipeng Tan Chuming Lin Chengming Xu F. Xu Xiaobin Hu Xiaozhong Ji Junwei Zhu Chengjie Wang Yanwei Fu 44 0 0 25 Apr 2025
SE4Lip: Speech-Lip Encoder for Talking Head Synthesis to Solve Phoneme-Viseme Alignment Ambiguity Yihuan Huang Jiajun Liu Yanzhen Ren Wuyang Liu Juhua Tang 21 0 0 08 Apr 2025
BrainMRDiff: A Diffusion Model for Anatomically Consistent Brain MRI Synthesis Moinak Bhattacharya Saumya Gupta Annie Singh C. L. P. Chen Gagandeep Singh Prateek Prasanna MedIm 26 0 0 06 Apr 2025
MoCha: Towards Movie-Grade Talking Character Synthesis Cong Wei Bo Sun Haoyu Ma Ji Hou F. Xu ... Kunpeng Li Tingbo Hou Animesh Sinha Peter Vajda Wenhu Chen VGen 105 0 0 30 Mar 2025
STSA: Spatial-Temporal Semantic Alignment for Visual Dubbing Zijun Ding Mingdie Xiong Congcong Zhu Jingrun Chen DiffM 56 0 0 29 Mar 2025
DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model Kangwei Liu Junwu Liu Yun Cao Jinlin Guo Xiaowei Yi DiffM 41 0 0 24 Mar 2025
Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation Dingcheng Zhen Shunshun Yin Shiyang Qin Hou Yi Ziwei Zhang Siyuan Liu Gan Qi Ming Tao VGen 69 0 0 24 Mar 2025
3D Engine-ready Photorealistic Avatars via Dynamic Textures Yifan Wang Ivan Molodetskikh Ondrej Texler Dimitar Dinev 43 0 0 19 Mar 2025
SyncDiff: Diffusion-based Talking Head Synthesis with Bottlenecked Temporal Visual Prior for Improved Synchronization Xulin Fan Heting Gao Ziyi Chen Peng Chang Mei Han Mark Hasegawa-Johnson DiffM 57 0 0 17 Mar 2025
Unlock Pose Diversity: Accurate and Efficient Implicit Keypoint-based Spatiotemporal Diffusion for Audio-driven Talking Portrait Chaolong Yang Kai Yao Yuyao Yan Chenru Jiang Weiguang Zhao Jie Sun Guangliang Cheng Yifei Zhang Bin Dong K. Huang DiffM 67 0 0 17 Mar 2025
Personalized Generation In Large Model Era: A Survey Yiyan Xu Jinghao Zhang Alireza Salemi Xinting Hu W. Wang Fuli Feng Hamed Zamani Xiangnan He Tat-Seng Chua 3DV 79 2 0 04 Mar 2025
Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation Baptiste Chopin Tashvik Dhamija P. Balaji Yaohui Wang A. Dantcheva DiffM VGen 44 0 0 24 Feb 2025
SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion Junxian Ma Shiwen Wang Jian Yang Junyi Hu Jian Liang Guosheng Lin Jingbo Chen Kai Li Yu Meng DiffM VGen 61 3 0 17 Feb 2025
DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis Kaijun Deng Dezhi Zheng Jindong Xie Jinbao Wang Weicheng Xie L. Shen Siyang Song 3DGS 32 0 0 31 Dec 2024
Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters S. Hogue Chenxu Zhang Yapeng Tian Xiaohu Guo DiffM 69 0 0 18 Dec 2024
UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models Yuning Han Bingyin Zhao Rui Chu Feng Luo Biplab Sikdar Yingjie Lao DiffM AAML 72 1 0 16 Dec 2024
Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation Zilyu Ye Zhiyang Chen Tiancheng Li Zemin Huang Weijian Luo Guo-jun Qi DiffM 78 5 0 02 Dec 2024
Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer Jiahao Cui Hui Li Yun Zhan Hanlin Shang K. Cheng Yuqi Ma Shan Mu Hang Zhou Jingdong Wang Siyu Zhu ViT VGen 99 6 0 01 Dec 2024
Sonic: Shifting Focus to Global Audio Perception in Portrait Animation Xiaozhong Ji Xiaobin Hu Zhihong Xu Junwei Zhu Chuming Lin ... Donghao Luo Yi Chen Qin Lin Qinglin Lu Chengjie Wang VGen 73 3 0 25 Nov 2024
Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts Xiang Deng Youxin Pang Xiaochen Zhao Chao Xu Lizhen Wang Hongjiang Xiao Shi Yan Hongwen Zhang Yebin Liu DiffM VGen 38 1 0 31 Oct 2024
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation Hanbo Cheng Limin Lin Chenyu Liu Pengcheng Xia Pengfei Hu Jiefeng Ma Jun Du Jia Pan DiffM VGen 107 0 0 17 Oct 2024
Titanic Calling: Low Bandwidth Video Conference from the Titanic Wreck Fevziye Irem Eyiokur Christian Huber Thai-Binh Nguyen T. Nguyen Fabian Retkowski Enes Yavuz Ugan Dogucan Yaman Alexander Waibel 27 0 0 15 Oct 2024
Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation Jiahao Cui Hui Li Yao Yao Hao Zhu Hanlin Shang Kaihui Cheng Hang Zhou Siyu Zhu Jingdong Wang DiffM VGen 38 22 0 10 Oct 2024
LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details Jian Yang Xukun Wang Wentao Wang Guoming Li Qihang Fang Ruihong Yuan Tianyang Wang Jason Zhaoxin Fan Yeying Jin Zhaoxin Fan VGen 41 1 0 01 Oct 2024
Text Image Generation for Low-Resource Languages with Dual Translation Learning Chihiro Noguchi Shun Fukuda Shoichiro Mihara Masao Yamanaka DiffM 26 0 0 26 Sep 2024
FaceVid-1K: A Large-Scale High-Quality Multiracial Human Face Video Dataset Donglin Di H. Feng Wenzhang Sun Yongjia Ma Hao Li Wei Chen Xiaofei Gou Tonghua Su Xun Yang CVBM 46 2 0 23 Sep 2024
SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model Weipeng Tan Chuming Lin Chengming Xu Xiaozhong Ji Junwei Zhu Chengjie Wang Yanwei Fu DiffM 41 0 0 05 Sep 2024
G3FA: Geometry-guided GAN for Face Animation Alireza Javanmardi A. Pagani Didier Stricker CVBM 3DH 32 2 0 23 Aug 2024
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony Chao Xu Mingze Sun Zhi-Qi Cheng Fei-Yue Wang Yang Liu Baigui Sun Ruqi Huang Alexander G. Hauptmann VGen 37 2 0 18 Aug 2024
Content and Style Aware Audio-Driven Facial Animation Qingju Liu Hyeongwoo Kim Gaurav Bharaj DiffM 30 1 0 13 Aug 2024
High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model Weizhi Zhong Junfan Lin Peixin Chen Liang Lin Guanbin Li 34 1 0 10 Aug 2024
Style-Preserving Lip Sync via Audio-Aware Style Reference Weizhi Zhong Jichang Li Yinqi Cai Liang Lin Guanbin Li 26 2 0 10 Aug 2024
ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer Jiazhi Guan Zhiliang Xu Hang Zhou Kaisiyuan Wang Shengyi He ... Errui Ding Jingtuo Liu Jingdong Wang Youjian Zhao Ziwei Liu VGen 46 2 0 06 Aug 2024
GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer Yihong Lin Zhaoxin Fan Lingyu Xiong Liang Peng Xiandong Li Wenxiong Kang Xianjia Wu Songju Lei Huang Xu 34 3 0 03 Aug 2024
Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation Jintao Tan Xize Cheng Lingyu Xiong Lei Zhu Xiandong Li Xianjia Wu Kai Gong Minglei Li Yi Cai DiffM 28 2 0 03 Aug 2024
JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Language Model Farzaneh Jafari Stefano Berretti Anup Basu Mamba 31 1 0 03 Aug 2024
D $^4$ M: Dataset Distillation via Disentangled Diffusion Model Duo Su Junjie Hou Weizhi Gao Yingjie Tian Bowen Tang DD 35 18 0 21 Jul 2024
Text-based Talking Video Editing with Cascaded Conditional Diffusion Bo Han Heqing Zou Haoyang Li Guangcong Wang Chng Eng Siong VGen DiffM 29 2 0 20 Jul 2024
One-Shot Pose-Driving Face Animation Platform He Feng Donglin Di Yongjia Ma Wei Chen Tonghua Su CVBM 21 1 0 12 Jul 2024
LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control Jianzhu Guo Dingyun Zhang Xiaoqiang Liu Zhizhou Zhong Yuan Zhang Pengfei Wan Di Zhang VGen 59 53 0 03 Jul 2024
RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network Xiaozhong Ji Chuming Lin Zhonggan Ding Ying Tai Junwei Zhu Xiaobin Hu Donghao Luo Yanhao Ge Chengjie Wang CVBM 24 2 0 26 Jun 2024
A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing Ming Meng Yufei Zhao Bo Zhang Yonggui Zhu Weimin Shi Maxwell Wen Zhaoxin Fan VGen 34 1 0 15 Jun 2024
Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation Mingwang Xu Hui Li Qingkun Su Hanlin Shang Liwei Zhang Ce Liu Jingdong Wang Yao Yao Siyu Zhu VGen 29 67 0 13 Jun 2024
Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation Jiadong Liang Feng Lu CVBM 29 0 0 12 Jun 2024
OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance Shuheng Ge Haoyu Xing Li Zhang Xiangqian Wu 31 0 0 23 May 2024
Embedded Representation Learning Network for Animating Styled Video Portrait Tianyong Wang Xiangyu Liang Wangguandong Zheng Dan Niu Haifeng Xia Siyu Xia 3DH 16 0 0 29 Apr 2024
EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis Shuai Tan Bin Ji Mengxiao Bi Ye Pan 33 26 0 02 Apr 2024
MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation Seyeon Kim Siyoon Jin Jihye Park Kihong Kim Jiyoung Kim Jisu Nam Seungryong Kim DiffM VGen 58 3 0 28 Mar 2024