Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

13 June 2024

Qingkun Su

Ce Liu

Yao Yao

Siyu Zhu

VGen

ArXiv PDF HTML

Papers citing "Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation"

50 / 56 papers shown

Title
DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution Ming-Hui Liu Xiao-Qian Liu Xin Luo Xin-Shun Xu 51 0 0 07 May 2025
KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution Antoni Bigata Rodrigo Mira Stella Bounareli Michał Stypułkowski Konstantinos Vougioukas Stavros Petridis Maja Pantic 49 0 0 01 May 2025
Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation Weipeng Tan Chuming Lin Chengming Xu F. Xu Xiaobin Hu Xiaozhong Ji Junwei Zhu Chengjie Wang Yanwei Fu 44 0 0 25 Apr 2025
FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis Mengchao Wang Qiang Wang Fan Jiang Yaqi Fan Yunpeng Zhang Yonggang Qi Kun Zhao Mu Xu DiffM VGen 29 0 0 07 Apr 2025
DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion Maksim Siniukov Di Chang Minh Tran Hongkun Gong Ashutosh Chaubey Mohammad Soleymani DiffM VGen 23 0 0 05 Apr 2025
OmniTalker: Real-Time Text-Driven Talking Head Generation with In-Context Audio-Visual Style Replication Zhongjian Wang Peng Zhang Jinwei Qi Guangyuan Wang Sheng Xu Bang Zhang Liefeng Bo DiffM VGen 36 0 0 03 Apr 2025
Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation Fa-Ting Hong Zunnan Xu Zixiang Zhou Jun Zhou Xiu Li Qin Lin Qinglin Lu D. Xu DiffM VGen 51 2 0 03 Apr 2025
MoCha: Towards Movie-Grade Talking Character Synthesis Cong Wei Bo Sun Haoyu Ma Ji Hou F. Xu ... Kunpeng Li Tingbo Hou Animesh Sinha Peter Vajda Wenhu Chen VGen 63 0 0 30 Mar 2025
AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers Jiazhi Guan Kaisiyuan Wang Zhiliang Xu Quanwei Yang Yasheng Sun ... Errui Ding J. Wang Youjian Zhao Hang Zhou Ziwei Liu VGen 44 0 0 25 Mar 2025
MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation Yukang Lin Hokit Fung Jianjin Xu Zeping Ren Adela S.M. Lau Guosheng Yin Xiu Li VGen 39 5 0 25 Mar 2025
Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation Dingcheng Zhen Shunshun Yin Shiyang Qin Hou Yi Ziwei Zhang Siyuan Liu Gan Qi Ming Tao VGen 69 0 0 24 Mar 2025
DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model Kangwei Liu Junwu Liu Yun Cao Jinlin Guo Xiaowei Yi DiffM 41 0 0 24 Mar 2025
Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model Yingying Fan Quanwei Yang Kaisiyuan Wang Hang Zhou Yingying Li Haocheng Feng Errui Ding Y. Wu J. Wang DiffM 42 0 0 21 Mar 2025
PoseTraj: Pose-Aware Trajectory Control in Video Diffusion Longbin Ji Lei Zhong Pengfei Wei Changjian Li DiffM VGen 39 0 0 20 Mar 2025
ExDDV: A New Dataset for Explainable Deepfake Detection in Video Vlad Hondru Eduard Hogea Darian M. Onchis Radu Tudor Ionescu 56 1 0 18 Mar 2025
SyncDiff: Diffusion-based Talking Head Synthesis with Bottlenecked Temporal Visual Prior for Improved Synchronization Xulin Fan Heting Gao Ziyi Chen Peng Chang Mei Han Mark Hasegawa-Johnson DiffM 55 0 0 17 Mar 2025
RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editing Tianrui Pan Lin Liu Jie Liu X. Zhang J. Tang Gangshan Wu Q. Tian DiffM VGen 46 0 0 14 Mar 2025
FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis Ziqi Ni Ao Fu Yi Zhou 61 0 0 06 Mar 2025
KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation Antoni Bigata Michał Stypułkowski Rodrigo Mira Stella Bounareli Konstantinos Vougioukas Zoe Landgraf Nikita Drobyshev Maciej Ziȩba Stavros Petridis M. Pantic DiffM VGen 63 2 0 03 Mar 2025
FLAP: Fully-controllable Audio-driven Portrait Video Generation through 3D head conditioned diffusion model Lingzhou Mu Baiji Liu Ruonan Zhang Guiming Mo Jiawei Jin Kai Zhang Haozhi Huang DiffM VGen 53 1 0 26 Feb 2025
LAM: Large Avatar Model for One-shot Animatable Gaussian Head Yisheng He Xiaodong Gu Xiaodan Ye Chao Xu Zhengyi Zhao Yuan Dong Weihao Yuan Zilong Dong Liefeng Bo 3DGS 74 0 0 25 Feb 2025
X-Dancer: Expressive Music to Human Dance Video Generation Zeyuan Chen Hongyi Xu Guoxian Song You Xie Chenxu Zhang X. Chen Chao Wang Di Chang Linjie Luo VGen 39 0 0 24 Feb 2025
SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion Junxian Ma Shiwen Wang Jian Yang Junyi Hu Jian Liang Guosheng Lin Jingbo Chen Kai Li Yu Meng DiffM VGen 61 3 0 17 Feb 2025
OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models Gaojie Lin Jianwen Jiang Jiaqi Yang Zerong Zheng Chao Liang DiffM VGen 167 11 0 03 Feb 2025
Joint Learning of Depth and Appearance for Portrait Image Animation Xinya Ji Gaspard Zoss Prashanth Chandran Lingchen Yang Xun Cao B. Solenthaler D. Bradley 3DH MDE 42 0 0 15 Jan 2025
DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis Kaijun Deng Dezhi Zheng Jindong Xie Jinbao Wang Weicheng Xie L. Shen Siyang Song 3DGS 27 0 0 31 Dec 2024
FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation Tianyun Zhong Chao Liang Jianwen Jiang Gaojie Lin Jiaqi Yang Zhou Zhao DiffM 94 1 0 22 Dec 2024
Real-time One-Step Diffusion-based Expressive Portrait Videos Generation Hanzhong Guo Hongwei Yi Daquan Zhou Alexander William Bergman Michael Lingelbach Yizhou Yu DiffM 73 1 0 18 Dec 2024
Playable Game Generation Mingyu Yang Junyou Li Zhongbin Fang Sheng Chen Yangbin Yu Qiang Fu Wei Yang Deheng Ye VGen 76 7 0 01 Dec 2024
Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer Jiahao Cui Hui Li Yun Zhan Hanlin Shang K. Cheng Yuqi Ma Shan Mu Hang Zhou Jingdong Wang Siyu Zhu ViT VGen 99 6 0 01 Dec 2024
Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook Florinel-Alin Croitoru Andrei Iulian Hiji Vlad Hondru Nicolae-Cătălin Ristea Paul Irofti Marius Popescu Cristian Rusu Radu Tudor Ionescu F. Khan Mubarak Shah 79 2 0 29 Nov 2024
OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation Hui Li Mingwang Xu Yun Zhan Shan Mu Jiaye Li ... Y. Chen Tan Chen Mao Ye Jingdong Wang Siyu Zhu VGen 99 2 0 28 Nov 2024
Sonic: Shifting Focus to Global Audio Perception in Portrait Animation Xiaozhong Ji Xiaobin Hu Zhihong Xu Junwei Zhu Chuming Lin ... Donghao Luo Yi Chen Qin Lin Qinglin Lu Chengjie Wang VGen 65 3 0 25 Nov 2024
EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation Rang Meng Xingyu Zhang Yuming Li Chenguang Ma 26 5 0 15 Nov 2024
JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation Xuyang Cao Guoxin Wang Sheng Shi Jun Zhao Yang Yao Jintao Fei Minyu Gao VGen 37 1 0 14 Nov 2024
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation Hanbo Cheng Limin Lin Chenyu Liu Pengcheng Xia Pengfei Hu Jiefeng Ma Jun Du Jia Pan DiffM VGen 59 0 0 17 Oct 2024
MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling Yue Zhang Minhao Liu Zhaokang Chen Bin Wu Yubin Zeng Chao Zhan Yingjie He Junxin Huang Wenjiang Zhou Wenjiang Zhou 34 6 0 14 Oct 2024
Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation Jiahao Cui Hui Li Yao Yao Hao Zhu Hanlin Shang Kaihui Cheng Hang Zhou Siyu Zhu Jingdong Wang DiffM VGen 34 22 0 10 Oct 2024
JoyHallo: Digital human model for Mandarin Sheng Shi Xuyang Cao Jun Zhao Guoxin Wang VGen 26 1 0 20 Sep 2024
PainDiffusion: Learning to Express Pain Quang Tien Dam Tri Tung Nguyen Nguyen D. Tran Joo-Ho Lee Joo-Ho Lee VGen 30 0 0 18 Sep 2024
SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model Weipeng Tan Chuming Lin Chengming Xu Xiaozhong Ji Junwei Zhu Chengjie Wang Yanwei Fu DiffM 22 0 0 05 Sep 2024
CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention Gaojie Lin Jianwen Jiang Chao Liang Tianyun Zhong Jiaqi Yang Yanbo Zheng VGen DiffM 61 13 0 03 Sep 2024
MegActor- $Σ$ : Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer Shurong Yang Huadong Li Juhao Wu Minhao Jing Linze Li Renhe Ji Jiajun Liang Haoqiang Fan Jin Wang VGen DiffM 35 9 0 27 Aug 2024
4D Diffusion for Dynamic Protein Structure Prediction with Reference Guided Motion Alignment Kaihui Cheng Ce Liu Qingkun Su Jun Wang Liwei Zhang Yining Tang Yao Yao Siyu Zhu Yuan Qi DiffM 18 0 0 22 Aug 2024
Dynamic PDB: A New Dataset and a SE(3) Model Extension by Integrating Dynamic Behaviors and Physical Properties in Protein Structures Ce Liu Jun Wang Zhiqiang Cai Yingxu Wang Huizhen Kuang ... Yining Tang Fenglei Cao Limei Han Siyu Zhu Yuan Qi 3DV 13 0 0 22 Aug 2024
DEGAS: Detailed Expressions on Full-Body Gaussian Avatars Zhijing Shao D. B. Wang Qing-Yao Tian Yao-Dong Yang Hengyu Meng Zeyu Cai Bo Dong Yu Zhang Kang Zhang Z. Wang 3DGS 34 3 0 20 Aug 2024
JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Language Model Farzaneh Jafari Stefano Berretti Anup Basu Mamba 29 1 0 03 Aug 2024
LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement Rui Zhang Yixiao Fang Zhen-Zhong Lu Pei Cheng Zebiao Huang Bin-Bin Fu DiffM VGen 32 1 0 26 Jul 2024
EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions Zhiyuan Chen Jiajiong Cao Zhiquan Chen Yuming Li Chenguang Ma VGen 16 48 0 11 Jul 2024
AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding Tao Liu Feilong Chen Shuai Fan Chenpeng Du Qi Chen Xie Chen Kai Yu DiffM PINN 34 25 0 06 May 2024