Learning Speech-driven 3D Conversational Gestures from Video

13 February 2021

Hans-Peter Seidel

Papers citing "Learning Speech-driven 3D Conversational Gestures from Video"

50 / 67 papers shown

Title
Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication Jinhe Huang Yongkang Cheng Yuming Hang Gaoge Han J. Li Jing Zhang Xingjian Gu 41 0 0 08 May 2025
$Co$^{3}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion$ Co $^{3}$ Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion Xingqun Qi Yatian Wang Hengyuan Zhang J. Pan Wei Xue Shanghang Zhang Wenhan Luo Qifeng Liu Yike Guo SLR 53 0 0 03 May 2025
OT-Talk: Animating 3D Talking Head with Optimal Transportation Xinmu Wang Xiang Gao Xiyun Song Heather Yu Zongfang Lin Liang Peng Xianfeng Gu 10 0 0 03 May 2025
EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation Xiangyue Zhang Jianfang Li Jiaxu Zhang Jianqiang Ren Liefeng Bo Zhigang Tu 20 0 0 12 Apr 2025
DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech Yongkang Cheng Shaoli Huang Xuelin Chen J. Ning Mingming Gong DiffM 47 1 0 21 Mar 2025
MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization Binjie Liu Lina Liu Sanyi Zhang Songen Gu Yihao Zhi Tianyi Zhu Lei Yang Long Ye SLR 68 0 0 18 Mar 2025
Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers Yasheng Sun Zhiliang Xu Hang Zhou Jiazhi Guan Quanwei Yang ... Yingying Li Haocheng Feng J. Wang Ziwei Liu Koike Hideki VGen 54 0 0 13 Mar 2025
ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis Xukun Zhou Fengxin Li Ming Chen Yan Zhou Pengfei Wan Di Zhang Yeying Jin Zhaoxin Fan Hongyan Liu Jun He DiffM VGen 43 0 0 09 Mar 2025
Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters S. Hogue Chenxu Zhang Yapeng Tian Xiaohu Guo DiffM 64 0 0 18 Dec 2024
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion Changan Chen Juze Zhang S. K. Lakshmikanth Yusu Fang Ruizhi Shao Gordon Wetzstein L. Fei-Fei Ehsan Adeli VGen 65 3 0 13 Dec 2024
Multi-Resolution Generative Modeling of Human Motion from Limited Data David Eduardo Moreno-Villamarín A. Hilsmann Peter Eisert DiffM 3DH 81 0 0 25 Nov 2024
Mitigation of gender bias in automatic facial non-verbal behaviors generation Alice Delbosc M. Ochs Nicolas Sabouret Brian Ravenet Stéphane Ayache 21 0 0 09 Oct 2024
LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis Haozhou Pang Tianwei Ding Lanshan He Ming Tao Lu Zhang Qi Gan 18 1 0 06 Oct 2024
Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation Bohong Chen Yumeng Li Yao-Xiang Ding Tianjia Shao Kun Zhou 27 7 0 01 Oct 2024
FastTalker: Jointly Generating Speech and Conversational Gestures from Text Zixin Guo Jian Zhang 24 1 0 24 Sep 2024
2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation? Teo Guichoux Laure Soulier Nicolas Obin Catherine Pelachaud SLR 27 0 0 16 Sep 2024
T3M: Text Guided 3D Human Motion Synthesis from Speech Wenshuo Peng Kaipeng Zhang Sai Qian Zhang 20 0 0 23 Aug 2024
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony Chao Xu Mingze Sun Zhi-Qi Cheng Fei-Yue Wang Yang Liu Baigui Sun Ruqi Huang Alexander G. Hauptmann VGen 29 2 0 18 Aug 2024
CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimation Ci Li Elin Hernlund Hedvig Kjellström Silvia Zuffi 3DH 26 2 0 01 Jul 2024
Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective Face and Body Expressions from Affordable Inputs Uttaran Bhattacharya Aniket Bera Dinesh Manocha CVBM 27 2 0 26 Jun 2024
Investigating the impact of 2D gesture representation on co-speech gesture generation Teo Guichoux Laure Soulier Nicolas Obin Catherine Pelachaud SLR 19 0 0 21 Jun 2024
RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text Jiaben Chen Xin Yan Yihang Chen Siyuan Cen Qinwei Ma Haoyu Zhen Kaizhi Qian Lie Lu Chuang Gan 32 0 0 30 May 2024
CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild Xingqun Qi Hengyuan Zhang Yatian Wang J. Pan Chen Liu ... Qixun Zhang Shanghang Zhang Wenhan Luo Qifeng Liu Qi-fei Liu DiffM SLR 108 5 0 27 May 2024
SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models Qingrong Cheng Xu Li Xinghui Fu DiffM 27 2 0 22 May 2024
Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis Zeyi Zhang Tenglong Ao Yuyao Zhang Qingzhe Gao Chuan Lin Baoquan Chen Libin Liu SLR 21 15 0 16 May 2024
Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis Shivam Mehta Anna Deichler Jim O'Regan Birger Moëll Jonas Beskow G. Henter Simon Alexanderson 34 4 0 30 Apr 2024
Towards Variable and Coordinated Holistic Co-Speech Motion Generation Yifei Liu Qiong Cao Yandong Wen Huaiguang Jiang Changxing Ding SLR 63 13 0 30 Mar 2024
Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication Mingze Sun Chao Xu Xinyu Jiang Yang Liu Baigui Sun Ruqi Huang 36 3 0 28 Mar 2024
ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis Muhammad Hamza Mughal Rishabh Dabral I. Habibie Lucia Donatelli Marc Habermann Christian Theobalt SLR 30 14 0 26 Mar 2024
MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models Zunnan Xu Yukang Lin Haonan Han Sicheng Yang Ronghui Li Yachao Zhang Xiu Li Mamba 46 24 0 14 Mar 2024
NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interaction Snehesh Shrestha Yantian Zha Saketh Banagiri Ge Gao Yiannis Aloimonos Cornelia Fermuller 26 5 0 04 Mar 2024
DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation Junming Chen Yunfei Liu Jianan Wang Ailing Zeng Yu Li Qifeng Chen VGen 25 10 0 09 Jan 2024
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling Haiyang Liu Zihao Zhu Giorgio Becherini Yichen Peng Mingyang Su You Zhou Xuefei Zhe Naoya Iwamoto Bo Zheng Michael J. Black SLR 18 29 0 31 Dec 2023
BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics Wenqian Zhang Molin Huang Yuxuan Zhou Juze Zhang Jingyi Yu Jingya Wang Lan Xu 21 5 0 13 Dec 2023
Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion Kiran Chhatre Radek Danvevcek Nikos Athanasiou Giorgio Becherini Christopher Peters Michael J. Black Timo Bolkart DiffM 27 14 0 07 Dec 2023
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation Xingqun Qi Jiahao Pan Peng Li Ruibin Yuan Xiaowei Chi ... Wenhan Luo Wei Xue Shanghang Zhang Qi-fei Liu Yi-Ting Guo SLR 26 11 0 29 Nov 2023
SpeechAct: Towards Generating Whole-body Motion from Speech Jinsong Zhang Minjie Zhu Yuxiang Zhang Yebin Liu Kun Li 21 0 0 29 Nov 2023
HumanTOMATO: Text-aligned Whole-body Motion Generation Shunlin Lu Ling-Hao Chen Ailing Zeng Jing Lin Ruimao Zhang Lei Zhang H. Shum VGen 24 60 0 19 Oct 2023
State of the Art on Diffusion Models for Visual Computing Ryan Po Wang Yifan Vladislav Golyanik Kfir Aberman Jonathan T. Barron ... Matthias Nießner Bjorn Ommer Christian Theobalt Peter Wonka Gordon Wetzstein 22 101 0 11 Oct 2023
ACT2G: Attention-based Contrastive Learning for Text-to-Gesture Generation Hitoshi Teshima Naoki Wake Diego Thomas Yuta Nakashima Hiroshi Kawasaki Katsushi Ikeuchi 20 0 0 28 Sep 2023
The Importance of Multimodal Emotion Conditioning and Affect Consistency for Embodied Conversational Agents Che-Jui Chang Samuel S. Sohn Sen Zhang R. Jayashankar Muhammad Usman Mubbasir Kapadia 18 7 0 26 Sep 2023
Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent Alice Delbosc M. Ochs Nicolas Sabouret Brian Ravenet Stéphane Ayache 21 7 0 15 Sep 2023
UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons Sicheng Yang Z. Wang Zhiyong Wu Minglei Li Zhensong Zhang ... Lei Hao Songcen Xu Xiaofei Wu Changpeng Yang Zonghong Dai DiffM 29 14 0 13 Sep 2023
BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer Kunkun Pang Dafei Qin Yingruo Fan Julian Habekost Takaaki Shiratori Junichi Yamagishi Taku Komura SLR ViT 14 19 0 07 Sep 2023
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion Se Jin Park Joanna Hong Minsu Kim Y. Ro 19 3 0 23 Aug 2023
Human Motion Generation: A Survey Wentao Zhu Xiaoxuan Ma Dongwoo Ro Hai Ci Jinlu Zhang Jiaxin Shi Feng Gao Qi Tian Yizhou Wang VGen 19 50 0 20 Jul 2023
EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model Li-Ping Yin Yijun Wang Tianyu He Jinming Liu Wei Zhao Bohan Li Xin Jin Jianxin Lin DiffM 9 14 0 20 Jun 2023
QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation Sicheng Yang Zhiyong Wu Minglei Li Zhensong Zhang Lei Hao Weihong Bao Hao-Wen Zhuang SLR 11 40 0 18 May 2023
DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models Sicheng Yang Zhiyong Wu Minglei Li Zhensong Zhang Lei Hao Weihong Bao Ming Cheng Long Xiao 11 64 0 08 May 2023
GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents Tenglong Ao Zeyi Zhang Libin Liu DiffM VGen 65 143 0 26 Mar 2023