Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity

ACM Transactions on Graphics (TOG), 2020

4 September 2020

ArXiv (abs)PDF HTML Github (264★)

Papers citing "Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity"

50 / 168 papers shown

Title
Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation Huan Yang Jiahui Chen Chaofan Ding Runhua Shi Siyu Xiong Qingqi Hong Xiaoqi Mo Xinhan Di 137 1 0 26 Sep 2024
FastTalker: Jointly Generating Speech and Conversational Gestures from Text Zixin Guo Jian Zhang 351 4 0 24 Sep 2024
2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?International Conference on Intelligent Virtual Agents (IVA), 2024 Teo Guichoux Laure Soulier Nicolas Obin Catherine Pelachaud SLR 173 1 0 16 Sep 2024
DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures S. Hogue Chenxu Zhang Hamza Daruger Yapeng Tian Xiaohu Guo VGen 235 21 0 11 Sep 2024
Gesture Generation from Trimodal Context for Humanoid RobotsInternational Conference on Human-Agent Interaction (HAI), 2024 Shiyi Tang Christian Dondrup 113 2 0 08 Sep 2024
Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic EvaluationInternational Conference on Multimodal Interaction (ICMI), 2024 E. Ghaleb Bulat Khaertdinov Wim Pouw Marlou Rasenberg Judith Holler Aslı Özyürek Raquel Fernández SSL 183 2 0 31 Aug 2024
Empowering Sign Language Communication: Integrating Sentiment and Semantics for Facial Expression SynthesisComputers & graphics (CG), 2024 Rafael Azevedo Thiago M. Coutinho Joao Klock Ferreira Thiago L. Gomes Erickson R. Nascimento SLR 179 8 0 27 Aug 2024
Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmonyIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024 Chao Xu Mingze Sun Zhi-Qi Cheng Haiwei Yang Yang Liu Baigui Sun Ruqi Huang Alexander G. Hauptmann VGen 356 5 0 18 Aug 2024
DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face AnimationAAAI Conference on Artificial Intelligence (AAAI), 2024 Jisoo Kim Jungbin Cho Joonho Park Soonmin Hwang Da Eun Kim Geon Kim Youngjae Yu 442 5 0 12 Aug 2024
MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture GenerationACM Multimedia (MM), 2024 Xiaofeng Mao Zhengkai Jiang Qilin Wang Chencan Fu Jiangning Zhang Jiafu Wu Yabiao Wang Chengjie Wang Wei Li Mingmin Chi 324 9 0 06 Aug 2024
DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework Fan Zhang Naye Ji Fuxing Gao Bozuo Zhao Jingmei Wu ... Zhenqing Ye Jiayang Zhu WeiFan Zhong Leyao Yan Xiaomeng Ma 176 1 0 01 Aug 2024
Modeling and Driving Human Body Soundfields through Acoustic Primitives Chao Huang Dejan Marković Chenliang Xu Alexander Richard 278 12 0 18 Jul 2024
A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights Wentao Lei Jinting Wang Fengji Ma Guanjie Huang Li Liu VGen EGVM 281 16 0 11 Jul 2024
Labeling Sentences with Symbolic and Deictic Gestures via Semantic Similarity Ariel Gjaci Carmine Tommaso Recchiuto A. Sgorbissa 197 1 0 02 Jul 2024
Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective Face and Body Expressions from Affordable Inputs Uttaran Bhattacharya Aniket Bera Dinesh Manocha CVBM 260 4 0 26 Jun 2024
Investigating the impact of 2D gesture representation on co-speech gesture generation Teo Guichoux Laure Soulier Nicolas Obin Catherine Pelachaud SLR 254 0 0 21 Jun 2024
PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance Qijun Gan Song Wang Shengtao Wu Jianke Zhu 502 2 0 13 Jun 2024
CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild Xingqun Qi Hengyuan Zhang Yatian Wang J. Pan Chen Liu ... Qixun Zhang Shanghang Zhang Wenhan Luo Qifeng Liu Qi-fei Liu DiffM SLR 430 8 0 27 May 2024
SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models Qingrong Cheng Xu Li Xinghui Fu DiffM 201 13 0 22 May 2024
Semantic Gesticulator: Semantics-Aware Co-Speech Gesture SynthesisACM Transactions on Graphics (TOG), 2024 Zeyi Zhang Tenglong Ao Yuyao Zhang Qingzhe Gao Chuan Lin Baoquan Chen Libin Liu SLR 238 30 0 16 May 2024
LLAniMAtion: LLAMA Driven Gesture Animation John T. Windle Iain Matthews Sarah Taylor 231 1 0 13 May 2024
Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of MetricsComputer Vision and Image Understanding (CVIU), 2024 Ali Ismail-Fawaz Maxime Devanne Stefano Berretti Jonathan Weber Germain Forestier EGVM 160 7 0 13 May 2024
Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis Shivam Mehta Anna Deichler Jim O'Regan Birger Moëll Jonas Beskow G. Henter Simon Alexanderson 228 7 0 30 Apr 2024
Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model Wen-Ling Lei Li Liu Jun Wang DiffM 235 4 0 30 Apr 2024
MCM: Multi-condition Motion Synthesis Framework Zeyu Ling Bo Han Yongkang Wang Han Lin Mohan Kankanhalli Weidong Geng 145 0 0 19 Apr 2024
A Unified Editing Method for Co-Speech Gesture Generation via Diffusion InversionACM Multimedia Asia (MMAsia), 2024 Zeyu Zhao Nan Gao Zhi Zeng Guixuan Zhang Jie Liu Shuwu Zhang DiffM 286 1 0 03 Apr 2024
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion ModelComputer Vision and Pattern Recognition (CVPR), 2024 Xu He Qiaochu Huang Zhensong Zhang Zhiwei Lin Zhiyong Wu Sicheng Yang Minglei Li Zhiyi Chen Songcen Xu Xiaofei Wu 187 28 0 02 Apr 2024
Large Motion Model for Unified Multi-Modal Motion Generation Mingyuan Zhang Daisheng Jin Chenyang Gu Fangzhou Hong Zhongang Cai ... Chongzhi Zhang Xinying Guo Lei Yang Ying He Ziwei Liu VGen 259 58 0 01 Apr 2024
Towards Variable and Coordinated Holistic Co-Speech Motion Generation Yifei Liu Qiong Cao Yandong Wen Huaiguang Jiang Changxing Ding SLR 264 30 0 30 Mar 2024
Beyond Talking -- Generating Holistic 3D Human Dyadic Motion for Communication Mingze Sun Chao Xu Xinyu Jiang Yang Liu Baigui Sun Ruqi Huang 201 12 0 28 Mar 2024
ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis Muhammad Hamza Mughal Rishabh Dabral I. Habibie Lucia Donatelli Marc Habermann Christian Theobalt SLR 135 32 0 26 Mar 2024
Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference Fan Zhang Zhaohan Wang Xin Lyu Siyuan Zhao Mengjian Li ... Naye Ji Hui Du Fuxing Gao Hao Wu Shunman Li VGen 251 8 0 16 Mar 2024
MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space ModelsNeural Information Processing Systems (NeurIPS), 2024 Zunnan Xu Yukang Lin Haonan Han Sicheng Yang Ronghui Li Yachao Zhang Xiu Li Mamba 564 40 0 14 Mar 2024
Text-to-Image Cross-Modal Generation: A Systematic Review Maciej Żelaszczyk Jacek Mańdziuk 309 6 0 21 Jan 2024
Cascaded Cross-Modal Transformer for Audio-Textual ClassificationArtificial Intelligence Review (Artif Intell Rev), 2024 Nicolae-Cătălin Ristea Andrei Anghel Radu Tudor Ionescu 238 2 0 15 Jan 2024
DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture GenerationComputer Vision and Pattern Recognition (CVPR), 2024 Junming Chen Yunfei Liu Jianan Wang Ailing Zeng Yu Li Qifeng Chen VGen 259 60 0 09 Jan 2024
Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness Sicheng Yang Zunnan Xu Haiwei Xue Yongkang Cheng Shaoli Huang Biwei Huang Zhiyong Wu DiffM VGen 210 19 0 07 Jan 2024
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture ModelingComputer Vision and Pattern Recognition (CVPR), 2023 Haiyang Liu Zihao Zhu Giorgio Becherini Yichen Peng Mingyang Su You Zhou Xuefei Zhe Naoya Iwamoto Bo Zheng Michael J. Black SLR 860 74 0 31 Dec 2023
Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control Zunnan Xu Yachao Zhang Sicheng Yang Ronghui Li Xiu Li SLR 230 18 0 26 Dec 2023
SAiD: Speech-driven Blendshape Facial Animation with Diffusion Inkyu Park Jaewoong Cho 276 10 0 25 Dec 2023
Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models Haiwei Xue Sicheng Yang Zhensong Zhang Zhiyong Wu Minglei Li Zonghong Dai Helen M. Meng DiffM 261 1 0 25 Dec 2023
Realistic Human Motion Generation with Cross-Diffusion Models Zeping Ren Shaoli Huang Xiu Li VGen 244 12 0 18 Dec 2023
Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion Kiran Chhatre Radek Danvevcek Nikos Athanasiou Giorgio Becherini Christopher Peters Michael J. Black Timo Bolkart DiffM 477 41 0 07 Dec 2023
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture GenerationComputer Vision and Pattern Recognition (CVPR), 2023 Xingqun Qi Jiahao Pan Peng Li Ruibin Yuan Yatian Wang ... Wenhan Luo Wei Xue Shanghang Zhang Qi-fei Liu Yi-Ting Guo SLR 252 19 0 29 Nov 2023
SpeechAct: Towards Generating Whole-body Motion from SpeechIEEE Transactions on Visualization and Computer Graphics (TVCG), 2023 Jinsong Zhang Minjie Zhu Yuxiang Zhang Yebin Liu Kun Li 269 3 0 29 Nov 2023
Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers Staphord Bengesi Hoda El-Sayed Md Kamruzzaman Sarker Yao Houkpati John Irungu T. Oladunni 318 167 0 17 Nov 2023
META4: Semantically-Aligned Generation of Metaphoric Gestures Using Self-Supervised Text and Speech Representation Mireille Fares Catherine Pelachaud Nicolas Obin 148 1 0 09 Nov 2023
Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and AudioNeural Information Processing Systems (NeurIPS), 2023 Xudong Xu Dejan Marković Jacob Sandakly Todd Keebler Steven Krenn Alexander Richard 125 8 0 01 Nov 2023
State of the Art on Diffusion Models for Visual Computing Ryan Po Wang Yifan Vladislav Golyanik Kfir Aberman Jonathan T. Barron ... Matthias Nießner Bjorn Ommer Christian Theobalt Peter Wonka Gordon Wetzstein 253 152 0 11 Oct 2023
Large language models in textual analysis for gesture selectionInternational Conference on Multimodal Interaction (ICMI), 2023 Laura Birka Hensel Nutchanon Yongsatianchot P. Torshizi E. Minucci Stacy Marsella SLR 199 12 0 04 Oct 2023