Multimodal Transformer for Unaligned Multimodal Language Sequences

1 June 2019

Yao-Hung Hubert Tsai

Shaojie Bai

Paul Pu Liang

J. Zico Kolter

Louis-Philippe Morency

Ruslan Salakhutdinov

ArXiv PDF HTML

Papers citing "Multimodal Transformer for Unaligned Multimodal Language Sequences"

44 / 44 papers shown

Title
Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection SungHeon Jeong Jihong Park Mohsen Imani 43 0 0 05 May 2025
Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs Yijie Jin Junjie Peng Xuanchao Lin Haochen Yuan Lan Wang Cangzhi Zheng 25 0 0 02 May 2025
PREMISE: Matching-based Prediction for Accurate Review Recommendation Wei Han Hui Chen Soujanya Poria 21 0 0 02 May 2025
FROG: Effective Friend Recommendation in Online Games via Modality-aware User Preferences Qiwei Wang Dandan Lin Wenqing Lin Ziming Wu OffRL 20 0 0 13 Apr 2025
Heterogeneous bimodal attention fusion for speech emotion recognition Jiachen Luo Huy Phan Lin Wang Joshua Reiss 35 0 0 09 Mar 2025
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention Joe Dhanith Shravan Venkatraman Modigari Narendra Vigya Sharma Santhosh Malarvannan 62 0 0 20 Feb 2025
Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues David Sasu Zehui Wu Ziwei Gong Run Chen Pengyuan Shi Lin Ai Julia Hirschberg Natalie Schluter 43 1 0 16 Feb 2025
A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastoma A. Gomaa Yixing Huang Pluvio Stephan Katharina Breininger Benjamin Frey ... U. Gaipl Christoph Bert R. Fietkau M. Schmidt F. Putz 84 0 0 06 Feb 2025
Towards Explainable Multimodal Depression Recognition for Clinical Interviews Wenjie Zheng Qiming Xie Zengzhi Wang Jianfei Yu Rui Xia 57 0 0 28 Jan 2025
Are Transformers Truly Foundational for Robotics? James A. R. Marshall Andrew B. Barron AI4CE 63 0 0 25 Nov 2024
Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention Yuzhe Weng Haotian Wang Tian Gao Kewei Li Shutong Niu Jun Du 23 0 0 19 Oct 2024
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition Peihao Xiang Chaohao Lin Kaida Wu Ou Bai 14 3 0 28 Apr 2024
TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis Ming Zhou Yunfei Feng Ziqi Zhou Kai Wang Tong Wang Dong-ming Yan 33 0 0 06 Apr 2024
iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer Fengtao Zhou Ying Xu Yanfen Cui Shenyang Zhang Yun Zhu ... Louis Ho Shing Lau Chu Han Dafu Zhang Zhenhui Li Hao Chen 22 1 0 01 Apr 2024
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition Yash Jain David M. Chan Pranav Dheram Aparna Khare Olabanji Shonibare Venkatesh Ravichandran Shalini Ghosh 14 2 0 28 Mar 2024
Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation Zhekai Du Xinyao Li Fengling Li Ke Lu Lei Zhu Jingjing Li 25 15 0 05 Mar 2024
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion Xing Han Huy Nguyen Carl Harris Nhat Ho S. Saria MoE 39 16 0 05 Feb 2024
Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models Weijiao Zhang Jindong Han Zhao Xu Hang Ni Hao Liu Hui Xiong Hui Xiong AI4CE 77 14 0 30 Jan 2024
Triple Disentangled Representation Learning for Multimodal Affective Analysis Ying Zhou Xuefeng Liang Han Chen Yin Zhao Xin Chen Lida Yu 38 3 0 29 Jan 2024
Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach Weide Liu Huijing Zhan Hao Chen Fengmao Lv 16 1 0 28 Dec 2023
Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition Chengxin Chen Pengyuan Zhang 24 5 0 26 Dec 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 12 4 0 21 Nov 2023
RBA-GCN: Relational Bilevel Aggregation Graph Convolutional Network for Emotion Recognition Lin Yuan Guoheng Huang Fenghuan Li Xiaochen Yuan Chi-Man Pun Guo Zhong 9 10 0 18 Aug 2023
Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning Songning Lai Jiakang Li Guinan Guo Xifeng Hu Yulong Li ... Yutong Liu Zhaoxia Ren Chun Wan Danmin Miao Zhi Liu SSL 28 9 0 15 May 2023
A vector quantized masked autoencoder for audiovisual speech emotion recognition Samir Sadok Simon Leglaive Renaud Séguier SSL 47 6 0 05 May 2023
Local Contrastive Learning for Medical Image Recognition S. A. Rizvi Ruixiang Tang X. Jiang X. Ma X. Hu 11 4 0 24 Mar 2023
Transformadores: Fundamentos teoricos y Aplicaciones J. D. L. Torre 60 0 0 18 Feb 2023
Mind's Eye: Grounded Language Model Reasoning through Simulation Ruibo Liu Jason W. Wei S. Gu Te-Yen Wu Soroush Vosoughi Claire Cui Denny Zhou Andrew M. Dai ReLM LRM 106 78 0 11 Oct 2022
COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition M. Tellamekala Shahin Amiriparian Björn W. Schuller Elisabeth André T. Giesbrecht M. Valstar 8 25 0 12 Jun 2022
TransTab: Learning Transferable Tabular Transformers Across Tables Zifeng Wang Jimeng Sun LMTD 10 135 0 19 May 2022
i-Code: An Integrative and Composable Multimodal Learning Framework Ziyi Yang Yuwei Fang Chenguang Zhu Reid Pryzant Dongdong Chen ... Bin Xiao Yuanxun Lu Takuya Yoshioka Michael Zeng Xuedong Huang 22 45 0 03 May 2022
Are Multimodal Transformers Robust to Missing Modality? Mengmeng Ma Jian Ren Long Zhao Davide Testuggine Xi Peng ViT 15 145 0 12 Apr 2022
Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition Vincent Karas M. Tellamekala Adria Mallol-Ragolta M. Valstar Björn W. Schuller 9 13 0 24 Mar 2022
Detecting Dementia from Speech and Transcripts using Transformers Loukas Ilias D. Askounis J. Psarras 6 31 0 27 Oct 2021
MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition Jinming Zhao Ruichen Li Qin Jin Xinchao Wang Haizhou Li 11 25 0 27 Oct 2021
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation Tongkun Xu Weihua Chen Pichao Wang Fan Wang Hao Li R. L. Jin ViT 27 213 0 13 Sep 2021
TEASEL: A Transformer-Based Speech-Prefixed Language Model Mehdi Arjmand M. Dousti H. Moradi 22 18 0 12 Sep 2021
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization Tiezheng Yu Wenliang Dai Zihan Liu Pascale Fung 21 71 0 06 Sep 2021
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis Wei Han Hui Chen Soujanya Poria 19 307 0 01 Sep 2021
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis Wenmeng Yu Hua Xu Ziqi Yuan Jiele Wu SSL 42 430 0 09 Feb 2021
Contextualized Attention-based Knowledge Transfer for Spoken Conversational Question Answering Chenyu You Nuo Chen Yuexian Zou 6 35 0 21 Oct 2020
Self-Supervised MultiModal Versatile Networks Jean-Baptiste Alayrac Adrià Recasens R. Schneider Relja Arandjelović Jason Ramapuram J. Fauw Lucas Smaira Sander Dieleman Andrew Zisserman SSL 12 371 0 29 Jun 2020
Multiresolution and Multimodal Speech Recognition with Transformers Georgios Paraskevopoulos Srinivas Parthasarathy Aparna Khare Shiva Sundaram 10 29 0 29 Apr 2020
A Decomposable Attention Model for Natural Language Inference Ankur P. Parikh Oscar Täckström Dipanjan Das Jakob Uszkoreit 187 1,358 0 06 Jun 2016