Multimodal Transformer for Unaligned Multimodal Language Sequences

1 June 2019

Yao-Hung Hubert Tsai

Shaojie Bai

Paul Pu Liang

J. Zico Kolter

Louis-Philippe Morency

Ruslan Salakhutdinov

ArXiv PDF HTML

Papers citing "Multimodal Transformer for Unaligned Multimodal Language Sequences"

50 / 66 papers shown

Title
Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection SungHeon Jeong Jihong Park Mohsen Imani 43 0 0 05 May 2025
PREMISE: Matching-based Prediction for Accurate Review Recommendation Wei Han Hui Chen Soujanya Poria 24 0 0 02 May 2025
Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs Yijie Jin Junjie Peng Xuanchao Lin Haochen Yuan Lan Wang Cangzhi Zheng 30 0 0 02 May 2025
FROG: Effective Friend Recommendation in Online Games via Modality-aware User Preferences Qiwei Wang Dandan Lin Wenqing Lin Ziming Wu OffRL 22 0 0 13 Apr 2025
Heterogeneous bimodal attention fusion for speech emotion recognition Jiachen Luo Huy Phan Lin Wang Joshua Reiss 37 0 0 09 Mar 2025
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention Joe Dhanith Shravan Venkatraman Modigari Narendra Vigya Sharma Santhosh Malarvannan 65 0 0 20 Feb 2025
Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues David Sasu Zehui Wu Ziwei Gong Run Chen Pengyuan Shi Lin Ai Julia Hirschberg Natalie Schluter 48 1 0 16 Feb 2025
A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastoma A. Gomaa Yixing Huang Pluvio Stephan Katharina Breininger Benjamin Frey ... U. Gaipl Christoph Bert R. Fietkau M. Schmidt F. Putz 84 0 0 06 Feb 2025
Towards Explainable Multimodal Depression Recognition for Clinical Interviews Wenjie Zheng Qiming Xie Zengzhi Wang Jianfei Yu Rui Xia 57 0 0 28 Jan 2025
Are Transformers Truly Foundational for Robotics? James A. R. Marshall Andrew B. Barron AI4CE 65 0 0 25 Nov 2024
Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention Yuzhe Weng Haotian Wang Tian Gao Kewei Li Shutong Niu Jun Du 28 0 0 19 Oct 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 24 4 0 18 Oct 2024
Towards Robust Multimodal Sentiment Analysis with Incomplete Data Haoyu Zhang Wenbin Wang Tianshu Yu 27 1 0 30 Sep 2024
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition Peihao Xiang Chaohao Lin Kaida Wu Ou Bai 16 3 0 28 Apr 2024
TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis Ming Zhou Yunfei Feng Ziqi Zhou Kai Wang Tong Wang Dong-ming Yan 39 0 0 06 Apr 2024
iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer Fengtao Zhou Ying Xu Yanfen Cui Shenyang Zhang Yun Zhu ... Louis Ho Shing Lau Chu Han Dafu Zhang Zhenhui Li Hao Chen 24 1 0 01 Apr 2024
UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause Guimin Hu Zhihong Zhu Daniel Hershcovich Hasti Seifi Jiayuan Xie 21 6 0 30 Mar 2024
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition Yash Jain David M. Chan Pranav Dheram Aparna Khare Olabanji Shonibare Venkatesh Ravichandran Shalini Ghosh 14 2 0 28 Mar 2024
Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation Zhekai Du Xinyao Li Fengling Li Ke Lu Lei Zhu Jingjing Li 27 15 0 05 Mar 2024
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion Xing Han Huy Nguyen Carl Harris Nhat Ho S. Saria MoE 42 16 0 05 Feb 2024
Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models Weijiao Zhang Jindong Han Zhao Xu Hang Ni Hao Liu Hui Xiong Hui Xiong AI4CE 77 14 0 30 Jan 2024
Triple Disentangled Representation Learning for Multimodal Affective Analysis Ying Zhou Xuefeng Liang Han Chen Yin Zhao Xin Chen Lida Yu 38 3 0 29 Jan 2024
Automatically Detecting Confusion and Conflict During Collaborative Learning Using Linguistic, Prosodic, and Facial Cues Yingbo Ma Yukyeong Song Mehmet Celepkolu K. Boyer Eric Wiebe Collin F. Lynch Maya Israel 19 2 0 26 Jan 2024
SDIF-DA: A Shallow-to-Deep Interaction Framework with Data Augmentation for Multi-modal Intent Detection Shijue Huang Libo Qin Bingbing Wang Geng Tu Ruifeng Xu 8 4 0 31 Dec 2023
Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach Weide Liu Huijing Zhan Hao Chen Fengmao Lv 16 1 0 28 Dec 2023
Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition Chengxin Chen Pengyuan Zhang 24 5 0 26 Dec 2023
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 12 4 0 21 Nov 2023
Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data Muyu Wang Shiyu Fan Yichen Li Hui Chen MedIm 4 1 0 27 Sep 2023
RBA-GCN: Relational Bilevel Aggregation Graph Convolutional Network for Emotion Recognition Lin Yuan Guoheng Huang Fenghuan Li Xiaochen Yuan Chi-Man Pun Guo Zhong 9 10 0 18 Aug 2023
Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning Songning Lai Jiakang Li Guinan Guo Xifeng Hu Yulong Li ... Yutong Liu Zhaoxia Ren Chun Wan Danmin Miao Zhi Liu SSL 28 9 0 15 May 2023
A vector quantized masked autoencoder for audiovisual speech emotion recognition Samir Sadok Simon Leglaive Renaud Séguier SSL 47 6 0 05 May 2023
Local Contrastive Learning for Medical Image Recognition S. A. Rizvi Ruixiang Tang X. Jiang X. Ma X. Hu 13 5 0 24 Mar 2023
Transformadores: Fundamentos teoricos y Aplicaciones J. D. L. Torre 60 0 0 18 Feb 2023
EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis Feng Qiu Chengyang Xie Yu-qiong Ding Wanzeng Kong 8 1 0 16 Dec 2022
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition Guimin Hu Ting-En Lin Yi Zhao Guangming Lu Yuchuan Wu Yongbin Li 22 108 0 21 Nov 2022
Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis Peipei Liu Xin Zheng Hong Li Jie Liu Yimo Ren Hongsong Zhu Limin Sun AI4TS 6 3 0 28 Oct 2022
Mind's Eye: Grounded Language Model Reasoning through Simulation Ruibo Liu Jason W. Wei S. Gu Te-Yen Wu Soroush Vosoughi Claire Cui Denny Zhou Andrew M. Dai ReLM LRM 106 78 0 11 Oct 2022
TVLT: Textless Vision-Language Transformer Zineng Tang Jaemin Cho Yixin Nie Mohit Bansal VLM 31 28 0 28 Sep 2022
Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection Xiang Zhang Huiyuan Yang Taoyue Wang Xiaotian Li L. Yin 11 7 0 25 Sep 2022
Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model Yixiao Zhang Junyan Jiang Gus Xia S. Dixon 17 9 0 24 Aug 2022
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module Yih-Ling Liu Ziqi Yuan Huisheng Mao Zhiyun Liang Wanqiuyue Yang Yuanzhe Qiu Tie Cheng Xiaoteng Li Hua Xu Kai Gao 18 44 0 22 Aug 2022
Multi-Attention Network for Compressed Video Referring Object Segmentation Weidong Chen Dexiang Hong Yuankai Qi Zhenjun Han Shuhui Wang Laiyun Qing Qingming Huang Guorong Li VOS 16 35 0 26 Jul 2022
A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cues Jason Armitage L. Impett Rico Sennrich 6 5 0 24 Jul 2022
Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis Teng Sun Wenjie Wang Liqiang Jing Yiran Cui Xuemeng Song Liqiang Nie OODD 8 33 0 24 Jul 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 21 518 0 13 Jun 2022
COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition M. Tellamekala Shahin Amiriparian Björn W. Schuller Elisabeth André T. Giesbrecht M. Valstar 10 25 0 12 Jun 2022
Analyzing Modality Robustness in Multimodal Sentiment Analysis Devamanyu Hazarika Yingting Li Bo Cheng Shuai Zhao Roger Zimmermann Soujanya Poria 23 32 0 30 May 2022
TransTab: Learning Transferable Tabular Transformers Across Tables Zifeng Wang Jimeng Sun LMTD 12 135 0 19 May 2022
i-Code: An Integrative and Composable Multimodal Learning Framework Ziyi Yang Yuwei Fang Chenguang Zhu Reid Pryzant Dongdong Chen ... Bin Xiao Yuanxun Lu Takuya Yoshioka Michael Zeng Xuedong Huang 24 45 0 03 May 2022
Are Multimodal Transformers Robust to Missing Modality? Mengmeng Ma Jian Ren Long Zhao Davide Testuggine Xi Peng ViT 15 145 0 12 Apr 2022