See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

7 December 2022

Li Fei-Fei

Jiajun Wu

Papers citing "See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation"

44 / 44 papers shown

Title
GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion Shulong Jiang Shiqi Zhao Yuxuan Fan Peng Yin 24 0 0 12 May 2025
M2R2: MulitModal Robotic Representation for Temporal Action Segmentation Daniel Sliwowski Dongheui Lee 22 1 0 25 Apr 2025
Modality Selection and Skill Segmentation via Cross-Modality Attention Jiawei Jiang Kei Ota Devesh K. Jha Asako Kanezaki 22 0 0 20 Apr 2025
TacCap: A Wearable FBG-Based Tactile Sensor for Seamless Human-to-Robot Skill Transfer Chengyi Xing Hao Li Yi-Lin Wei Tian-Ao Ren Tianyu Tu Yuhao Lin Elizabeth Schumann Wei-Shi Zheng M. Cutkosky 64 0 0 03 Mar 2025
AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors Ruoxuan Feng Jiangyu Hu Wenke Xia Tianci Gao Ao Shen Yuhao Sun Bin Fang Di Hu 42 3 0 15 Feb 2025
Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor Trevor Ablett Oliver Limoyo Adam Sigal Affan Jilani Jonathan Kelly Kaleem Siddiqi F. Hogan Gregory Dudek 43 4 0 28 Jan 2025
Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins Venkatesh Pattabiraman Yifeng Cao Siddhant Haldar Lerrel Pinto Raunaq M. Bhirangi 32 4 0 22 Oct 2024
Visual-auditory Extrinsic Contact Estimation Xili Yi Jayjun Lee Nima Fazeli 15 1 0 22 Sep 2024
DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks Xutong Jin Chenxi Xu Ruohan Gao Jiajun Wu Guoping Wang Sheng Li DiffM 33 2 0 20 Sep 2024
Learning to Singulate Objects in Packed Environments using a Dexterous Hand Hao Jiang Yuhai Wang Hanyang Zhou Daniel Seita 33 2 0 01 Sep 2024
Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation Runze Yuan Tao Liu Wenke Ma Xuelong Li 32 7 0 02 Aug 2024
RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing Bo Ai Stephen Tian Haochen Shi Yixuan Wang Cheston Tan Yunzhu Li Jiajun Wu 44 3 0 01 Jul 2024
ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data Zeyi Liu Cheng Chi Eric A. Cousineau Naveen Kuppuswamy Benjamin Burchfiel Shuran Song VGen 36 23 0 27 Jun 2024
SonicSense: Object Perception from In-Hand Acoustic Vibration Jiaxun Liu Boyuan Chen 29 2 0 25 Jun 2024
Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks Jialiang Zhao Yuxiang Ma Lirui Wang Edward H. Adelson 19 15 0 19 Jun 2024
When Vision Meets Touch: A Contemporary Review for Visuotactile Sensors from the Signal Processing Perspective Shoujie Li Zihan Wang Changsheng Wu Xiang Li Shan Luo Bin Fang Fuchun Sun Xiao-Ping Zhang Wenbo Ding AI4TS 24 11 0 18 Jun 2024
Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation Jared Mejia Victoria Dean Tess Hellebrekers Abhinav Gupta 35 12 0 14 May 2024
RoboCAP: Robotic Classification and Precision Pouring of Diverse Liquids and Granular Media with Capacitive Sensing Yexin Hu Alexandra Gillespie Akhil Padmanabha Kavya Puthuveetil Wesley Lewis Karan Khokar Zackory M. Erickson 16 0 0 13 May 2024
Multi-modal perception for soft robotic interactions using generative models Enrico Donato Egidio Falotico T. G. Thuruthel 26 2 0 05 Apr 2024
A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches Zhigen Zhao Shuo Cheng Yan Ding Ziyi Zhou Shiqi Zhang Danfei Xu Ye Zhao 38 22 0 03 Apr 2024
Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning Entong Su Chengzhe Jia Yuzhe Qin Wenxuan Zhou Annabella Macaluso Binghao Huang Xiaolong Wang 28 4 0 18 Mar 2024
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations Fengyu Yang Chao Feng Ziyang Chen Hyoungseob Park Daniel Wang ... Ziyao Zeng Xien Chen Rit Gangopadhyay Andrew Owens Alex Wong 36 53 0 31 Jan 2024
M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation Fotios Lygerakis Vedant Dave Elmar Rueckert SSL 27 2 0 30 Jan 2024
Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training Vedant Dave Fotios Lygerakis Elmar Rueckert SSL 22 16 0 22 Jan 2024
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World Yining Hong Zishuo Zheng Peihao Chen Yian Wang Junyan Li Chuang Gan 13 32 0 16 Jan 2024
Robot Learning in the Era of Foundation Models: A Survey Xuan Xiao Jiahang Liu Zhipeng Wang Yanmin Zhou Yong Qi Qian Cheng Bin He Shuo Jiang AI4CE LM&Ro 16 26 0 24 Nov 2023
Force-Constrained Visual Policy: Safe Robot-Assisted Dressing via Multi-Modal Sensing Zhanyi Sun Yufei Wang David Held Zackory M. Erickson 19 4 0 07 Nov 2023
The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning Carmelo Sferrazza Younggyo Seo Hao Liu Youngwoon Lee Pieter Abbeel 43 14 0 02 Nov 2023
MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation Kelin Yu Yunhai Han Matthew Zhu Vaibhav Saxena Danfei Xu Ye Zhao 16 11 0 25 Oct 2023
MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception Gyan Tatiya Jonathan M Francis Ho-Hsiang Wu Yonatan Bisk Jivko Sinapov 29 1 0 15 Sep 2023
WSTac: Interactive Surface Perception based on Whisker-Inspired and Self-Illuminated Vision-Based Tactile Sensor K. Lei Kitajima Sou Wang Sing Chan Jiayi Yan Siqi Ping D. Peng Wenbo Ding Xiao-Ping Zhang 11 4 0 25 Aug 2023
Multi-Stage Cable Routing through Hierarchical Imitation Learning Jianlan Luo Charles Xu Xinyang Geng Gilbert Feng Kuan Fang L. Tan S. Schaal Sergey Levine 31 52 0 18 Jul 2023
Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations Xiaofeng Mao Yucheng Xu Ruoshi Wen M. Kasaei Wanming Yu Efi Psomopoulou Nathan Lepora Zhibin Li SSL 19 0 0 10 Jul 2023
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning Paul Pu Liang Yiwei Lyu Xiang Fan Arav Agarwal Yun Cheng Louis-Philippe Morency Ruslan Salakhutdinov VLM 27 6 0 28 Jun 2023
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects Ruohan Gao Yiming Dou Hao Li Tanmay Agarwal Jeannette Bohg Yunzhu Li Li Fei-Fei Jiajun Wu 11 29 0 01 Jun 2023
LLM as A Robotic Brain: Unifying Egocentric Memory and Control Jinjie Mai Jun Chen Bing Li Guocheng Qian Mohamed Elhoseiny Bernard Ghanem LM&Ro 12 33 0 19 Apr 2023
Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning Maximilian Du Olivia Y. Lee Suraj Nair Chelsea Finn OffRL 47 32 0 30 May 2022
ShapeMap 3-D: Efficient shape mapping through dense touch and vision Sudharshan Suresh Zilin Si Joshua G. Mangelson Wenzhen Yuan Michael Kaess 69 56 0 20 Sep 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Yin Cui Boqing Gong ViT 240 577 0 22 Apr 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 253 4,764 0 24 Feb 2021
Playing with Food: Learning Food Item Representations through Interactive Exploration A. Sawhney Steven Lee Kevin Zhang Manuela Veloso Oliver Kroemer 29 23 0 06 Jan 2021
Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering Maria Bauzá Eric Valls Bryan Lim Theo Sechopoulos Alberto Rodriguez 82 74 0 09 Dec 2020
Deep Dynamics Models for Learning Dexterous Manipulation Anusha Nagabandi K. Konolige Sergey Levine Vikash Kumar 143 407 0 25 Sep 2019
Tactile-Based Insertion for Dense Box-Packing Siyuan Dong Alberto Rodriguez 95 52 0 12 Sep 2019