Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes

Interactive 3D segmentation has emerged as a promising solution for generating accurate object masks in complex 3D scenes by incorporating user-provided clicks. However, two critical challenges remain underexplored: (1) effectively generalizing from sparse user clicks to produce accurate segmentation, and (2) quantifying predictive uncertainty to help users identify unreliable regions. In this work, we propose NPISeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges. Specifically, NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization by capturing both global context and object-specific characteristics. Additionally, we design a probabilistic prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model's ability to capture object-aware context and quantify predictive uncertainty. Experiments on four 3D point cloud datasets demonstrate that NPISeg3D achieves superior segmentation performance with fewer clicks while providing reliable uncertainty estimations.

@article{liu2025_2505.01726,
  title={ Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes },
  author={ Jie Liu and Pan Zhou and Zehao Xiao and Jiayi Shen and Wenzhe Yin and Jan-Jakob Sonke and Efstratios Gavves },
  journal={arXiv preprint arXiv:2505.01726},
  year={ 2025 }
}