Learning Dexterous In-Hand Manipulation with Multifingered Hands via Visuomotor Diffusion

We present a framework for learning dexterous in-hand manipulation with multifingered hands using visuomotor diffusion policies. Our system enables complex in-hand manipulation tasks, such as unscrewing a bottle lid with one hand, by leveraging a fast and responsive teleoperation setup for the four-fingered Allegro Hand. We collect high-quality expert demonstrations using an augmented reality (AR) interface that tracks hand movements and applies inverse kinematics and motion retargeting for precise control. The AR headset provides real-time visualization, while gesture controls streamline teleoperation. To enhance policy learning, we introduce a novel demonstration outlier removal approach based on HDBSCAN clustering and the Global-Local Outlier Score from Hierarchies (GLOSH) algorithm, effectively filtering out low-quality demonstrations that could degrade performance. We evaluate our approach extensively in real-world settings and provide all experimental videos on the project website:this https URL

@article{koczy2025_2503.02587,
  title={ Learning Dexterous In-Hand Manipulation with Multifingered Hands via Visuomotor Diffusion },
  author={ Piotr Koczy and Michael C. Welle and Danica Kragic },
  journal={arXiv preprint arXiv:2503.02587},
  year={ 2025 }
}