Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning

Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM 2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available atthis https URL

@article{mosbach2025_2505.02232,
  title={ Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning },
  author={ Malte Mosbach and Sven Behnke },
  journal={arXiv preprint arXiv:2505.02232},
  year={ 2025 }
}