Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints

Foundation models trained on internet-scale data, such as Vision-Language Models (VLMs), excel at performing a wide variety of common sense tasks like visual question answering. Despite their impressive capabilities, these models cannot currently be directly applied to challenging robot manipulation problems that require complex and precise continuous reasoning over long horizons. Task and Motion Planning (TAMP) systems can control high-dimensional continuous systems over long horizons via a hybrid search over traditional primitive robot skills. However, these systems require detailed models of how the robot can impact its environment, preventing them from directly interpreting and addressing novel human objectives, for example, an arbitrary natural language goal. We propose deploying VLMs within TAMP systems by having them generate discrete and continuous language-parameterized constraints that enable TAMP to reason about open-world concepts. Specifically, we propose algorithms for VLM partial planning that constrain a TAMP system's discrete temporal search and VLM continuous constraints interpretation to augment the traditional manipulation constraints that TAMP systems seek to satisfy. Experiments demonstrate that our approach -- OWL-TAMP -- outperforms several related baselines, including those that solely use TAMP or VLMs for planning, across several long-horizon manipulation tasks specified directly through natural language. We additionally demonstrate that our approach is compatible with a variety of TAMP systems and can be deployed to solve challenging manipulation tasks on real-world hardware.

@article{kumar2025_2411.08253,
  title={ Open-World Task and Motion Planning via Vision-Language Model Inferred Constraints },
  author={ Nishanth Kumar and William Shen and Fabio Ramos and Dieter Fox and Tomás Lozano-Pérez and Leslie Pack Kaelbling and Caelan Reed Garrett },
  journal={arXiv preprint arXiv:2411.08253},
  year={ 2025 }
}