From Monocular Vision to Autonomous Action: Guiding Tumor Resection via 3D Reconstruction

Surgical automation requires precise guidance and understanding of the scene. Current methods in the literature rely on bulky depth cameras to create maps of the anatomy, however this does not translate well to space-limited clinical applications. Monocular cameras are small and allow minimally invasive surgeries in tight spaces but additional processing is required to generate 3D scene understanding. We propose a 3D mapping pipeline that uses only RGB images to create segmented point clouds of the target anatomy. To ensure the most precise reconstruction, we compare different structure from motion algorithms' performance on mapping the central airway obstructions, and test the pipeline on a downstream task of tumor resection. In several metrics, including post-procedure tissue model evaluation, our pipeline performs comparably to RGB-D cameras and, in some cases, even surpasses their performance. These promising results demonstrate that automation guidance can be achieved in minimally invasive procedures with monocular cameras. This study is a step toward the complete autonomy of surgical robots.

@article{acar2025_2503.16263,
  title={ From Monocular Vision to Autonomous Action: Guiding Tumor Resection via 3D Reconstruction },
  author={ Ayberk Acar and Mariana Smith and Lidia Al-Zogbi and Tanner Watts and Fangjie Li and Hao Li and Nural Yilmaz and Paul Maria Scheikl and Jesse F. dÁlmeida and Susheela Sharma and Lauren Branscombe and Tayfun Efe Ertop and Robert J. Webster III and Ipek Oguz and Alan Kuntz and Axel Krieger and Jie Ying Wu },
  journal={arXiv preprint arXiv:2503.16263},
  year={ 2025 }
}