Text Promptable Surgical Instrument Segmentation with Vision-Language Models

Zijian Zhou, Oluwatosin Alabi, Meng Wei, Tom Vercauteren, Miaojing Shi*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

22 Downloads (Pure)


In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.
Original languageEnglish
Title of host publicationThirty-seventh Annual Conference on Neural Information Processing Systems (NeurIPS 2023)
Publication statusPublished - 10 Dec 2023
Event37th International Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, LA, New Orleans, United States
Duration: 10 Dec 202316 Dec 2023
Conference number: 37


Conference37th International Conference on Neural Information Processing Systems, NeurIPS 2023
Abbreviated titleNeurIPS 2023
Country/TerritoryUnited States
CityNew Orleans
Internet address


Dive into the research topics of 'Text Promptable Surgical Instrument Segmentation with Vision-Language Models'. Together they form a unique fingerprint.

Cite this