Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning

Guotai Wang, Wenqi Li, Maria A. Zuluaga, Rosalind Pratt, Premal A. Patel, Michael Aertsen, Tom Doel, Anna L. David, Jan Deprest, Sebastien Ourselin, Tom Vercauteren

Research output: Contribution to journalArticlepeer-review

610 Citations (Scopus)
601 Downloads (Pure)


Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes (a.k.a. zero-shot learning). To address these problems, we propose a novel deep learning-based interactive segmentation framework by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine-tuning. We applied this framework to two applications: 2D segmentation of multiple organs from fetal Magnetic Resonance (MR) slices, where only two types of these organs were annotated for training; and 3D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only the tumor core in one MR sequence was annotated for training. Experimental results show that 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) imagespecific fine-tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

Original languageEnglish
Pages (from-to)1562 - 1573
Number of pages12
JournalIEEE Transactions on Medical Imaging
Issue number7
Early online date26 Jan 2018
Publication statusPublished - 30 Jun 2018


  • Adaptation models
  • Biomedical imaging
  • brain tumor
  • convolutional neural network
  • fetal MRI
  • fine-tuning
  • Image segmentation
  • Interactive image segmentation
  • Testing
  • Three-dimensional displays
  • Training
  • Two dimensional displays


Dive into the research topics of 'Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning'. Together they form a unique fingerprint.

Cite this