TISS-net: Brain tumor image synthesis and segmentation using cascaded dual-task networks and error-prediction consistency

Jianghao Wu, Dong Guo, Lu Wang, Shuojue Yang, Yuanjie Zheng, Jonathan Shapey, Tom Vercauteren, Sotirios Bisdas, Robert Bradford, Shakeel Saeed, Neil Kitchen, Sebastien Ourselin, Shaoting Zhang, Guotai Wang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)


Accurate segmentation of brain tumors from medical images is important for diagnosis and treatment planning, and it often requires multi-modal or contrast-enhanced images. However, in practice some modalities of a patient may be absent. Synthesizing the missing modality has a potential for filling this gap and achieving high segmentation performance. Existing methods often treat the synthesis and segmentation tasks separately or consider them jointly but without effective regularization of the complex joint model, leading to limited performance. We propose a novel brain Tumor Image Synthesis and Segmentation network (TISS-Net) that obtains the synthesized target modality and segmentation of brain tumors end-to-end with high performance. First, we propose a dual-task-regularized generator that simultaneously obtains a synthesized target modality and a coarse segmentation, which leverages a tumor-aware synthesis loss with perceptibility regularization to minimize the high-level semantic domain gap between synthesized and real target modalities. Based on the synthesized image and the coarse segmentation, we further propose a dual-task segmentor that predicts a refined segmentation and error in the coarse segmentation simultaneously, where a consistency between these two predictions is introduced for regularization. Our TISS-Net was validated with two applications: synthesizing FLAIR images for whole glioma segmentation, and synthesizing contrast-enhanced T1 images for Vestibular Schwannoma segmentation. Experimental results showed that our TISS-Net largely improved the segmentation accuracy compared with direct segmentation from the available modalities, and it outperformed state-of-the-art image synthesis-based segmentation methods.

Original languageEnglish
Article number126295
Publication statusPublished - 1 Aug 2023


  • Brain tumor
  • Deep learning
  • Image synthesis
  • Regularization
  • Segmentation


Dive into the research topics of 'TISS-net: Brain tumor image synthesis and segmentation using cascaded dual-task networks and error-prediction consistency'. Together they form a unique fingerprint.

Cite this