A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations

Virginia Fernandez; Walter Hugo Lopez Pinaya; Pedro Borges; Mark S. Graham; Tom Vercauteren; M. Jorge Cardoso

doi:10.1007/978-3-031-53767-7_13

A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations

Virginia Fernandez^*, Walter Hugo Lopez Pinaya, Pedro Borges, Mark S. Graham, Tom Vercauteren, M. Jorge Cardoso

^*Corresponding author for this work

King's College London

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

1 Citation (Scopus)

Abstract

Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.

Original language	English
Title of host publication	Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings
Editors	Anirban Mukhopadhyay, Ilkay Oksuz, Sandy Engelhardt, Dajiang Zhu, Yixuan Yuan
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	132-142
Number of pages	11
ISBN (Print)	9783031537660
DOIs	https://doi.org/10.1007/978-3-031-53767-7_13
Publication status	Published - 2024
Event	3rd Workshop on Deep Generative Models for Medical Image Computing and Computer Assisted Intervention, DGM4MICCAI 2023 Held in Conjunction with 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023 - Vancouver, Canada Duration: 8 Oct 2023 → 12 Oct 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14533 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	3rd Workshop on Deep Generative Models for Medical Image Computing and Computer Assisted Intervention, DGM4MICCAI 2023 Held in Conjunction with 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023
Country/Territory	Canada
City	Vancouver
Period	8/10/2023 → 12/10/2023

Access to Document

10.1007/978-3-031-53767-7_13

Cite this

Fernandez, V., Pinaya, W. H. L., Borges, P., Graham, M. S., Vercauteren, T., & Cardoso, M. J. (2024). A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. In A. Mukhopadhyay, I. Oksuz, S. Engelhardt, D. Zhu, & Y. Yuan (Eds.), Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings (pp. 132-142). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14533 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-53767-7_13

Fernandez, Virginia ; Pinaya, Walter Hugo Lopez ; Borges, Pedro et al. / A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings. editor / Anirban Mukhopadhyay ; Ilkay Oksuz ; Sandy Engelhardt ; Dajiang Zhu ; Yixuan Yuan. Springer Science and Business Media Deutschland GmbH, 2024. pp. 132-142 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{b69f73dee3484ce0aee31041a8d8950c,

title = "A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations",

abstract = "Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.",

author = "Virginia Fernandez and Pinaya, {Walter Hugo Lopez} and Pedro Borges and Graham, {Mark S.} and Tom Vercauteren and Cardoso, {M. Jorge}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 3rd Workshop on Deep Generative Models for Medical Image Computing and Computer Assisted Intervention, DGM4MICCAI 2023 Held in Conjunction with 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023 ; Conference date: 08-10-2023 Through 12-10-2023",

year = "2024",

doi = "10.1007/978-3-031-53767-7_13",

language = "English",

isbn = "9783031537660",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "132--142",

editor = "Anirban Mukhopadhyay and Ilkay Oksuz and Sandy Engelhardt and Dajiang Zhu and Yixuan Yuan",

booktitle = "Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings",

address = "Germany",

}

Fernandez, V, Pinaya, WHL, Borges, P, Graham, MS, Vercauteren, T & Cardoso, MJ 2024, A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. in A Mukhopadhyay, I Oksuz, S Engelhardt, D Zhu & Y Yuan (eds), Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14533 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 132-142, 3rd Workshop on Deep Generative Models for Medical Image Computing and Computer Assisted Intervention, DGM4MICCAI 2023 Held in Conjunction with 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023, Vancouver, Canada, 8/10/2023. https://doi.org/10.1007/978-3-031-53767-7_13

A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. / Fernandez, Virginia; Pinaya, Walter Hugo Lopez; Borges, Pedro et al.
Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings. ed. / Anirban Mukhopadhyay; Ilkay Oksuz; Sandy Engelhardt; Dajiang Zhu; Yixuan Yuan. Springer Science and Business Media Deutschland GmbH, 2024. p. 132-142 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14533 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations

AU - Fernandez, Virginia

AU - Pinaya, Walter Hugo Lopez

AU - Borges, Pedro

AU - Graham, Mark S.

AU - Vercauteren, Tom

AU - Cardoso, M. Jorge

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.

AB - Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.

UR - http://www.scopus.com/inward/record.url?scp=85187713980&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-53767-7_13

DO - 10.1007/978-3-031-53767-7_13

M3 - Conference paper

AN - SCOPUS:85187713980

SN - 9783031537660

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 132

EP - 142

BT - Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings

A2 - Mukhopadhyay, Anirban

A2 - Oksuz, Ilkay

A2 - Engelhardt, Sandy

A2 - Zhu, Dajiang

A2 - Yuan, Yixuan

PB - Springer Science and Business Media Deutschland GmbH

T2 - 3rd Workshop on Deep Generative Models for Medical Image Computing and Computer Assisted Intervention, DGM4MICCAI 2023 Held in Conjunction with 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023

Y2 - 8 October 2023 through 12 October 2023

ER -

Fernandez V, Pinaya WHL, Borges P, Graham MS, Vercauteren T , Cardoso MJ. A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations. In Mukhopadhyay A, Oksuz I, Engelhardt S, Zhu D, Yuan Y, editors, Deep Generative Models - Third MICCAI Workshop, DGM4MICCAI 2023, Held in Conjunction with MICCAI 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 132-142. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-53767-7_13