From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review

Francesco Galati; Sébastien Ourselin; Maria A. Zuluaga

doi:10.3390/app12083936

From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review

Francesco Galati^*, Sébastien Ourselin, Maria A. Zuluaga

^*Corresponding author for this work

Surgical & Interventional Engineering

Research output: Contribution to journal › Article › peer-review

3 Citations (Scopus)

Abstract

Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.

Original language	English
Article number	3936
Journal	Applied Sciences (Switzerland)
Volume	12
Issue number	8
DOIs	https://doi.org/10.3390/app12083936
Publication status	Published - 1 Apr 2022

Keywords

cardiac image segmentation
cardiac magnetic resonance imaging
deep learning
reliability and robustness

Access to Document

10.3390/app12083936

Cite this

@article{ba6992cec8984168a9d095e7476e7c67,

title = "From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review",

abstract = "Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.",

keywords = "cardiac image segmentation, cardiac magnetic resonance imaging, deep learning, reliability and robustness",

author = "Francesco Galati and S{\'e}bastien Ourselin and Zuluaga, {Maria A.}",

note = "Funding Information: Funding: This work is supported by the French government, through the 3IA C{\^o}te d{\textquoteright}Azur Investments in the future project managed by the National Research Agency (ANR) (ANR-19-P3IA-0002). Publisher Copyright: {\textcopyright} 2022 by the authorsLicensee MDPI, Basel, Switzerland.",

year = "2022",

month = apr,

day = "1",

doi = "10.3390/app12083936",

language = "English",

volume = "12",

journal = "Applied Sciences (Switzerland)",

issn = "2076-3417",

publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",

number = "8",

}

TY - JOUR

T1 - From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation

T2 - A Review

AU - Galati, Francesco

AU - Ourselin, Sébastien

AU - Zuluaga, Maria A.

N1 - Funding Information: Funding: This work is supported by the French government, through the 3IA Côte d’Azur Investments in the future project managed by the National Research Agency (ANR) (ANR-19-P3IA-0002). Publisher Copyright: © 2022 by the authorsLicensee MDPI, Basel, Switzerland.

PY - 2022/4/1

Y1 - 2022/4/1

N2 - Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.

AB - Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.

KW - cardiac image segmentation

KW - cardiac magnetic resonance imaging

KW - deep learning

KW - reliability and robustness

UR - http://www.scopus.com/inward/record.url?scp=85128842183&partnerID=8YFLogxK

U2 - 10.3390/app12083936

DO - 10.3390/app12083936

M3 - Article

AN - SCOPUS:85128842183

SN - 2076-3417

VL - 12

JO - Applied Sciences (Switzerland)

JF - Applied Sciences (Switzerland)

IS - 8

M1 - 3936

ER -

From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this