Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities?

TY - CHAP

T1 - Can Non-specialists Provide High Quality Gold Standard Labels in Challenging Modalities?

AU - Budd, Samuel

AU - Day, Thomas

AU - Simpson, John

AU - Lloyd, Karen

AU - Matthew, Jacqueline

AU - Skelton, Emily

AU - Razavi, Reza

AU - Kainz, Bernhard

N1 - Publisher Copyright: © 2021, Springer Nature Switzerland AG. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021

Y1 - 2021

N2 - Probably yes.—Supervised Deep Learning dominates performance scores for many computer vision tasks and defines the state-of-the-art. However, medical image analysis lags behind natural image applications. One of the many reasons is the lack of well annotated medical image data available to researchers. One of the first things researchers are told is that we require significant expertise to reliably and accurately interpret and label such data. We see significant inter- and intra-observer variability between expert annotations of medical images. Still, it is a widely held assumption that novice annotators are unable to provide useful annotations for use by clinical Deep Learning models. In this work we challenge this assumption and examine the implications of using a minimally trained novice labelling workforce to acquire annotations for a complex medical image dataset. We study the time and cost implications of using novice annotators, the raw performance of novice annotators compared to gold-standard expert annotators, and the downstream effects on a trained Deep Learning segmentation model’s performance for detecting a specific congenital heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.

AB - Probably yes.—Supervised Deep Learning dominates performance scores for many computer vision tasks and defines the state-of-the-art. However, medical image analysis lags behind natural image applications. One of the many reasons is the lack of well annotated medical image data available to researchers. One of the first things researchers are told is that we require significant expertise to reliably and accurately interpret and label such data. We see significant inter- and intra-observer variability between expert annotations of medical images. Still, it is a widely held assumption that novice annotators are unable to provide useful annotations for use by clinical Deep Learning models. In this work we challenge this assumption and examine the implications of using a minimally trained novice labelling workforce to acquire annotations for a complex medical image dataset. We study the time and cost implications of using novice annotators, the raw performance of novice annotators compared to gold-standard expert annotators, and the downstream effects on a trained Deep Learning segmentation model’s performance for detecting a specific congenital heart disease (hypoplastic left heart syndrome) in fetal ultrasound imaging.

KW - Annotations

KW - Expert

KW - Labels

KW - Novice

UR - http://www.scopus.com/inward/record.url?scp=85116503777&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-87722-4_23

DO - 10.1007/978-3-030-87722-4_23

M3 - Conference paper

AN - SCOPUS:85116503777

SN - 9783030877217

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 251

EP - 262

BT - Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health - 3rd MICCAI Workshop, DART 2021, and 1st MICCAI Workshop, FAIR 2021, Held in Conjunction with MICCAI 2021, Proceedings

A2 - Albarqouni, Shadi

A2 - Cardoso, M. Jorge

A2 - Dou, Qi

A2 - Kamnitsas, Konstantinos

A2 - Khanal, Bishesh

A2 - Rekik, Islem

A2 - Rieke, Nicola

A2 - Sheet, Debdoot

A2 - Tsaftaris, Sotirios

A2 - Xu, Daguang

A2 - Xu, Ziyue

PB - Springer Science and Business Media Deutschland GmbH

T2 - 3rd MICCAI Workshop on Domain Adaptation and Representation Transfer, DART 2021, and the 1st MICCAI Workshop on Affordable Healthcare and AI for Resource Diverse Global Health, FAIR 2021, held in conjunction with 24th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2021

Y2 - 27 September 2021 through 1 October 2021

ER -