Let’s agree to disagree: Learning highly debatable multirater labelling

Carole H. Sudre; Beatriz Gomez Anson; Silvia Ingala; Chris D. Lane; Daniel Jimenez; Lukas Haider; Thomas Varsavsky; Ryutaro Tanno; Lorna Smith; Sébastien Ourselin; Rolf H. Jäger; M. Jorge Cardoso

doi:10.1007/978-3-030-32251-9_73

Let’s agree to disagree: Learning highly debatable multirater labelling

Carole H. Sudre^*, Beatriz Gomez Anson, Silvia Ingala, Chris D. Lane, Daniel Jimenez, Lukas Haider, Thomas Varsavsky, Ryutaro Tanno, Lorna Smith, Sébastien Ourselin, Rolf H. Jäger, M. Jorge Cardoso

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

17 Citations (Scopus)

Abstract

Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multi-rater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings
Editors	Dinggang Shen, Pew-Thian Yap, Tianming Liu, Terry M. Peters, Ali Khan, Lawrence H. Staib, Caroline Essert, Sean Zhou
Publisher	SPRINGER
Pages	665-673
Number of pages	9
ISBN (Print)	9783030322502
DOIs	https://doi.org/10.1007/978-3-030-32251-9_73
Publication status	Published - 1 Jan 2019
Event	22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019 - Shenzhen, China Duration: 13 Oct 2019 → 17 Oct 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11767 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019
Country/Territory	China
City	Shenzhen
Period	13/10/2019 → 17/10/2019

Keywords

Classification
Deep learning
Noisy labels

Access to Document

10.1007/978-3-030-32251-9_73

Cite this

Sudre, C. H., Anson, B. G., Ingala, S., Lane, C. D., Jimenez, D., Haider, L., Varsavsky, T., Tanno, R., Smith, L., Ourselin, S., Jäger, R. H., & Cardoso, M. J. (2019). Let’s agree to disagree: Learning highly debatable multirater labelling. In D. Shen, P.-T. Yap, T. Liu, T. M. Peters, A. Khan, L. H. Staib, C. Essert, & S. Zhou (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings (pp. 665-673). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11767 LNCS). SPRINGER. https://doi.org/10.1007/978-3-030-32251-9_73

Sudre, Carole H. ; Anson, Beatriz Gomez ; Ingala, Silvia et al. / Let’s agree to disagree : Learning highly debatable multirater labelling. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. editor / Dinggang Shen ; Pew-Thian Yap ; Tianming Liu ; Terry M. Peters ; Ali Khan ; Lawrence H. Staib ; Caroline Essert ; Sean Zhou. SPRINGER, 2019. pp. 665-673 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{f6b81d94ce174c78b3202ade8190ea61,

title = "Let{\textquoteright}s agree to disagree: Learning highly debatable multirater labelling",

abstract = "Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multi-rater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.",

keywords = "Classification, Deep learning, Noisy labels",

author = "Sudre, {Carole H.} and Anson, {Beatriz Gomez} and Silvia Ingala and Lane, {Chris D.} and Daniel Jimenez and Lukas Haider and Thomas Varsavsky and Ryutaro Tanno and Lorna Smith and S{\'e}bastien Ourselin and J{\"a}ger, {Rolf H.} and Cardoso, {M. Jorge}",

year = "2019",

month = jan,

day = "1",

doi = "10.1007/978-3-030-32251-9_73",

language = "English",

isbn = "9783030322502",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "SPRINGER",

pages = "665--673",

editor = "Dinggang Shen and Pew-Thian Yap and Tianming Liu and Peters, {Terry M.} and Ali Khan and Staib, {Lawrence H.} and Caroline Essert and Sean Zhou",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings",

note = "22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019 ; Conference date: 13-10-2019 Through 17-10-2019",

}

Sudre, CH, Anson, BG, Ingala, S, Lane, CD, Jimenez, D, Haider, L, Varsavsky, T, Tanno, R, Smith, L, Ourselin, S, Jäger, RH & Cardoso, MJ 2019, Let’s agree to disagree: Learning highly debatable multirater labelling. in D Shen, P-T Yap, T Liu, TM Peters, A Khan, LH Staib, C Essert & S Zhou (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11767 LNCS, SPRINGER, pp. 665-673, 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, Shenzhen, China, 13/10/2019. https://doi.org/10.1007/978-3-030-32251-9_73

Let’s agree to disagree: Learning highly debatable multirater labelling. / Sudre, Carole H.; Anson, Beatriz Gomez; Ingala, Silvia et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. ed. / Dinggang Shen; Pew-Thian Yap; Tianming Liu; Terry M. Peters; Ali Khan; Lawrence H. Staib; Caroline Essert; Sean Zhou. SPRINGER, 2019. p. 665-673 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11767 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

TY - CHAP

T1 - Let’s agree to disagree

T2 - 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019

AU - Sudre, Carole H.

AU - Anson, Beatriz Gomez

AU - Ingala, Silvia

AU - Lane, Chris D.

AU - Jimenez, Daniel

AU - Haider, Lukas

AU - Varsavsky, Thomas

AU - Tanno, Ryutaro

AU - Smith, Lorna

AU - Ourselin, Sébastien

AU - Jäger, Rolf H.

AU - Cardoso, M. Jorge

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multi-rater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.

AB - Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multi-rater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.

KW - Classification

KW - Deep learning

KW - Noisy labels

UR - http://www.scopus.com/inward/record.url?scp=85075689775&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-32251-9_73

DO - 10.1007/978-3-030-32251-9_73

M3 - Conference paper

AN - SCOPUS:85075689775

SN - 9783030322502

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 665

EP - 673

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings

A2 - Shen, Dinggang

A2 - Yap, Pew-Thian

A2 - Liu, Tianming

A2 - Peters, Terry M.

A2 - Khan, Ali

A2 - Staib, Lawrence H.

A2 - Essert, Caroline

A2 - Zhou, Sean

PB - SPRINGER

Y2 - 13 October 2019 through 17 October 2019

ER -

Sudre CH, Anson BG, Ingala S, Lane CD, Jimenez D, Haider L et al. Let’s agree to disagree: Learning highly debatable multirater labelling. In Shen D, Yap PT, Liu T, Peters TM, Khan A, Staib LH, Essert C, Zhou S, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 - 22nd International Conference, Proceedings. SPRINGER. 2019. p. 665-673. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-32251-9_73

Let’s agree to disagree: Learning highly debatable multirater labelling

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this