TY - CHAP
T1 - The empirical variance estimator for computer aided diagnosis
T2 - 17th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2014
AU - Mendelson, Alex F.
AU - Zuluaga, Maria A.
AU - Thurfjell, Lennart
AU - Hutton, Brian F.
AU - Ourselin, Sébastien
PY - 2014/1/1
Y1 - 2014/1/1
N2 - Computer aided diagnosis is an established field in medical image analysis; a great deal of effort goes into the development and refinement of pipelines to achieve greater performance. This improvement is dependent on reliable comparison, which is intimately related to variance estimation. For supervised methods, this can be confounded by statistical issues at the comparatively small sample sizes typical of the field. Given the importance of reliable comparison to pipeline development, this issue has received relatively little attention. As a solution, we advocate an empirical variance estimator based on validation within disjoint subsets of the available data. Using Alzheimer's disease classification in the ADNI dataset as an examplar, we investigate the behaviour of different variance estimators in a series of resampling experiments. We show that the proposed estimator is unbiased, and that it exceeds the estimates of naive approaches, which are biased down. Because the estimator avoids independence assumptions, it is able to accommodate arbitrary validation strategies and performance metrics. As it is unbiased, it is able to provide statistically convincing comparison and confidence intervals for algorithm performance. Finally, we show how the estimator can be used to compare different validation strategies, and make some recommendations about which should be used.
AB - Computer aided diagnosis is an established field in medical image analysis; a great deal of effort goes into the development and refinement of pipelines to achieve greater performance. This improvement is dependent on reliable comparison, which is intimately related to variance estimation. For supervised methods, this can be confounded by statistical issues at the comparatively small sample sizes typical of the field. Given the importance of reliable comparison to pipeline development, this issue has received relatively little attention. As a solution, we advocate an empirical variance estimator based on validation within disjoint subsets of the available data. Using Alzheimer's disease classification in the ADNI dataset as an examplar, we investigate the behaviour of different variance estimators in a series of resampling experiments. We show that the proposed estimator is unbiased, and that it exceeds the estimates of naive approaches, which are biased down. Because the estimator avoids independence assumptions, it is able to accommodate arbitrary validation strategies and performance metrics. As it is unbiased, it is able to provide statistically convincing comparison and confidence intervals for algorithm performance. Finally, we show how the estimator can be used to compare different validation strategies, and make some recommendations about which should be used.
UR - https://www.scopus.com/pages/publications/84906977068
U2 - 10.1007/978-3-319-10470-6_30
DO - 10.1007/978-3-319-10470-6_30
M3 - Conference paper
AN - SCOPUS:84906977068
SN - 9783319104690
VL - 8674 LNCS
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 236
EP - 243
BT - Medical Image Computing and Computer-Assisted Intervention, MICCAI 2014 - 17th International Conference, Proceedings
PB - Springer Verlag
Y2 - 14 September 2014 through 18 September 2014
ER -