The meaning of significant mean group differences for biomarker discovery

Eva Loth; Jumana Ahmad; Chris Chatham; Beatriz Lopez; Ben Carter; Daisy Crawley; Beth Oakley; Hannah Hayward; Jennifer Cooke; Antonia San José Cáceres; Danilo Bzdok; Emily Jones; Tony Charman; Christian F. Beckmann; Thomas Bourgeron; Roberto Toro; Jan Buitelaar; Declan Murphy; Guillame Dumas

doi:10.1371/journal.pcbi.1009477

The meaning of significant mean group differences for biomarker discovery

Eva Loth^*, Jumana Ahmad, Chris Chatham, Beatriz Lopez, Ben Carter, Daisy Crawley, Beth Oakley, Hannah Hayward, Jennifer Cooke, Antonia San José Cáceres, Danilo Bzdok, Emily Jones, Tony Charman, Christian F. Beckmann, Thomas Bourgeron, Roberto Toro, Jan Buitelaar, Declan Murphy, Guillame Dumas

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

21 Citations (Scopus)

Abstract

Over the past decade, biomaker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between "cases"and "controls,"which tends to ignore withingroup variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research - autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen's d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the "on average"when summarising their findings in their abstracts ("autistic people have deficits in X"), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.

Original language	English
Article number	e1009477
Journal	PLoS Computational Biology
Volume	17
Issue number	11
DOIs	https://doi.org/10.1371/journal.pcbi.1009477
Publication status	Published - 18 Nov 2021

Access to Document

10.1371/journal.pcbi.1009477

Cite this

Loth, E., Ahmad, J., Chatham, C., Lopez, B., Carter, B., Crawley, D., Oakley, B., Hayward, H., Cooke, J., San José Cáceres, A., Bzdok, D., Jones, E., Charman, T., Beckmann, C. F., Bourgeron, T., Toro, R., Buitelaar, J., Murphy, D., & Dumas, G. (2021). The meaning of significant mean group differences for biomarker discovery. PLoS Computational Biology, 17(11), Article e1009477. https://doi.org/10.1371/journal.pcbi.1009477

@article{a6940d0ef9044693870c2be91b7d404f,

title = "The meaning of significant mean group differences for biomarker discovery",

abstract = "Over the past decade, biomaker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between {"}cases{"}and {"}controls,{"}which tends to ignore withingroup variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research - autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen's d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the {"}on average{"}when summarising their findings in their abstracts ({"}autistic people have deficits in X{"}), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.",

author = "Eva Loth and Jumana Ahmad and Chris Chatham and Beatriz Lopez and Ben Carter and Daisy Crawley and Beth Oakley and Hannah Hayward and Jennifer Cooke and {San Jos{\'e} C{\'a}ceres}, Antonia and Danilo Bzdok and Emily Jones and Tony Charman and Beckmann, {Christian F.} and Thomas Bourgeron and Roberto Toro and Jan Buitelaar and Declan Murphy and Guillame Dumas",

note = "Funding Information: EL, JA, BL, BC, DC, BO, HH, JC, ASJC, EJ, TC, CB, TB, RT, JB, DM, and GD have received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 777394 for the project AIMS-2-TRIALS. This Joint Undertaking is a joint support from the European Union's Horizon 2020 research and innovation programme, EFPIA, AUTISM SPEAKS, Autistica, and SFARI. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: {\textcopyright} 2021 Loth et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.",

year = "2021",

month = nov,

day = "18",

doi = "10.1371/journal.pcbi.1009477",

language = "English",

volume = "17",

journal = "PLoS Computational Biology",

issn = "1553-734X",

publisher = "Public Library of Science",

number = "11",

}

Loth, E, Ahmad, J, Chatham, C, Lopez, B, Carter, B , Crawley, D , Oakley, B , Hayward, H , Cooke, J, San José Cáceres, A, Bzdok, D, Jones, E, Charman, T, Beckmann, CF, Bourgeron, T, Toro, R, Buitelaar, J, Murphy, D & Dumas, G 2021, 'The meaning of significant mean group differences for biomarker discovery', PLoS Computational Biology, vol. 17, no. 11, e1009477. https://doi.org/10.1371/journal.pcbi.1009477

TY - JOUR

T1 - The meaning of significant mean group differences for biomarker discovery

AU - Loth, Eva

AU - Ahmad, Jumana

AU - Chatham, Chris

AU - Lopez, Beatriz

AU - Carter, Ben

AU - Crawley, Daisy

AU - Oakley, Beth

AU - Hayward, Hannah

AU - Cooke, Jennifer

AU - San José Cáceres, Antonia

AU - Bzdok, Danilo

AU - Jones, Emily

AU - Charman, Tony

AU - Beckmann, Christian F.

AU - Bourgeron, Thomas

AU - Toro, Roberto

AU - Buitelaar, Jan

AU - Murphy, Declan

AU - Dumas, Guillame

N1 - Funding Information: EL, JA, BL, BC, DC, BO, HH, JC, ASJC, EJ, TC, CB, TB, RT, JB, DM, and GD have received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 777394 for the project AIMS-2-TRIALS. This Joint Undertaking is a joint support from the European Union's Horizon 2020 research and innovation programme, EFPIA, AUTISM SPEAKS, Autistica, and SFARI. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: © 2021 Loth et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

PY - 2021/11/18

Y1 - 2021/11/18

N2 - Over the past decade, biomaker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between "cases"and "controls,"which tends to ignore withingroup variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research - autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen's d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the "on average"when summarising their findings in their abstracts ("autistic people have deficits in X"), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.

AB - Over the past decade, biomaker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between "cases"and "controls,"which tends to ignore withingroup variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research - autism research. Across the most influential areas of autism research, effect size estimates ranged from small (d = 0.21, anatomical structure) to medium (d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large (d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen's d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the "on average"when summarising their findings in their abstracts ("autistic people have deficits in X"), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.

UR - http://www.scopus.com/inward/record.url?scp=85119933490&partnerID=8YFLogxK

U2 - 10.1371/journal.pcbi.1009477

DO - 10.1371/journal.pcbi.1009477

M3 - Article

SN - 1553-734X

VL - 17

JO - PLoS Computational Biology

JF - PLoS Computational Biology

IS - 11

M1 - e1009477

ER -

The meaning of significant mean group differences for biomarker discovery

Abstract

Access to Document

Other files and links

Fingerprint

Cite this