Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

Kristin E. Wickstrom; Valeria Vitelli; Ewan Carr; Aleksander R. Holten; Rebecca Bendayan; Andrew H. Reiner; Daniel Bean; Tom Searle; Anthony Shek; Zeljko Kraljevic; James Teo; Richard Dobson; Kristian Tonby; Alvaro Kohn-Luque; Erik K. Amundsen

doi:10.1371/journal.pone.0255748

Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

Kristin E. Wickstrom, Valeria Vitelli, Ewan Carr, Aleksander R. Holten, Rebecca Bendayan, Andrew H. Reiner, Daniel Bean, Tom Searle, Anthony Shek, Zeljko Kraljevic, James Teo, Richard Dobson, Kristian Tonby, Alvaro Kohn-Luque, Erik K. Amundsen

Research output: Contribution to journal › Article › peer-review

3 Citations (Scopus)

Abstract

BACKGROUND: Prediction models should be externally validated to assess their performance before implementation. Several prediction models for coronavirus disease-19 (COVID-19) have been published. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors.

METHODS: Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration.

RESULTS: We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95% confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration.

CONCLUSIONS: The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

Original language	English
Article number	e0255748
Pages (from-to)	e0255748
Journal	PLoS ONE
Volume	16
Issue number	8
DOIs	https://doi.org/10.1371/journal.pone.0255748
Publication status	Published - 25 Aug 2021

Access to Document

10.1371/journal.pone.0255748

Cite this

Wickstrom, K. E., Vitelli, V., Carr, E., Holten, A. R., Bendayan, R., Reiner, A. H., Bean, D., Searle, T., Shek, A., Kraljevic, Z., Teo, J., Dobson, R., Tonby, K., Kohn-Luque, A., & Amundsen, E. K. (2021). Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study. PLoS ONE, 16(8), e0255748. Article e0255748. https://doi.org/10.1371/journal.pone.0255748

@article{95f3d726fcaf4942a4c87c289fb34bcc,

title = "Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study",

abstract = "BACKGROUND: Prediction models should be externally validated to assess their performance before implementation. Several prediction models for coronavirus disease-19 (COVID-19) have been published. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors.METHODS: Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration.RESULTS: We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95% confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration.CONCLUSIONS: The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.",

author = "Wickstrom, {Kristin E.} and Valeria Vitelli and Ewan Carr and Holten, {Aleksander R.} and Rebecca Bendayan and Reiner, {Andrew H.} and Daniel Bean and Tom Searle and Anthony Shek and Zeljko Kraljevic and James Teo and Richard Dobson and Kristian Tonby and Alvaro Kohn-Luque and Amundsen, {Erik K.}",

note = "Funding Information: One of the authors (JTHT) have previously received research support and funding from InnovateUL, Bristol-Myers-Squibb, iRhytm Technologies, and hold shares under ?5000 in Glaxo SmithKline and Biogen. No external commercial financial support was received for this work. Institutional support and grants are stated under funding information. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: {\textcopyright} 2021 Wickstr{\o}m et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = aug,

day = "25",

doi = "10.1371/journal.pone.0255748",

language = "English",

volume = "16",

pages = "e0255748",

journal = "PLoS ONE",

issn = "1932-6203",

publisher = "Public Library of Science",

number = "8",

}

Wickstrom, KE, Vitelli, V, Carr, E, Holten, AR, Bendayan, R, Reiner, AH, Bean, D , Searle, T , Shek, A , Kraljevic, Z , Teo, J , Dobson, R, Tonby, K, Kohn-Luque, A & Amundsen, EK 2021, 'Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study', PLoS ONE, vol. 16, no. 8, e0255748, pp. e0255748. https://doi.org/10.1371/journal.pone.0255748

TY - JOUR

T1 - Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission

T2 - An observational multi-centre cohort study

AU - Wickstrom, Kristin E.

AU - Vitelli, Valeria

AU - Carr, Ewan

AU - Holten, Aleksander R.

AU - Bendayan, Rebecca

AU - Reiner, Andrew H.

AU - Bean, Daniel

AU - Searle, Tom

AU - Shek, Anthony

AU - Kraljevic, Zeljko

AU - Teo, James

AU - Dobson, Richard

AU - Tonby, Kristian

AU - Kohn-Luque, Alvaro

AU - Amundsen, Erik K.

N1 - Funding Information: One of the authors (JTHT) have previously received research support and funding from InnovateUL, Bristol-Myers-Squibb, iRhytm Technologies, and hold shares under ?5000 in Glaxo SmithKline and Biogen. No external commercial financial support was received for this work. Institutional support and grants are stated under funding information. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: © 2021 Wickstrøm et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/8/25

Y1 - 2021/8/25

N2 - BACKGROUND: Prediction models should be externally validated to assess their performance before implementation. Several prediction models for coronavirus disease-19 (COVID-19) have been published. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors.METHODS: Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration.RESULTS: We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95% confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration.CONCLUSIONS: The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

AB - BACKGROUND: Prediction models should be externally validated to assess their performance before implementation. Several prediction models for coronavirus disease-19 (COVID-19) have been published. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors.METHODS: Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration.RESULTS: We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95% confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration.CONCLUSIONS: The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

UR - http://www.scopus.com/inward/record.url?scp=85113734650&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0255748

DO - 10.1371/journal.pone.0255748

M3 - Article

C2 - 34432797

SN - 1932-6203

VL - 16

SP - e0255748

JO - PLoS ONE

JF - PLoS ONE

IS - 8

M1 - e0255748

ER -

Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

Abstract

Access to Document

Other files and links

Fingerprint

Cite this