Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports

Siddharth Agarwal; David Wood; Benjamin A. K. Murray; Yiran Wei; Ayisha Al Busaidi; Sina Kafiabadi; Emily Guilhem; Jeremy Lynch; Matthew Townend; Asif Mazumder; Gareth J Barker; James H Cole; Peter Sasieni; Sebastien Ourselin; Marc Modat; Thomas C. Booth

doi:10.1007/s00330-025-11500-9

Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports

Siddharth Agarwal, David Wood, Benjamin A. K. Murray, Yiran Wei, Ayisha Al Busaidi, Sina Kafiabadi, Emily Guilhem, Jeremy Lynch, Matthew Townend, Asif Mazumder, Gareth J Barker, James H Cole, Peter Sasieni, Sebastien Ourselin, Marc Modat, Thomas C. Booth

Research output: Contribution to journal › Article › peer-review

36 Downloads (Pure)

Abstract

Objectives: To determine the effectiveness of hospital-specific domain adaptation through masked language modelling (MLM) on BERT-based models’ performance in classifying neuroradiology reports, and to compare these models with open-source large language models (LLMs). Materials and methods: This retrospective study (2008–2019) utilised 126,556 and 86,032 MRI brain reports from two tertiary hospitals—King’s College Hospital (KCH) and Guys and St Thomas’ Trust (GSTT). Various BERT-based models, including RoBERTa, BioBERT and RadBERT, underwent MLM on unlabelled reports from these centres. The downstream tasks were binary abnormality classification and multi-label classification. Performances of models with and without hospital-specific domain adaptation were compared against each other and LLMs on internal (KCH) and external (GSTT) hold-out test sets. Model performances for binary classification were compared using 2-way and 1-way ANOVA. Results: All models that underwent hospital-specific domain adaptation performed better than their baseline counterparts (all p-values < 0.001). For binary classification, MLM on all available unlabelled reports (194,467 reports) yielded the highest balanced accuracies (KCH: mean 97.0 ± 0.4% (standard deviation), GSTT: 95.5 ± 1.0%), after which no differences between BERT-based models remained (1-way ANOVA, p-values > 0.05). There was a log-linear relationship between the number of reports and performance. LLama-3.0 70B was the best-performing LLM (KCH: 97.1%, GSTT: 94.0%). Multi-label classification demonstrated consistent performance improvements from MLM for all abnormality categories. Conclusion: Hospital-specific domain adaptation should be considered best practice when deploying BERT-based models in new clinical settings. When labelled data is scarce or unavailable, LLMs can serve as a viable alternative, assuming adequate computational power is accessible. Key Points: Question BERT-based models can classify radiology reports, but it is unclear if there is any incremental benefit from additional hospital-specific domain adaptation. Findings Hospital-specific domain adaptation resulted in the highest BERT-based model accuracies and performance scaled log-linearly with the number of reports. Clinical relevance BERT-based models after hospital-specific domain adaptation achieve the best classification results provided sufficient high-quality training labels. When labelled data is scarce, LLMs such as Llama-3.0 70B are a viable alternative provided there are sufficient computational resources.

Original language	English
Article number	102391
Journal	European Radiology
Early online date	17 Mar 2025
DOIs	https://doi.org/10.1007/s00330-025-11500-9
Publication status	E-pub ahead of print - 17 Mar 2025

Access to Document

10.1007/s00330-025-11500-9Licence: CC BY

Impact of hospital-specific_AGARWAL_Publishedonline17March2025_GOLD_VoR (CC BY)
© The Author(s) 2025. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Final published version, 2.01 MBLicence: CC BY

Cite this

Agarwal, S., Wood, D., Murray, B. A. K., Wei, Y., Busaidi, A. A., Kafiabadi, S., Guilhem, E., Lynch, J., Townend, M., Mazumder, A., Barker, G. J., Cole, J. H., Sasieni, P., Ourselin, S., Modat, M., & Booth, T. C. (2025). Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports. European Radiology, Article 102391. Advance online publication. https://doi.org/10.1007/s00330-025-11500-9

@article{0ba99fc6e5954659b7b954ea38a5598c,

title = "Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports",

abstract = "Objectives: To determine the effectiveness of hospital-specific domain adaptation through masked language modelling (MLM) on BERT-based models{\textquoteright} performance in classifying neuroradiology reports, and to compare these models with open-source large language models (LLMs). Materials and methods: This retrospective study (2008–2019) utilised 126,556 and 86,032 MRI brain reports from two tertiary hospitals—King{\textquoteright}s College Hospital (KCH) and Guys and St Thomas{\textquoteright} Trust (GSTT). Various BERT-based models, including RoBERTa, BioBERT and RadBERT, underwent MLM on unlabelled reports from these centres. The downstream tasks were binary abnormality classification and multi-label classification. Performances of models with and without hospital-specific domain adaptation were compared against each other and LLMs on internal (KCH) and external (GSTT) hold-out test sets. Model performances for binary classification were compared using 2-way and 1-way ANOVA. Results: All models that underwent hospital-specific domain adaptation performed better than their baseline counterparts (all p-values < 0.001). For binary classification, MLM on all available unlabelled reports (194,467 reports) yielded the highest balanced accuracies (KCH: mean 97.0 ± 0.4% (standard deviation), GSTT: 95.5 ± 1.0%), after which no differences between BERT-based models remained (1-way ANOVA, p-values > 0.05). There was a log-linear relationship between the number of reports and performance. LLama-3.0 70B was the best-performing LLM (KCH: 97.1%, GSTT: 94.0%). Multi-label classification demonstrated consistent performance improvements from MLM for all abnormality categories. Conclusion: Hospital-specific domain adaptation should be considered best practice when deploying BERT-based models in new clinical settings. When labelled data is scarce or unavailable, LLMs can serve as a viable alternative, assuming adequate computational power is accessible. Key Points: Question BERT-based models can classify radiology reports, but it is unclear if there is any incremental benefit from additional hospital-specific domain adaptation. Findings Hospital-specific domain adaptation resulted in the highest BERT-based model accuracies and performance scaled log-linearly with the number of reports. Clinical relevance BERT-based models after hospital-specific domain adaptation achieve the best classification results provided sufficient high-quality training labels. When labelled data is scarce, LLMs such as Llama-3.0 70B are a viable alternative provided there are sufficient computational resources.",

author = "Siddharth Agarwal and David Wood and Murray, {Benjamin A. K.} and Yiran Wei and Busaidi, {Ayisha Al} and Sina Kafiabadi and Emily Guilhem and Jeremy Lynch and Matthew Townend and Asif Mazumder and Barker, {Gareth J} and Cole, {James H} and Peter Sasieni and Sebastien Ourselin and Marc Modat and Booth, {Thomas C.}",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.",

year = "2025",

month = mar,

day = "17",

doi = "10.1007/s00330-025-11500-9",

language = "English",

journal = "European Radiology",

issn = "0938-7994",

publisher = "Springer Verlag",

}

TY - JOUR

T1 - Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports

AU - Agarwal, Siddharth

AU - Wood, David

AU - Murray, Benjamin A. K.

AU - Wei, Yiran

AU - Busaidi, Ayisha Al

AU - Kafiabadi, Sina

AU - Guilhem, Emily

AU - Lynch, Jeremy

AU - Townend, Matthew

AU - Mazumder, Asif

AU - Barker, Gareth J

AU - Cole, James H

AU - Sasieni, Peter

AU - Ourselin, Sebastien

AU - Modat, Marc

AU - Booth, Thomas C.

PY - 2025/3/17

Y1 - 2025/3/17

N2 - Objectives: To determine the effectiveness of hospital-specific domain adaptation through masked language modelling (MLM) on BERT-based models’ performance in classifying neuroradiology reports, and to compare these models with open-source large language models (LLMs). Materials and methods: This retrospective study (2008–2019) utilised 126,556 and 86,032 MRI brain reports from two tertiary hospitals—King’s College Hospital (KCH) and Guys and St Thomas’ Trust (GSTT). Various BERT-based models, including RoBERTa, BioBERT and RadBERT, underwent MLM on unlabelled reports from these centres. The downstream tasks were binary abnormality classification and multi-label classification. Performances of models with and without hospital-specific domain adaptation were compared against each other and LLMs on internal (KCH) and external (GSTT) hold-out test sets. Model performances for binary classification were compared using 2-way and 1-way ANOVA. Results: All models that underwent hospital-specific domain adaptation performed better than their baseline counterparts (all p-values < 0.001). For binary classification, MLM on all available unlabelled reports (194,467 reports) yielded the highest balanced accuracies (KCH: mean 97.0 ± 0.4% (standard deviation), GSTT: 95.5 ± 1.0%), after which no differences between BERT-based models remained (1-way ANOVA, p-values > 0.05). There was a log-linear relationship between the number of reports and performance. LLama-3.0 70B was the best-performing LLM (KCH: 97.1%, GSTT: 94.0%). Multi-label classification demonstrated consistent performance improvements from MLM for all abnormality categories. Conclusion: Hospital-specific domain adaptation should be considered best practice when deploying BERT-based models in new clinical settings. When labelled data is scarce or unavailable, LLMs can serve as a viable alternative, assuming adequate computational power is accessible. Key Points: Question BERT-based models can classify radiology reports, but it is unclear if there is any incremental benefit from additional hospital-specific domain adaptation. Findings Hospital-specific domain adaptation resulted in the highest BERT-based model accuracies and performance scaled log-linearly with the number of reports. Clinical relevance BERT-based models after hospital-specific domain adaptation achieve the best classification results provided sufficient high-quality training labels. When labelled data is scarce, LLMs such as Llama-3.0 70B are a viable alternative provided there are sufficient computational resources.

AB - Objectives: To determine the effectiveness of hospital-specific domain adaptation through masked language modelling (MLM) on BERT-based models’ performance in classifying neuroradiology reports, and to compare these models with open-source large language models (LLMs). Materials and methods: This retrospective study (2008–2019) utilised 126,556 and 86,032 MRI brain reports from two tertiary hospitals—King’s College Hospital (KCH) and Guys and St Thomas’ Trust (GSTT). Various BERT-based models, including RoBERTa, BioBERT and RadBERT, underwent MLM on unlabelled reports from these centres. The downstream tasks were binary abnormality classification and multi-label classification. Performances of models with and without hospital-specific domain adaptation were compared against each other and LLMs on internal (KCH) and external (GSTT) hold-out test sets. Model performances for binary classification were compared using 2-way and 1-way ANOVA. Results: All models that underwent hospital-specific domain adaptation performed better than their baseline counterparts (all p-values < 0.001). For binary classification, MLM on all available unlabelled reports (194,467 reports) yielded the highest balanced accuracies (KCH: mean 97.0 ± 0.4% (standard deviation), GSTT: 95.5 ± 1.0%), after which no differences between BERT-based models remained (1-way ANOVA, p-values > 0.05). There was a log-linear relationship between the number of reports and performance. LLama-3.0 70B was the best-performing LLM (KCH: 97.1%, GSTT: 94.0%). Multi-label classification demonstrated consistent performance improvements from MLM for all abnormality categories. Conclusion: Hospital-specific domain adaptation should be considered best practice when deploying BERT-based models in new clinical settings. When labelled data is scarce or unavailable, LLMs can serve as a viable alternative, assuming adequate computational power is accessible. Key Points: Question BERT-based models can classify radiology reports, but it is unclear if there is any incremental benefit from additional hospital-specific domain adaptation. Findings Hospital-specific domain adaptation resulted in the highest BERT-based model accuracies and performance scaled log-linearly with the number of reports. Clinical relevance BERT-based models after hospital-specific domain adaptation achieve the best classification results provided sufficient high-quality training labels. When labelled data is scarce, LLMs such as Llama-3.0 70B are a viable alternative provided there are sufficient computational resources.

UR - http://www.scopus.com/inward/record.url?scp=105000299752&partnerID=8YFLogxK

U2 - 10.1007/s00330-025-11500-9

DO - 10.1007/s00330-025-11500-9

M3 - Article

C2 - 40097844

SN - 0938-7994

JO - European Radiology

JF - European Radiology

M1 - 102391

ER -

Impact of hospital-specific domain adaptation on BERT-based models to classify neuroradiology reports

Abstract

Access to Document

Other files and links

Fingerprint

Cite this