Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression

Yamiko Joseph Msosa; Arturas Grauslys; Yifan Zhou; Tao Wang; Iain Buchan; Paul Langan; Steven Foster; Michael Walker; Michael Pearson; Amos Folarin; Angus Roberts; Simon Maskell; Richard Dobson; Cecil Kullu; Dennis Kehoe

doi:10.1109/JBHI.2023.3312011

Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression

Yamiko Joseph Msosa, Arturas Grauslys, Yifan Zhou, Tao Wang, Iain Buchan, Paul Langan, Steven Foster, Michael Walker, Michael Pearson, Amos Folarin, Angus Roberts, Simon Maskell, Richard Dobson, Cecil Kullu, Dennis Kehoe

Research output: Contribution to journal › Article › peer-review

9 Citations (Scopus)

Abstract

Depression is a common mental health condition that often occurs in association with other chronic illnesses, and varies considerably in severity. Electronic Health Records (EHRs) contain rich information about a patient's medical history and can be used to train, test and maintain predictive models to support and improve patient care. This work evaluated the feasibility of implementing an environment for predicting mental health crisis among people living with depression based on both structured and unstructured EHRs. A large EHR from a mental health provider, Mersey Care, was pseudonymised and ingested into the Natural Language Processing (NLP) platform CogStack, allowing text content in binary clinical notes to be extracted. All unstructured clinical notes and summaries were semantically annotated by MedCAT and BioYODIE NLP services. Cases of crisis in patients with depression were then identified. Random forest models, gradient boosting trees, and Long Short-Term Memory (LSTM) networks, with varying feature arrangement, were trained to predict the occurrence of crisis. The results showed that all the prediction models can use a combination of structured and unstructured EHR information to predict crisis in patients with depression with good and useful accuracy. The LSTM network that was trained on a modified dataset with only 1000 most-important features from the random forest model with temporality showed the best performance with a mean AUC of 0.901 and a standard deviation of 0.006 using a training dataset and a mean AUC of 0.810 and 0.01 using a hold-out test dataset. Comparing the results from the technical evaluation with the views of psychiatrists shows that there are now opportunities to refine and integrate such prediction models into pragmatic point-of-care clinical decision support tools for supporting mental healthcare delivery.

Original language	English
Pages (from-to)	5588-5598
Number of pages	11
Journal	IEEE Journal of Biomedical and Health Informatics
Volume	27
Issue number	11
DOIs	https://doi.org/10.1109/JBHI.2023.3312011
Publication status	Published - 1 Nov 2023

Keywords

Annotations
Artificial Intelligence
Bioinformatics
Depression
Digital Health
Medical services
Mental health
Mental Health Crisis
Natural Language Processing
Natural language processing
Predictive models

Access to Document

10.1109/JBHI.2023.3312011

Cite this

Msosa, Y. J., Grauslys, A., Zhou, Y., Wang, T., Buchan, I., Langan, P., Foster, S., Walker, M., Pearson, M., Folarin, A., Roberts, A., Maskell, S., Dobson, R., Kullu, C., & Kehoe, D. (2023). Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression. IEEE Journal of Biomedical and Health Informatics, 27(11), 5588-5598. https://doi.org/10.1109/JBHI.2023.3312011

@article{5b88ebc8544b40f5bc7671dbb2a1e75e,

title = "Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression",

abstract = "Depression is a common mental health condition that often occurs in association with other chronic illnesses, and varies considerably in severity. Electronic Health Records (EHRs) contain rich information about a patient's medical history and can be used to train, test and maintain predictive models to support and improve patient care. This work evaluated the feasibility of implementing an environment for predicting mental health crisis among people living with depression based on both structured and unstructured EHRs. A large EHR from a mental health provider, Mersey Care, was pseudonymised and ingested into the Natural Language Processing (NLP) platform CogStack, allowing text content in binary clinical notes to be extracted. All unstructured clinical notes and summaries were semantically annotated by MedCAT and BioYODIE NLP services. Cases of crisis in patients with depression were then identified. Random forest models, gradient boosting trees, and Long Short-Term Memory (LSTM) networks, with varying feature arrangement, were trained to predict the occurrence of crisis. The results showed that all the prediction models can use a combination of structured and unstructured EHR information to predict crisis in patients with depression with good and useful accuracy. The LSTM network that was trained on a modified dataset with only 1000 most-important features from the random forest model with temporality showed the best performance with a mean AUC of 0.901 and a standard deviation of 0.006 using a training dataset and a mean AUC of 0.810 and 0.01 using a hold-out test dataset. Comparing the results from the technical evaluation with the views of psychiatrists shows that there are now opportunities to refine and integrate such prediction models into pragmatic point-of-care clinical decision support tools for supporting mental healthcare delivery.",

keywords = "Annotations, Artificial Intelligence, Bioinformatics, Depression, Digital Health, Medical services, Mental health, Mental Health Crisis, Natural Language Processing, Natural language processing, Predictive models",

author = "Msosa, {Yamiko Joseph} and Arturas Grauslys and Yifan Zhou and Tao Wang and Iain Buchan and Paul Langan and Steven Foster and Michael Walker and Michael Pearson and Amos Folarin and Angus Roberts and Simon Maskell and Richard Dobson and Cecil Kullu and Dennis Kehoe",

note = "Publisher Copyright: {\textcopyright} 2013 IEEE.",

year = "2023",

month = nov,

day = "1",

doi = "10.1109/JBHI.2023.3312011",

language = "English",

volume = "27",

pages = "5588--5598",

journal = "IEEE Journal of Biomedical and Health Informatics",

issn = "2168-2194",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "11",

}

Msosa, YJ, Grauslys, A, Zhou, Y, Wang, T, Buchan, I, Langan, P, Foster, S, Walker, M, Pearson, M, Folarin, A , Roberts, A, Maskell, S, Dobson, R, Kullu, C & Kehoe, D 2023, 'Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression', IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 11, pp. 5588-5598. https://doi.org/10.1109/JBHI.2023.3312011

TY - JOUR

T1 - Trustworthy Data and AI Environments for Clinical Prediction

T2 - Application to Crisis-Risk in People with Depression

AU - Msosa, Yamiko Joseph

AU - Grauslys, Arturas

AU - Zhou, Yifan

AU - Wang, Tao

AU - Buchan, Iain

AU - Langan, Paul

AU - Foster, Steven

AU - Walker, Michael

AU - Pearson, Michael

AU - Folarin, Amos

AU - Roberts, Angus

AU - Maskell, Simon

AU - Dobson, Richard

AU - Kullu, Cecil

AU - Kehoe, Dennis

PY - 2023/11/1

Y1 - 2023/11/1

N2 - Depression is a common mental health condition that often occurs in association with other chronic illnesses, and varies considerably in severity. Electronic Health Records (EHRs) contain rich information about a patient's medical history and can be used to train, test and maintain predictive models to support and improve patient care. This work evaluated the feasibility of implementing an environment for predicting mental health crisis among people living with depression based on both structured and unstructured EHRs. A large EHR from a mental health provider, Mersey Care, was pseudonymised and ingested into the Natural Language Processing (NLP) platform CogStack, allowing text content in binary clinical notes to be extracted. All unstructured clinical notes and summaries were semantically annotated by MedCAT and BioYODIE NLP services. Cases of crisis in patients with depression were then identified. Random forest models, gradient boosting trees, and Long Short-Term Memory (LSTM) networks, with varying feature arrangement, were trained to predict the occurrence of crisis. The results showed that all the prediction models can use a combination of structured and unstructured EHR information to predict crisis in patients with depression with good and useful accuracy. The LSTM network that was trained on a modified dataset with only 1000 most-important features from the random forest model with temporality showed the best performance with a mean AUC of 0.901 and a standard deviation of 0.006 using a training dataset and a mean AUC of 0.810 and 0.01 using a hold-out test dataset. Comparing the results from the technical evaluation with the views of psychiatrists shows that there are now opportunities to refine and integrate such prediction models into pragmatic point-of-care clinical decision support tools for supporting mental healthcare delivery.

AB - Depression is a common mental health condition that often occurs in association with other chronic illnesses, and varies considerably in severity. Electronic Health Records (EHRs) contain rich information about a patient's medical history and can be used to train, test and maintain predictive models to support and improve patient care. This work evaluated the feasibility of implementing an environment for predicting mental health crisis among people living with depression based on both structured and unstructured EHRs. A large EHR from a mental health provider, Mersey Care, was pseudonymised and ingested into the Natural Language Processing (NLP) platform CogStack, allowing text content in binary clinical notes to be extracted. All unstructured clinical notes and summaries were semantically annotated by MedCAT and BioYODIE NLP services. Cases of crisis in patients with depression were then identified. Random forest models, gradient boosting trees, and Long Short-Term Memory (LSTM) networks, with varying feature arrangement, were trained to predict the occurrence of crisis. The results showed that all the prediction models can use a combination of structured and unstructured EHR information to predict crisis in patients with depression with good and useful accuracy. The LSTM network that was trained on a modified dataset with only 1000 most-important features from the random forest model with temporality showed the best performance with a mean AUC of 0.901 and a standard deviation of 0.006 using a training dataset and a mean AUC of 0.810 and 0.01 using a hold-out test dataset. Comparing the results from the technical evaluation with the views of psychiatrists shows that there are now opportunities to refine and integrate such prediction models into pragmatic point-of-care clinical decision support tools for supporting mental healthcare delivery.

KW - Annotations

KW - Artificial Intelligence

KW - Bioinformatics

KW - Depression

KW - Digital Health

KW - Medical services

KW - Mental health

KW - Mental Health Crisis

KW - Natural Language Processing

KW - Natural language processing

KW - Predictive models

UR - http://www.scopus.com/inward/record.url?scp=85171537639&partnerID=8YFLogxK

U2 - 10.1109/JBHI.2023.3312011

DO - 10.1109/JBHI.2023.3312011

M3 - Article

C2 - 37669205

AN - SCOPUS:85171537639

SN - 2168-2194

VL - 27

SP - 5588

EP - 5598

JO - IEEE Journal of Biomedical and Health Informatics

JF - IEEE Journal of Biomedical and Health Informatics

IS - 11

ER -

Trustworthy Data and AI Environments for Clinical Prediction: Application to Crisis-Risk in People with Depression

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this