Developing a Natural Language Processing tool to identify perinatal self-harm in electronic healthcare records

Karyn Ayre; André Bittar; Joyce Kam; Somain Verma; Louise M Howard; Rina Dutta

doi:10.1371/journal.pone.0253809

Developing a Natural Language Processing tool to identify perinatal self-harm in electronic healthcare records

Karyn Ayre, André Bittar, Joyce Kam, Somain Verma, Louise M Howard, Rina Dutta

Research output: Contribution to journal › Article › peer-review

10 Citations (Scopus)

Abstract

BACKGROUND: Self-harm occurring within pregnancy and the postnatal year ("perinatal self-harm") is a clinically important yet under-researched topic. Current research likely under-estimates prevalence due to methodological limitations. Electronic healthcare records (EHRs) provide a source of clinically rich data on perinatal self-harm.

AIMS: (1) To create a Natural Language Processing (NLP) tool that can, with acceptable precision and recall, identify mentions of acts of perinatal self-harm within EHRs. (2) To use this tool to identify service-users who have self-harmed perinatally, based on their EHRs.

METHODS: We used the Clinical Record Interactive Search system to extract de-identified EHRs of secondary mental healthcare service-users at South London and Maudsley NHS Foundation Trust. We developed a tool that applied several layers of linguistic processing based on the spaCy NLP library for Python. We evaluated mention-level performance in the following domains: span, status, temporality and polarity. Evaluation was done against a manually coded reference standard. Mention-level performance was reported as precision, recall, F-score and Cohen's kappa for each domain. Performance was also assessed at 'service-user' level and explored whether a heuristic rule improved this. We report per-class statistics for service-user performance, as well as likelihood ratios and post-test probabilities.

RESULTS: Mention-level performance: micro-averaged F-score, precision and recall for span, polarity and temporality >0.8. Kappa for status 0.68, temporality 0.62, polarity 0.91. Service-user level performance with heuristic: F-score, precision, recall of minority class 0.69, macro-averaged F-score 0.81, positive LR 9.4 (4.8-19), post-test probability 69.0% (53-82%). Considering the task difficulty, the tool performs well, although temporality was the attribute with the lowest level of annotator agreement.

CONCLUSIONS: It is feasible to develop an NLP tool that identifies, with acceptable validity, mentions of perinatal self-harm within EHRs, although with limitations regarding temporality. Using a heuristic rule, it can also function at a service-user-level.

Original language	English
Article number	e0253809
Pages (from-to)	e0253809
Journal	PLoS ONE
Volume	16
Issue number	8 August
DOIs	https://doi.org/10.1371/journal.pone.0253809
Publication status	Published - Aug 2021

Access to Document

10.1371/journal.pone.0253809

Cite this

@article{53ab33d077b64875a2367b7a90480ee7,

title = "Developing a Natural Language Processing tool to identify perinatal self-harm in electronic healthcare records",

abstract = "BACKGROUND: Self-harm occurring within pregnancy and the postnatal year ({"}perinatal self-harm{"}) is a clinically important yet under-researched topic. Current research likely under-estimates prevalence due to methodological limitations. Electronic healthcare records (EHRs) provide a source of clinically rich data on perinatal self-harm.AIMS: (1) To create a Natural Language Processing (NLP) tool that can, with acceptable precision and recall, identify mentions of acts of perinatal self-harm within EHRs. (2) To use this tool to identify service-users who have self-harmed perinatally, based on their EHRs.METHODS: We used the Clinical Record Interactive Search system to extract de-identified EHRs of secondary mental healthcare service-users at South London and Maudsley NHS Foundation Trust. We developed a tool that applied several layers of linguistic processing based on the spaCy NLP library for Python. We evaluated mention-level performance in the following domains: span, status, temporality and polarity. Evaluation was done against a manually coded reference standard. Mention-level performance was reported as precision, recall, F-score and Cohen's kappa for each domain. Performance was also assessed at 'service-user' level and explored whether a heuristic rule improved this. We report per-class statistics for service-user performance, as well as likelihood ratios and post-test probabilities.RESULTS: Mention-level performance: micro-averaged F-score, precision and recall for span, polarity and temporality >0.8. Kappa for status 0.68, temporality 0.62, polarity 0.91. Service-user level performance with heuristic: F-score, precision, recall of minority class 0.69, macro-averaged F-score 0.81, positive LR 9.4 (4.8-19), post-test probability 69.0% (53-82%). Considering the task difficulty, the tool performs well, although temporality was the attribute with the lowest level of annotator agreement.CONCLUSIONS: It is feasible to develop an NLP tool that identifies, with acceptable validity, mentions of perinatal self-harm within EHRs, although with limitations regarding temporality. Using a heuristic rule, it can also function at a service-user-level.",

author = "Karyn Ayre and Andr{\'e} Bittar and Joyce Kam and Somain Verma and Howard, {Louise M} and Rina Dutta",

note = "Funding Information: KA is funded by a National Institute for Health Research Doctoral Research Fellowship (NIHR-DRF-2016-09-042). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. https://www.nihr.ac.uk/ RD is funded by a Clinician Scientist Fellowship (research project e-HOST-IT) from the Health Foundation in partnership with the Academy of Medical Sciences which also party funds AB. https://health.org.uk/https://acmedsci.ac.uk/ AB?s work was also part supported by Health Data Research UK, an initiative funded by UK Research and Innovation, Department of Health and Social Care (England) and the devolved administrations, and leading medical research charities, as well as the Maudsley Charity. https://www.ukri.org/ https://maudsleycharity.org/ Professor Louise M Howard receives salary support from NIHR South London and Maudsley/ King?s College London Biomedical Research Council and the NIHR South London Applied Research Collaboration. https://www.nihr.ac.uk/ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: {\textcopyright} 2021 Public Library of Science. All rights reserved. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = aug,

doi = "10.1371/journal.pone.0253809",

language = "English",

volume = "16",

pages = "e0253809",

journal = "PLoS ONE",

issn = "1932-6203",

publisher = "Public Library of Science",

number = "8 August",

}

TY - JOUR

T1 - Developing a Natural Language Processing tool to identify perinatal self-harm in electronic healthcare records

AU - Ayre, Karyn

AU - Bittar, André

AU - Kam, Joyce

AU - Verma, Somain

AU - Howard, Louise M

AU - Dutta, Rina

N1 - Funding Information: KA is funded by a National Institute for Health Research Doctoral Research Fellowship (NIHR-DRF-2016-09-042). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. https://www.nihr.ac.uk/ RD is funded by a Clinician Scientist Fellowship (research project e-HOST-IT) from the Health Foundation in partnership with the Academy of Medical Sciences which also party funds AB. https://health.org.uk/https://acmedsci.ac.uk/ AB?s work was also part supported by Health Data Research UK, an initiative funded by UK Research and Innovation, Department of Health and Social Care (England) and the devolved administrations, and leading medical research charities, as well as the Maudsley Charity. https://www.ukri.org/ https://maudsleycharity.org/ Professor Louise M Howard receives salary support from NIHR South London and Maudsley/ King?s College London Biomedical Research Council and the NIHR South London Applied Research Collaboration. https://www.nihr.ac.uk/ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Publisher Copyright: © 2021 Public Library of Science. All rights reserved. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/8

Y1 - 2021/8

N2 - BACKGROUND: Self-harm occurring within pregnancy and the postnatal year ("perinatal self-harm") is a clinically important yet under-researched topic. Current research likely under-estimates prevalence due to methodological limitations. Electronic healthcare records (EHRs) provide a source of clinically rich data on perinatal self-harm.AIMS: (1) To create a Natural Language Processing (NLP) tool that can, with acceptable precision and recall, identify mentions of acts of perinatal self-harm within EHRs. (2) To use this tool to identify service-users who have self-harmed perinatally, based on their EHRs.METHODS: We used the Clinical Record Interactive Search system to extract de-identified EHRs of secondary mental healthcare service-users at South London and Maudsley NHS Foundation Trust. We developed a tool that applied several layers of linguistic processing based on the spaCy NLP library for Python. We evaluated mention-level performance in the following domains: span, status, temporality and polarity. Evaluation was done against a manually coded reference standard. Mention-level performance was reported as precision, recall, F-score and Cohen's kappa for each domain. Performance was also assessed at 'service-user' level and explored whether a heuristic rule improved this. We report per-class statistics for service-user performance, as well as likelihood ratios and post-test probabilities.RESULTS: Mention-level performance: micro-averaged F-score, precision and recall for span, polarity and temporality >0.8. Kappa for status 0.68, temporality 0.62, polarity 0.91. Service-user level performance with heuristic: F-score, precision, recall of minority class 0.69, macro-averaged F-score 0.81, positive LR 9.4 (4.8-19), post-test probability 69.0% (53-82%). Considering the task difficulty, the tool performs well, although temporality was the attribute with the lowest level of annotator agreement.CONCLUSIONS: It is feasible to develop an NLP tool that identifies, with acceptable validity, mentions of perinatal self-harm within EHRs, although with limitations regarding temporality. Using a heuristic rule, it can also function at a service-user-level.

AB - BACKGROUND: Self-harm occurring within pregnancy and the postnatal year ("perinatal self-harm") is a clinically important yet under-researched topic. Current research likely under-estimates prevalence due to methodological limitations. Electronic healthcare records (EHRs) provide a source of clinically rich data on perinatal self-harm.AIMS: (1) To create a Natural Language Processing (NLP) tool that can, with acceptable precision and recall, identify mentions of acts of perinatal self-harm within EHRs. (2) To use this tool to identify service-users who have self-harmed perinatally, based on their EHRs.METHODS: We used the Clinical Record Interactive Search system to extract de-identified EHRs of secondary mental healthcare service-users at South London and Maudsley NHS Foundation Trust. We developed a tool that applied several layers of linguistic processing based on the spaCy NLP library for Python. We evaluated mention-level performance in the following domains: span, status, temporality and polarity. Evaluation was done against a manually coded reference standard. Mention-level performance was reported as precision, recall, F-score and Cohen's kappa for each domain. Performance was also assessed at 'service-user' level and explored whether a heuristic rule improved this. We report per-class statistics for service-user performance, as well as likelihood ratios and post-test probabilities.RESULTS: Mention-level performance: micro-averaged F-score, precision and recall for span, polarity and temporality >0.8. Kappa for status 0.68, temporality 0.62, polarity 0.91. Service-user level performance with heuristic: F-score, precision, recall of minority class 0.69, macro-averaged F-score 0.81, positive LR 9.4 (4.8-19), post-test probability 69.0% (53-82%). Considering the task difficulty, the tool performs well, although temporality was the attribute with the lowest level of annotator agreement.CONCLUSIONS: It is feasible to develop an NLP tool that identifies, with acceptable validity, mentions of perinatal self-harm within EHRs, although with limitations regarding temporality. Using a heuristic rule, it can also function at a service-user-level.

UR - http://www.scopus.com/inward/record.url?scp=85112016735&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0253809

DO - 10.1371/journal.pone.0253809

M3 - Article

C2 - 34347787

SN - 1932-6203

VL - 16

SP - e0253809

JO - PLoS ONE

JF - PLoS ONE

IS - 8 August

M1 - e0253809

ER -

Developing a Natural Language Processing tool to identify perinatal self-harm in electronic healthcare records

Abstract

Access to Document

Other files and links

Fingerprint

Cite this