Can natural language processing models extract and classify instances of interpersonal violence in mental healthcare electronic records: an applied evaluative study

TY - JOUR

T1 - Can natural language processing models extract and classify instances of interpersonal violence in mental healthcare electronic records: an applied evaluative study

AU - Botelle, Riley

AU - Bhavsar, Vishal

AU - Kadra-Scalzo, Giouliana

AU - Mascio, Aurelie

AU - Williams, Marcus V.

AU - Roberts, Angus

AU - Velupillai, Sumithra

AU - Stewart, Robert

N1 - Funding Information: Competing interests RS has received research support in the last 36 months from Janssen, Takeda and GSK. GK-S has received funding from Janssen and Lundbeck. SV has received funding from Janssen. AM has received funding from Takeda California. Funding Information: Funding This project received no specific funding. The Clinical Record Interactive Search (CRIS) system was funded and developed by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London and by a joint infrastructure grant from Guy’s and St Thomas’ Charity and the Maudsley Charity (grant number BRC-2011-10035). RS, SV, AR and GK-S receive salary support from the NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King's College London. RS is a NIHR Senior Investigator. GK-S has received support from the Early Career Research Award, funded by the NIHR Maudsley BRC. VB receives salary support from King’s College London via a secondment to Lambeth Council. AR receives support from Health Data Research UK, an initiative funded by UK Research and Innovation, Department of Health and Social Care (England) and the devolved administrations, and leading medical research charities. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. Publisher Copyright: ©

PY - 2022/2/16

Y1 - 2022/2/16

N2 - Objective This paper evaluates the application of a natural language processing (NLP) model for extracting clinical text referring to interpersonal violence using electronic health records (EHRs) from a large mental healthcare provider. Design A multidisciplinary team iteratively developed guidelines for annotating clinical text referring to violence. Keywords were used to generate a dataset which was annotated (ie, classified as affirmed, negated or irrelevant) for: presence of violence, patient status (ie, as perpetrator, witness and/or victim of violence) and violence type (domestic, physical and/or sexual). An NLP approach using a pretrained transformer model, BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) was fine-tuned on the annotated dataset and evaluated using 10-fold cross-validation. Setting We used the Clinical Records Interactive Search (CRIS) database, comprising over 500 000 de-identified EHRs of patients within the South London and Maudsley NHS Foundation Trust, a specialist mental healthcare provider serving an urban catchment area. Participants Searches of CRIS were carried out based on 17 predefined keywords. Randomly selected text fragments were taken from the results for each keyword, amounting to 3771 text fragments from the records of 2832 patients. Outcome measures We estimated precision, recall and F1 score for each NLP model. We examined sociodemographic and clinical variables in patients giving rise to the text data, and frequencies for each annotated violence characteristic. Results Binary classification models were developed for six labels (violence presence, perpetrator, victim, domestic, physical and sexual). Among annotations affirmed for the presence of any violence, 78% (1724) referred to physical violence, 61% (1350) referred to patients as perpetrator and 33% (731) to domestic violence. NLP models' precision ranged from 89% (perpetrator) to 98% (sexual); recall ranged from 89% (victim, perpetrator) to 97% (sexual). Conclusions State of the art NLP models can extract and classify clinical text on violence from EHRs at acceptable levels of scale, efficiency and accuracy.

AB - Objective This paper evaluates the application of a natural language processing (NLP) model for extracting clinical text referring to interpersonal violence using electronic health records (EHRs) from a large mental healthcare provider. Design A multidisciplinary team iteratively developed guidelines for annotating clinical text referring to violence. Keywords were used to generate a dataset which was annotated (ie, classified as affirmed, negated or irrelevant) for: presence of violence, patient status (ie, as perpetrator, witness and/or victim of violence) and violence type (domestic, physical and/or sexual). An NLP approach using a pretrained transformer model, BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) was fine-tuned on the annotated dataset and evaluated using 10-fold cross-validation. Setting We used the Clinical Records Interactive Search (CRIS) database, comprising over 500 000 de-identified EHRs of patients within the South London and Maudsley NHS Foundation Trust, a specialist mental healthcare provider serving an urban catchment area. Participants Searches of CRIS were carried out based on 17 predefined keywords. Randomly selected text fragments were taken from the results for each keyword, amounting to 3771 text fragments from the records of 2832 patients. Outcome measures We estimated precision, recall and F1 score for each NLP model. We examined sociodemographic and clinical variables in patients giving rise to the text data, and frequencies for each annotated violence characteristic. Results Binary classification models were developed for six labels (violence presence, perpetrator, victim, domestic, physical and sexual). Among annotations affirmed for the presence of any violence, 78% (1724) referred to physical violence, 61% (1350) referred to patients as perpetrator and 33% (731) to domestic violence. NLP models' precision ranged from 89% (perpetrator) to 98% (sexual); recall ranged from 89% (victim, perpetrator) to 97% (sexual). Conclusions State of the art NLP models can extract and classify clinical text on violence from EHRs at acceptable levels of scale, efficiency and accuracy.

UR - http://www.scopus.com/inward/record.url?scp=85124775253&partnerID=8YFLogxK

U2 - 10.1136/bmjopen-2021-052911

DO - 10.1136/bmjopen-2021-052911

M3 - Article

SN - 2044-6055

VL - 12

JO - BMJ Open

JF - BMJ Open

IS - 2

M1 - e052911

ER -