TY - JOUR
T1 - In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving
AU - Marshall, Iain J.
AU - Trikalinos, Thomas A.
AU - Soboczenski, Frank
AU - Yun, Hye Sun
AU - Kell, Gregory
AU - Marshall, Rachel
AU - Wallace, Byron C.
N1 - Funding Information:
Funding Statement: This work has been supported by the National Institutes of Health under the National Library of Medicine grant R01-LM012086 and by the National Science Foundation under grant 1750978 : “CAREER: Structured Scientific Evidence Extraction: Models and Corpora.” The work has also been partially supported by the UK Medical Research Council , through its Skills Development Fellowship program, fellowship MR/N015185/1 .
Publisher Copyright:
© 2022 The Author(s)
PY - 2023/1
Y1 - 2023/1
N2 - Objectives: The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise. Study Design and Setting: We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team. Results: The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation. Conclusion: Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research.
AB - Objectives: The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise. Study Design and Setting: We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team. Results: The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation. Conclusion: Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research.
KW - Artificial intelligence
KW - Evidence based medicine
KW - Living systematic reviews
KW - Machine learning
KW - Natural language processing
KW - Systematic reviews
UR - http://www.scopus.com/inward/record.url?scp=85143864329&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2022.08.013
DO - 10.1016/j.jclinepi.2022.08.013
M3 - Article
C2 - 36150548
AN - SCOPUS:85143864329
SN - 0895-4356
VL - 153
SP - 26
EP - 33
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -