TY - JOUR
T1 - Generating (Factual?) Narrative Summaries of RCTs
T2 - Experiments with Neural Multi-Document Summarization
AU - Wallace, Byron C.
AU - Saha, Sayantan
AU - Soboczenski, Frank
AU - Marshall, Iain J.
N1 - Publisher Copyright:
©2021 AMIA - All rights reserved.
Copyright:
This record is sourced from MEDLINE/PubMed, a database of the U.S. National Library of Medicine
PY - 2021
Y1 - 2021
N2 - We consider the problem of automatically generating a narrative biomedical evidence summary from multiple trial reports. We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews previously conducted by members of the Cochrane collaboration, using the authors conclusions section of the review abstract as our target. We enlist medical professionals to evaluate generated summaries, and we find that summarization systems yield consistently fluent and relevant synopses, but these often contain factual inaccuracies. We propose new approaches that capitalize on domain-specific models to inform summarization, e.g., by explicitly demarcating snippets of inputs that convey key findings, and emphasizing the reports of large and high-quality trials. We find that these strategies modestly improve the factual accuracy of generated summaries. Finally, we propose a new method for automatically evaluating the factuality of generated narrative evidence syntheses using models that infer the directionality of reported findings.
AB - We consider the problem of automatically generating a narrative biomedical evidence summary from multiple trial reports. We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews previously conducted by members of the Cochrane collaboration, using the authors conclusions section of the review abstract as our target. We enlist medical professionals to evaluate generated summaries, and we find that summarization systems yield consistently fluent and relevant synopses, but these often contain factual inaccuracies. We propose new approaches that capitalize on domain-specific models to inform summarization, e.g., by explicitly demarcating snippets of inputs that convey key findings, and emphasizing the reports of large and high-quality trials. We find that these strategies modestly improve the factual accuracy of generated summaries. Finally, we propose a new method for automatically evaluating the factuality of generated narrative evidence syntheses using models that infer the directionality of reported findings.
UR - http://www.scopus.com/inward/record.url?scp=85115280188&partnerID=8YFLogxK
M3 - Article
C2 - 34457176
AN - SCOPUS:85115280188
SN - 1942-597X
VL - 2021
SP - 605
EP - 614
JO - AMIA ... Annual Symposium proceedings. AMIA Symposium
JF - AMIA ... Annual Symposium proceedings. AMIA Symposium
ER -