Addressing the “elephant in the room” of AI clinical decision support through organisation-level regulation

Joe Zhang, Heather Mattie, Haris Shuaib, Tamishta Hensman, James T. Teo, Leo Anthony Celi

Research output: Contribution to journalArticlepeer-review


Consider the following proprietary artificial intelligence (AI) algorithm products: (1) continual monitoring to predict likelihood of acute kidney injury (Dascena Previse, Dascena, USA); (2) predicting significant events for patients on intensive care (CLEWICU, CLEW Medical, Israel); (3) an early warning system for acute inpatient deterioration (Wave Clinical Platform, Excel Medical, USA); and (4) using electronic health record (EHR) data to predict likelihood of sepsis (Epic Sepsis Model, Epic Systems Corporation, USA).

These algorithms provide early signals of potentially treatable events using real-time clinical data. However, the first three are considered software as a medical device (SaMD) under oversight of the US Food & Drug Administration (FDA) [1–3]. In contrast, the last has undergone no visible regulatory scrutiny [4] and demonstrates minimal data or algorithmic transparency [5], yet is actively used in hundreds of hospitals in the United States that employ the Epic EHR [6]. In 2021, an independent evaluation of this sepsis model demonstrated poor performance (relative to vendor reported metrics), failing to identify 67% of patients with sepsis, with a positive predictive value of 12% and substantial alert burden for clinicians [7]. Other technology vendors [8–10] and healthcare providers [11,12], are also known for hosting development and operationalisation of proprietary algorithmic clinical decision support (CDS). It is likely that many AI implementations fly under the radar.
Original languageEnglish
Pages (from-to)1-5
Number of pages5
JournalPLOS Digital Health
Issue number9
Publication statusPublished - 15 Sept 2022


Dive into the research topics of 'Addressing the “elephant in the room” of AI clinical decision support through organisation-level regulation'. Together they form a unique fingerprint.

Cite this