Intelligent Monitoring? Assessing the ability of the Care Quality Commission's statistical surveillance tool to predict quality and prioritise NHS hospital inspections

Alex Griffiths*, Anne Laure Beaussier, David Demeritt, Henry Rothstein

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

30 Citations (Scopus)
212 Downloads (Pure)

Abstract

Background The Care Quality Commission (CQC) is responsible for ensuring the quality of the health and social care delivered by more than 30000 registered providers in England. With only limited resources for conducting on-site inspections, the CQC has used statistical surveillance tools to help it identify which providers it should prioritise for inspection. In the face of planned funding cuts, the CQC plans to put more reliance on statistical surveillance tools to assess risks to quality and prioritise inspections accordingly. Objective To evaluate the ability of the CQC's latest surveillance tool, Intelligent Monitoring (IM), to predict the quality of care provided by National Health Service (NHS) hospital trusts so that those at greatest risk of providing poor-quality care can be identified and targeted for inspection. Methods The predictive ability of the IM tool is evaluated through regression analyses and Ï ‡ 2 testing of the relationship between the quantitative risk score generated by the IM tool and the subsequent quality rating awarded following detailed on-site inspection by large expert teams of inspectors. Results First, the continuous risk scores generated by the CQC's IM statistical surveillance tool cannot predict inspection-based quality ratings of NHS hospital trusts (OR 0.38 (0.14 to 1.05) for Outstanding/Good, OR 0.94 (0.80 to -1.10) for Good/Requires improvement, and OR 0.90 (0.76 to 1.07) for Requires improvement/Inadequate). Second, the risk scores cannot be used more simply to distinguish the trusts performing poorly - those subsequently rated either 'Requires improvement' or 'Inadequate' - from the trusts performing well - those subsequently rated either 'Good' or 'Outstanding' (OR 1.07 (0.91 to 1.26)). Classifying CQC's risk bandings 1-3 as high risk and 4-6 as low risk, 11 of the high risk trusts were performing well and 43 of the low risk trusts were performing poorly, resulting in an overall accuracy rate of 47.6%. Third, the risk scores cannot be used even more simply to distinguish the worst performing trusts - those subsequently rated 'Inadequate' - from the remaining, better performing trusts (OR 1.11 (0.94 to 1.32)). Classifying CQC's risk banding 1 as high risk and 2-6 as low risk, the highest overall accuracy rate of 72.8% was achieved, but still only 6 of the 13 Inadequate trusts were correctly classified as being high risk. Conclusions Since the IM statistical surveillance tool cannot predict the outcome of NHS hospital trust inspections, it cannot be used for prioritisation. A new approach to inspection planning is therefore required.

Original languageEnglish
Pages (from-to)120-130
Number of pages11
JournalBMJ Quality and Safety
Volume26
Issue number2
Early online date18 Jan 2017
DOIs
Publication statusPublished - 1 Feb 2017

Keywords

  • Health policy
  • Performance measures
  • Quality improvement methodologies
  • Quality measurement
  • Risk management

Fingerprint

Dive into the research topics of 'Intelligent Monitoring? Assessing the ability of the Care Quality Commission's statistical surveillance tool to predict quality and prioritise NHS hospital inspections'. Together they form a unique fingerprint.

Cite this