What is the probability of detecting poorly performing hospitals using funnel plots?

Recent high profile cases in the UK have highlighted the impact that the reporting of clinical outcomes can have for healthcare providers and patients. Great importance has been placed on the use of statistical methods to identify healthcare providers with observed poor performance. Such providers are highlighted as potential outliers and the possible causes investigated. It is crucial, therefore, that the methods used for identifying outliers are correctly understood and interpreted. While patients, funders, managers and clinical teams really want to know the true underlying performance of the provider, this true performance generally cannot be known directly and must be estimated using observed outcomes. However, differences between the true and observed performance are likely to arise due to chance variation. Clinical outcomes are often reported through the standardised mortality ratio (SMR), displayed using funnel plots. Providers whose observed SMR falls outside the funnel plot control limits will be identified as potential outliers. However, while it is obviously desirable that a healthcare provider with a true underlying performance different from that expected should be identified, the actual probability that it will be identified from observed SMRs has not previously been described. Here we show that funnel plots for the SMR should be used with caution when the expected number of events is small as the probability of identifying providers with true poor performance is likely to be small. On the other hand, when the expected number of events is large, care must be taken as a provider may be identified as an outlier even when its divergence is of little or no clinical importance.