Funnel plots for comparing provider performance based on patient-reported outcome measures.


Neuburger, J; Cromwell, DA; Hutchings, A; Black, N; van der Meulen, JH; (2011) Funnel plots for comparing provider performance based on patient-reported outcome measures. BMJ quality & safety, 20 (12). pp. 1020-6. ISSN 2044-5415 DOI: https://doi.org/10.1136/bmjqs-2011-000197

[img]
Preview
Text - Published Version
License: Creative Commons Attribution Non-commercial

Download (405kB) | Preview

Abstract

BACKGROUND: Patient-reported outcome measures (PROMs) often produce skewed distributions of individual scores after a healthcare intervention. For health performance indicators derived from skewed distributions, funnel plots designed with symmetric control limits may increase the risk of false alarms about poor performance.<br/> AIM: To investigate the accuracy of funnel plots with symmetric control limits when comparing provider performance based on PROMs.<br/> METHODS: The authors used a database containing condition-specific PROMs for 17,453 hip replacements and 7656 varicose vein procedures performed by providers in the English NHS. The mean postoperative PROM score, adjusted for patient characteristics, was used as the measure of performance. To compare performance, symmetric 99.8% control limits were calculated on funnel plots, 3 SDs away from the overall mean on either side. These were compared to control limits derived directly from percentiles of simulated (bootstrap) distributions of mean scores.<br/> RESULTS: The simulated control limits on funnel plots for both procedures were asymmetric. The empirical probability of falling outside the symmetric 99.8% 'poor performance' control limit was inflated from the stipulated rate of 0.1% to 0.2-0.3% for provider sample sizes of up to 150 procedures. The authors observed that, out of 237 providers of hip replacement, eight had adjusted mean scores that exceeded the symmetric 'poor performance' limit compared with only five that exceeded the corresponding simulated limit. In other words, three (1.3%) were differently classified. For varicose vein surgery, five out of 160 providers exceeded the symmetric limit and four exceeded the simulated limit, that is, 1 (0.6%) was differently classified.<br/> CONCLUSIONS: When designing funnel plots for comparisons of provider performance based on highly skewed data, the use of simulated control limits should be considered.<br/>

Item Type: Article
Faculty and Department: Faculty of Public Health and Policy > Dept of Health Services Research and Policy
PubMed ID: 21680560
Web of Science ID: 297619300004
URI: http://researchonline.lshtm.ac.uk/id/eprint/600

Statistics


Download activity - last 12 months
Downloads since deposit
117Downloads
361Hits
Accesses by country - last 12 months
Accesses by referrer - last 12 months
Impact and interest
Additional statistics for this record are available via IRStats2

Actions (login required)

Edit Item Edit Item