Using patient-reported outcomes (PROs) to compare the providers of surgery: does the choice of measure matter?

BACKGROUND: Patient-reported outcomes (PROs) are being used to compare health care providers with little knowledge of how the choice of measure affects such comparisons. OBJECTIVES: To assess how much difference the choice of PRO makes to a provider's adjusted outcome and whether the choice affects a provider's rating. RESEARCH DESIGN: PROs collected in England from patients undergoing: hip replacement (243 providers; 52,692 patients); knee replacement (244; 60,118); varicose vein surgery (100; 11,163); and groin hernia repair (201; 31,714). Four case-mix-adjusted outcomes (mean postoperative disease-specific and generic PRO; proportion achieving a minimally important difference in disease-specific PRO; proportion reporting improvement on single transitional item). We calculated the associations between measures and for each measure, the proportion of providers rated as statistically above or below average and the level of agreement in ratings. RESULTS: For major surgery, disease-specific PROs were strongly correlated with the generic PRO (hip 0.90; knee 0.88), they rated high proportions of providers as above or below average (hip 25.1%; knee 19.3%) and there was agreement in ratings with the generic PRO. Even so, for a large proportion of providers (hip 30%; knee 16%) their rating depended on the choice of measure. For minor surgery, correlations between measures were mostly weak. The single transitional item identified the most outliers (varicose vein 20%, hernia 10%). CONCLUSIONS: Choice of outcome measure can determine a provider's rating. Measure selection depends on whether the priority is to avoid missing "poor" providers or avoid mislabeling average providers as "poor."