Patient samples for measuring primary care physician performance: who should be included?

BACKGROUND: In measuring patients' experiences with individual primary care physicians (PCPs), the reliability and validity of data based on samples of "established" patients of a physician's panel have been shown. However, as large-scale initiatives seek the least costly approach to obtaining these data, little is known about the implications of expanding samples to include any patient who has seen the physician in the relevant time period.
METHODS: A brief validated patient questionnaire was administered to a random sample of patients visiting 67 PCPs in California between January and October 2005. We evaluated the concordance between administrative and patient-reported information on whether the physician was the patient's PCP. Response rates, data quality, and experiences reported by confirmed "established" patients were compared with those of "unestablished" patients.
RESULTS: Administrative data designating patients as established to a PCP were highly concordant with patient self-report (96.5%). For unestablished patients, concordance was considerably lower (40.0%). Response rates among established patients were higher than those of patients believed to be unestablished (35.5% vs. 22.2%). Compared with established patients of a PCP's practice, unestablished patients reported significantly less favorable experiences with the doctor (interaction quality, P < 0.001; health promotion, P < 0.001; access, P < 0.001; integration, P < 0.05). The ranking of individual physicians differed for established and unestablished patient samples.
CONCLUSIONS: Initiatives measuring patients' experiences with individual primary care physicians will achieve different results (response rates, physician scores) if samples include any patient who has seen the physician versus those whom administrative data indicate as established members of the physician's panel.