R Sorokin1. 1. Jefferson Medical College, Philadelphia, Penn., USA. rachel.sorokin@mail.tju.edu
Abstract
CONTEXT: Many managed care organizations grade physician groups with "report cards" developed from administrative data sets and chart reviews. OBJECTIVE: To investigate the accuracy of five report cards on a single group practice. DESIGN: Determination of report card accuracy by using the practice capitation list and a review of the patients' medical records. SETTING: Academic practice in Philadelphia, Pennsylvania (19 physicians), evaluated with five report cards by two capitated health plans between 1994 and 1997. RESULTS: Four major problems were uncovered. First, four of the five report cards included patients who were enrolled in our practice for only a portion of the reporting year (for the four report cards, the proportion of partial-year enrollees was 8%, 15%, 23%, and 100%). Second, there was a considerable number of false-positive diagnoses in the administrative algorithms. Eight of the 61 patients labeled with hypertension did not have this condition (error rate, 14%). Other error rates were 44% for coronary artery disease, 50% for congestive heart failure, 33% for atrial fibrillation, and 0% for diabetes. Third, the administrative data often failed to capture laboratory data. Laboratory performance measures for patients with diabetes (hemoglobin A1c and cholesterol measurement and screening for microalbuminuria) were 3 to 10 times higher when assessed by chart review. Finally, the uniformly small sample sizes used in the report cards make the estimates of performance imprecise. No report card reported 95% CIs. CONCLUSION: Five report cards on a group practice contained methodologic problems that led to systematic underestimation of the practice's performance. Larger surveys are needed to determine the accuracy of report cards in current use.
CONTEXT: Many managed care organizations grade physician groups with "report cards" developed from administrative data sets and chart reviews. OBJECTIVE: To investigate the accuracy of five report cards on a single group practice. DESIGN: Determination of report card accuracy by using the practice capitation list and a review of the patients' medical records. SETTING: Academic practice in Philadelphia, Pennsylvania (19 physicians), evaluated with five report cards by two capitated health plans between 1994 and 1997. RESULTS: Four major problems were uncovered. First, four of the five report cards included patients who were enrolled in our practice for only a portion of the reporting year (for the four report cards, the proportion of partial-year enrollees was 8%, 15%, 23%, and 100%). Second, there was a considerable number of false-positive diagnoses in the administrative algorithms. Eight of the 61 patients labeled with hypertension did not have this condition (error rate, 14%). Other error rates were 44% for coronary artery disease, 50% for congestive heart failure, 33% for atrial fibrillation, and 0% for diabetes. Third, the administrative data often failed to capture laboratory data. Laboratory performance measures for patients with diabetes (hemoglobin A1c and cholesterol measurement and screening for microalbuminuria) were 3 to 10 times higher when assessed by chart review. Finally, the uniformly small sample sizes used in the report cards make the estimates of performance imprecise. No report card reported 95% CIs. CONCLUSION: Five report cards on a group practice contained methodologic problems that led to systematic underestimation of the practice's performance. Larger surveys are needed to determine the accuracy of report cards in current use.
Authors: Hyungjin Myra Kim; Eric G Smith; Claire M Stano; Dara Ganoczy; Kara Zivin; Heather Walters; Marcia Valenstein Journal: BMC Health Serv Res Date: 2012-01-23 Impact factor: 2.655