Assessing progress toward becoming a patient-centered medical home: an assessment tool for practice transformation.

OBJECTIVE: To describe the properties of the Patient-Centered Medical Home Assessment (PCMH-A) as a tool to stimulate and monitor progress among primary care practices interested in transforming to patient-centered medical homes (PCMHs). STUDY
SETTING: Sixty-five safety net practices from five states participating in a national demonstration program for PCMH transformation. STUDY
DESIGN: Longitudinal analyses of PCMH-A scores were performed. Scores were reviewed for agreement and sites were categorized over time into one of five categories by external facilitators. Comparisons to key activity completion rates and NCQA PCMH recognition status were completed. DATA COLLECTION/EXTRACTION
METHODS: Multidisciplinary teams at each practice completed the 33-item self-assessment tool every 6 months between March 2010 and September 2012. PRINCIPAL
FINDINGS: Mean overall PCMH-A scores increased (7.2, March 2010, to 9.1, September 2012; [p < .01]). Increases were statistically significant for each of the change concepts (p < .05). Facilitators agreed with scores 82% of the time. NCQA-recognized sites had higher PCMH-A scores than sites that were not yet recognized. Sites that completed more transformation activities and progressed over defined tiers reported higher PCMH-A scores. Scores improved most in areas where technical assistance was provided.
CONCLUSIONS: The PCMH-A was sensitive to change over time and provided an accurate reflection of practice transformation.

The patient-centered medical home (PCMH) concept has garnered considerable attention from policy makers, purchasers, and providers, particularly since the passage of the Affordable Care Act. Forty-one state Medicaid programs have adopted policies or programs to support or encourage medical home implementation (National Academy for State Health Policy 2012), and several national demonstration projects sponsored by Centers for Medicare and Medicaid Services emphasize the PCMH model (Centers for Medicaid and Medicare Innovation 2013). Furthermore, a broad range of private sector PCMH-related reimbursement programs have been initiated by health plans and purchasers (Patient-Centered Primary Care Collaborative 2012).

The Safety Net Medical Home Initiative (SNMHI), a 5-year demonstration project sponsored by The Commonwealth Fund and led by Qualis Health and the MacColl Center for Health Care Innovation at the Group Health Research Institute, aimed to assist 65 safety net clinics in five states to accelerate PCMH transformation. To guide practices as they worked to become a medical home, we needed to define the specific practice characteristics and behaviors that comprise a PCMH. From literature review, we developed a framework consisting of eight domains or change concepts. We then convened an expert panel, which confirmed the eight change concepts and helped identify 3–5 more specific key changes within each change concept (Wagner et al. 2012). To support technical assistance provided to practices participating in the SNMHI, we sought to create a self-assessment tool that would help to target needed improvement activities and technical assistance, and to monitor progress of practice transformation.

Several instruments have been developed to assist in research or evaluation of PCMH demonstration projects, to measure the extent to which practices meet accreditation or recognition requirements, or to conform to various definitions of a PCMH (National Committee for Quality Assurance; Cooley et al. 2003; Jaén et al. 2010; Birnberg et al. 2011). Their characteristics have been compared previously (Birnberg et al. 2011). We found that existing PCMH recognition instruments and evaluation tools were not sufficiently aligned with the 8 change concepts and 33 key changes in our PCMH framework. Also, some assessments rely on yes/no answers or require documentation, which limits their ability to guide or monitor real-time progress in implementing those activities. System changes such as altering appointment systems to increase continuity of care or involving nonprovider team members in patient care are implemented incrementally. We sought a self-assessment that would describe the trajectory to full implementation. The Patient-Centered Medical Home Assessment (PCMH-A) was developed for the dual purposes of assisting practices in characterizing their progress and identifying opportunities to improve implementation of the changes.

In this study, we describe the PCMH-A and observations about its face and construct validity and responsiveness to change derived from semiannual administrations of the survey among 65 safety net sites over a -year period. To assess construct validity, we compare PCMH-A self-assessment scores overall and/or by change concept with four other indicators of progress toward becoming a PCMH: medical home facilitator (practice coach) agreement with a practice's PCMH-A self-assessment score, a structured assessment of a practice's transformation capacity and progress conducted by the medical home facilitator, a Key Activities Checklist completed by the practice, and a practice's NCQA Physician Practice Connection®—Patient Centered Medical Home™ recognition status.

Methods

Setting

The SNMHI was a 5-year (2008 through 2013) effort sponsored by The Commonwealth Fund designed to develop and test a replicable model for supporting acceleration of PCMH transformation among 65 safety net practices in five states. In the first year of the initiative, we reviewed various definitions of primary care and PCMHs and ongoing primary care transformation projects, and identified eight domains in which practices would need to implement changes to become medical homes. We then developed eight change concepts—general, high-level ideas for changing a process—corresponding to each of the eight domains, and suggested three to five more specific and actionable key changes associated with each change concept. The change concepts were evaluated and their face validity confirmed by a panel of national experts. The development process and rationale for the eight change concepts for practice transformation are described in detail elsewhere (Wagner et al. 2012). The change concepts and key changes are listed in Table 1.

Table 1

Change Concepts for Practice Transformation developed for the Safety Net Medical Home Initiative

Change Concept	Key Changes
Empanelment	• Assign all patients to a provider panel and confirm assignments with providers and patients; review and update panel assignments on a regular basis.
	• Assess practice supply and demand, and balance patient load accordingly.
	• Use panel data and registries to proactively contact, educate, and track patients by disease status, risk status, self-management status, community and family need.
Continuous and team-based healing relationships	• Establish and provide organizational support for care delivery teams accountable for the patient population/panel.
	• Link patients to a provider and care team so both patients and provider/care team recognize each other as partners in care.
	• Assure that patients are able to see their provider or care team whenever possible.
	• Define roles and distribute tasks among care team members to reflect the skills, abilities, and credentials of team members.
Patient-centered interactions	• Respect patient and family values and expressed needs.
	• Encourage patients to expand their role in decision making, health-related behaviors, and self-management.
	• Communicate with their patients in a culturally appropriate manner, in a language and at a level that the patient understands.
	• Provide self-management support at every visit through goal setting and action planning.
	• Obtain feedback from patients/family about their health care experience and use this information for quality improvement.
Engaged leadership	• Provide visible and sustained leadership to lead overall culture change as well as specific strategies to improve quality and spread and sustain change.
	• Ensure that the PCMH transformation effort has the time and resources needed to be successful.
	• Ensure that providers and other care team members have protected time to conduct activities beyond direct patient care that are consistent with the medical home model.
	• Build the practice's values on creating a medical home for patients into staff hiring and training processes.
Quality improvement strategy	• Choose and use a formal model for quality improvement.
	• Establish and monitor metrics to evaluateimprovement efforts and outcome; ensure all staff members understand the metrics for success.
	• Ensure that patients, families, providers, and care team members are involved in quality improvement activities.
	• Optimize use of health information technology to meet meaningful use criteria.
Enhanced access	• Promote and expand access by ensuring that established patients have 24/7 continuous access to their care team via phone, e-mail, or in-person visits.
	• Provide scheduling options that are patient and family centered and accessible to all patients.
	• Help patients attain and understand health insurance coverage.
Care coordination	• Link patients with community resources to facilitate referrals and respond to social service needs.
	• Integrate behavioral health and specialty care into care delivery through colocation or referral protocols.
	• Track and support patients when they obtain services outside the practice.
	• Follow-up with patients within a few days of an emergency room visit or hospital discharge.
	• Communicate test results and care plans to patients/families.
Organized, evidence-based care	• Use planned care according to patient need.
	• Identify high-risk patients and ensure that they are receiving appropriate care and case management services.
	• Use point-of-care reminders based on clinical guidelines.
	• Enable planned interactions with patients by making up-to-date information available to providers and the care team at the time of the visit.

Through a request for proposal process, we identified entities in five states, each of which served as a convening organization for 10–15 safety net practices with a defined geographic region. We required that each organization, referred to as a Regional Coordinating Center, identify one or more medical home facilitators (MHFs) who would serve as practice coaches responsible for coordinating and providing technical assistance to participating practices. Practice coaches were expected to have a firm grounding in quality improvement methodologies and change management, and to have familiarity with medical home principles. The team from Qualis Health and the MacColl Center was responsible for administering the initiative, for designing and implementing a technical assistance framework, and for supporting and coaching the MHFs who would be the primary source of direct technical assistance to the practices over a 4-year implementation period. In addition to on-site coaching provided by the MHFs, the regional and national project team also provided an electronic social networking platform, organized field trips and site visits, and conducted regional and national learning sessions to facilitate peer-to-peer learning for practice staff.

PCMH Assessment Tool Development and Administration

We modeled the new instrument on the Assessment of Chronic Illness Care (ACIC), a validated tool developed by the MacColl Center to measure the extent of implementation of the Chronic Care Model (CCM) (Bonomi et al. 2002). Both instruments directly relate to an implementation model—the CCM for the ACIC, and the SNMHI change concepts for practice transformation and 33 key areas for the PCMH-A (Safety Net Medical Home Initiative Patient-Centered Medical Home Assessment 2010, 2013). Both instruments ask respondents to locate their practice on a continuum ranging from no or limited implementation of a recommended practice change to full implementation. The continuum includes four text descriptions of extent of implementation and a numeric scale. The PCMH-A asks teams to rate their care delivery on 33 key areas associated with the eight change concepts. Each item is scored from 1 to 12 based on the extent to which recommended practices and behaviors are implemented at a site. An example of one of the 33 items is shown in Figure 1. The individual items under each change concept are aggregated to compute a subscale score reflecting the level of implementation of each change concept, and change concept scores were averaged to produce an overall PCMH implementation score.

Figure 1

Excerpts from the Patient-Centered Medical Home Assessment (PCMH-A) and the Key Activities Checklist Full versions of both PCMH-A and the Key Activities Checklist are available at: http://www.safetynetmedicalhome.org/resources-tools/assessment

We recommended that practice sites complete the assessment by incorporating the input of a multidisciplinary team, including clinical, administrative, clerical, and other staff, and discouraged limiting participation in the process to practice management and leadership. Some practices completed the survey by consensus in a group setting, and others asked multiple individuals to complete the survey independently and then met to reach consensus on final results. In the SNMHI, the 65 practices were introduced to the change concepts and key changes in the summer of 2009 and sites first completed the PCMH-A in March 2010. To monitor progress and identify opportunities for improvement, sites repeated the PCMH-A every 6 months until September 2012. Selected characteristics of the 65 sites are shown in Table 2.

Table 2

Characteristics of SNMHI Participating Sites (N = 65)

	Mean	Range
No. of clinics per region	13	10–15
Total annual patient visits	391,000	81,000–1,012,000
Patients with at least one visit in the past year	107,000	25,000–225,000
Percent Medicaid/public payer	43%	28–52%
Percent uninsured/self-pay	24%	11–37%
Clinics with EHR	80%	54–100%

Independent External Observer Evaluations

Each of the five SNMHI Regional Coordinating Centers employed one or more practice coaches referred to as MHFs assigned to help practices undertake PCMH changes. Although the roles, activities, and time spent on-site varied among the five regions, at a minimum each MHF had the opportunity to interact regularly with practice staff, review reports of activities and achievements, and observe interactions among practice staff (Petersen, Taylor, and Geonnotti 2013). Two MHF evaluative activities provide independent perspectives on the transformation efforts and activities in each site: MHF agreement with the site's PCMH-A scores, and MHF assessment of the extent to which practices had the capacity to transform and were making meaningful changes to practice systems—the Tiering tool.

The MHFs working with each site reviewed the eight PCMH-A change concept scores for each of their sites at each administration and indicated whether they agreed or disagreed with the practice's self-assessment score based on their observations. We describe the agreement of MHFs with practice self-assessments for September 2012. When MHFs disagreed with a subscale score, they were asked to indicate whether the score should be lower or higher, but not the magnitude of their disagreements with scores.

The MHF Tiering Tool

In addition, the MHFs evaluated each of their clinics' transformation efforts using a five-category rating instrument developed for the project. The “Tiering Tool” instrument asked the MHFs to assess the extent to which sites were building the capacity to make sustainable changes to the practice, and actually testing and implementing changes. The assessment includes an evaluation of leadership involvement, use of data to support change, extent of changes being made, and other characteristics to place each site in one of five tiers ranging from little transformation activity (1) to managing and sustaining multidimensional change (5). MHFs completed the Tiering tool on five occasions between June 2011 and October 2012.

Key Activities Checklist

The PCMH-A is intended to measure the extent to which the key changes under each change concept have been implemented in practice. However, it does not measure the specific activities undertaken by the practice in their efforts to make a change. On the basis of experience from previous quality improvement work (Pearson et al. 2005), practice reports, and recommendations of the MHFs, we compiled a list of specific activities that support progress in the key changes. The list was converted into an instrument called the Key Activities Checklist (Safety Net Medical Home Initiative Key Activities Checklist Tool 2013). For each key change, the instrument lists several general activities that could be conducted in order to implement the change, and then provides specific examples of more discrete activities that practices have tested. For instance, for the key change “Provide self-management support at every visit through goal setting and action planning,” one general activity is “Use validated communication strategies for goal setting with patients.” Examples of the more discrete component activities include ideas such as “Train appropriate staff on how to initiate self-management goal conversations with patients” and “Train staff and providers in the use of motivational interviewing techniques.” Figure 1 provides a visual example of a general activity, associated discrete component activities, and scoring options for the key change: “Provide self-management support at every visit through goal setting and action planning.”

Practice teams were asked to report the extent to which the site had implemented each activity using the following scale: not presently planning to test the change at this time (1); planning (but not yet begun) to test (2); testing or implementing the change (3); or completed implementation and maintaining the change (4); and the proportion of activities reported as being tested, implemented, or completed and being maintained was calculated for each site for each change concept. Participating sites submitted the Key Activities Checklist in April 2011, October 2011, April 2012, and October 2012. The corresponding response rates for these submissions were 100, 100, 95, and 98 percent. We examined the association between the proportion of activities that practices had acted upon (i.e., tested, implemented, or completed, and were maintaining) and PCMH-A scores from the corresponding time period for each change concept. We posited a positive relationship between the proportion of activities acted upon and improvements in the related PCMH-A subscale scores.

NCQA PCMH Recognition Status

SNMHI practices were encouraged, although not required, to pursue NCQA PPC®-PCMH™ recognition, and the initiative provided both financial and technical assistance to all those interested in seeking recognition. The standards that must be met to achieve NCQA® PCMH™ recognition correspond reasonably well with full implementation of the eight change concepts as described in the PCMH-A. We examined PCMH-A scores from March 2010 to September 2012 for practices receiving NCQA PPC-PCMH recognition before October 1, 2011 (approximately one quarter of the practices) to scores for practices pursuing recognition (as reflected by having purchased an application license as of the same date) or not pursuing NCQA PCMH recognition.

Analysis

A multivariate linear regression model with the average PCMH-A scores for each change concept as the dependent variables and the six time points as the independent variable was used to estimate the trend of PCMH-A scores over time from first to last administration. F-tests from the multivariate linear regression model were used to evaluate the statistical significance of changes in the PCMH-A scores over time.

To assess the relationships between key activities and PCMH-A scores by change concept, we stratified the intensity of implementation of key activities from the Key Activity Checklist into three categories: high intensity (67–100 percent of key activities underway), moderate intensity (34–66 percent), and low intensity (0–33 percent). We then conducted individual simple linear regression analysis using PCMH-A scores for each change concept as the dependent variable, and time and intensity of key activity implementation as the independent variables. We also used linear regression analyses to assess the strength of the associations between overall PCMH-A scores and both the MHF tiering assessments and Key Activities Checklist scores at four time points.

All analyses were conducted using SAS software, version 9.2 of the SAS System for Windows (copyright © 2002–2008; SAS Institute, Inc., Cary, NC, USA), with hypothesis testing performed using a two-sided alpha level set at .05.

Results

All 65 participating sites (100%) submitted complete PCMH-As for the first five time periods, and all but one (n = 64) submitted data for the sixth time period.

PCMH-A scores by change concept for each of the six time periods are shown in Figure 2. In aggregate, mean overall scores increased during each 6-month interval. The overall mean score increased from 7.2 in March 2010 to 9.1 in September 2012 (p < .0001). With two exceptions in March 2011 (Engaged Leadership and Quality Improvement Strategy), PCMH-A scores increased for each of the change concepts with each successive administration.

Figure 2

Mean Change Concept Scores, Safety Net Medical Home Initiative Sites, March 2010 through September 2012. (Numbers in boxes refer to increases in score from March 2010 to September 2012 administration)

The mean change concept scores at baseline ranged from a low of 6.1 for empanelment to a high of 8.0 for enhanced access. Increases in PCMH-A scores from March 2010 to September 2012 were statistically significant for each of the change concepts (p < .0001). The greatest increases in score were seen in empanelment, continuous and team-based healing relationships, and organized and evidence-based care. Empanelment and continuous and team-based healing relationships were the first two change concepts that were addressed in the initiative's technical assistance and coaching activities, which established a foundation for improvements in organized, evidence-based care.

Despite systematic and steady increases in PCMH-A scores across all 65 sites, substantial variation was observed at the individual practice level (data not shown). More commonly at the individual practice level increased scores were seen for several change concepts with stable or occasionally decreasing scores reported for other change concepts. Even among different practice sites affiliated with the same organization, there was considerable variability in magnitude and patterns of change from one time period to another.

Independent External Observer Agreement with PCMH-A Scores

Figure 3 shows substantial agreement between facilitators' assessment of practice progress and the sites' PCMH-A scores (range 73–86 percent). When MHFs disagreed with the scores they were more likely to report that they believed that practices had overestimated their accomplishments and scores should have been lower (mean 12 percent, range 6–22 percent) than they were to report that practice scores did not fully reflect their achievements (mean 6 percent, range 2–11 percent).

Figure 3

Medical Home Facilitator Agreement with PCMH-A Self-Assessment Scores, by Change Concept, September 2012

Medical Home Facilitator Tiering Instrument

Table 3 shows that mean tiering scores across all sites significantly increased over time along with mean PCMH-A scores. We studied the relationship between the assigned tier (1–5) and PCMH-A scores across all sites over the four time points using linear regression analysis. For every 1 point increase in tiering level, we found a .33 increase in PCMH-A score (t = 6.39, p < .001).

Table 3

Mean Values across all Sites for Overall PCMH-A Score, Tiering Level, and Percent of Key Activities Underway

Date of Submission	Mean Tiering Level† (N = 64)	Mean Percent of Key Activities Underway‡ (N = 64) (%)	Mean PCMH-A Score (N = 64)
March 2011	3.19	63.6	8.1
September 2011	3.33	72.7	8.4
March 2012	3.56	81.4	8.7
September 2012	3.81	85.5	9.1

Mean tiering scores increase over time (t = 3.10, p = .0021).

Mean key activities increase over time (t = 8.29, p < .0001).

The overall percentage of key activities being acted upon also significantly increased over time (Table 3). For each change concept, and for the aggregate, increasing implementation of the activities reflected on the Key Activities Checklist was associated with higher PCMH-A scores over time (p < .0001 for each change concept, and for overall scores). The difference in overall scores between the low- and high-activity groups early in the initiative ranged from 1.4 to 2.4 points. The interaction between the level of key activity implementation and time was not significant for any change concept score, or for overall scores. Linear regression analysis of the relationship between the percentage of key activities underway and PCMH-A score found that PCMH-A scores increased 0.32 for every 10 percent increase in key activities underway (t = 9.52, p < .0001).

Approximately one quarter (15 of 65; 23 percent) of participating practices achieved NCQA PCMH recognition by October 2011. The initial mean overall PCMH-A score for the 15 practices that achieved NCQA PCMH recognition was 1 point higher (7.9 vs 6.9) than that of practices pursuing NCQA PCMH recognition as of October 2011 (n = 29), and 0.7 points higher (7.9 vs 7.2) than the mean score of practices not pursuing NCQA recognition (n = 21). Over time the overall mean PCMH-A scores for each of these three recognition categories continued to increase similarly.

Discussion

A number of assessment tools exist to measure the presence of characteristics associated with the PCMH model of primary care, particularly for the purposes of research evaluations or accountability (e.g., PCMH recognition tools), rather than as a means to support and monitor quality improvement efforts. Other instruments have primarily been used to report point in time (Cooley et al. 2003, 2009; Birnberg et al. 2011) or before-and-after results for projects conducted over a period of years (Nutting et al. 2010a,b). We are not aware of published studies showing that other PCMH assessment instruments are sensitive to practice transformation efforts over short periods of time.

We found that the PCMH-A when administered longitudinally over a 2.5-year period appeared to be sensitive to practice change over a time period as short as 6 months. Several factors suggest that the self-assessment scores provide valid reflection of the extent to which specific characteristics of PCMH transformation are present in a practice. For example, over the course of the SNMHI, the magnitude of increase in PCMH-A scores was greatest for empanelment and continuous and team-based healing relationships—the change concepts that were the longest focus of technical assistance efforts. Also, we found consistently positive relationships between the quality improvement activities underway and PCMH-A change concept scores at each time point. These findings suggest that the instrument was sensitive to changes in specific components of the PCMH model, rather than simply reflecting global progress in practice redesign that might be associated with less systematic approaches to practice redesign.

The instrument as a whole appears to validly reflect progress toward becoming a PCMH. First, we found that observers external to the practice generally agreed with PCMH-A scores derived from self-assessment by practice teams. Second, we found a positive association between structured, independent assessments of practice transformation capacity and efforts completed by the same external observers and final PCMH-A self-assessment scores at the site level. Third, the percentage of quality improvement activities underway reported by the practice was also positively associated with the site's final overall PCMH-A score. Finally, practices that received NCQA PCMH recognition early in the SNMHI had initial overall PCMH-A scores that were nearly 1 point higher than the other sites.

This work is subject to several limitations. First, the PCMH-A consists of a relatively small (33) number of actionable items that collectively indicate a practice's progress toward PCMH. Although we organized the 33 items under eight change concepts, the instrument is not intended to provide comprehensive or valid measures of such complex constructs as team-based care or care coordination. Second, our construct validation analyses rely on comparisons of a practice's PCMH-A self-assessment score with other assessments done by either a MHF working with the practice or by the practice team itself. Both the MHF and the practice team have a stake in the transformation success of the practice that may well bias their perceptions. However, it is worth recalling that we developed the PCMH-A to help practice teams and those supporting them measure progress and identify opportunities for further action.

In addition, we did not formally evaluate psychometric properties of the PCMH-A such as interrater reliability. Furthermore, it is likely that turnover in clinic staff and other factors resulted in different individuals contributing to PCMH-A scoring throughout the course of the initiative. However, a major purpose of the tool was to generate discussion among practice staff, many of the questions require subjective judgment, and we did not design the instrument to serve primarily as a research tool. For it to be used as a research tool, one would want to compare self-assessment scores with PCMH-A scores generated by an independent, objective evaluator.

Although the PCMH-A scores, and their association with implementation of specific changes, reflect changes in practice design, we did not assess associations between PCMH-A scores and measures of clinical quality, patient and staff satisfaction, or financial efficiency of practices. While practices collected data on these characteristics for the purposes of monitoring quality improvement efforts, there was significant variability in the measures and their implementation among the 65 sites. Because the SNMHI will be the subject of an independent before-and-after evaluation conducted by researchers at the University of Chicago, it may be possible to examine such associations after the conclusion of the SNMHI in mid-2013.

The SNMHI included safety net sites, predominantly federally qualified health centers that applied for participation in a voluntary demonstration of the PCMH Model of Care. SNMHI sites were selected for participation based on a range of factors, including interest in PCMH implementation and perceived aptitude for successful transformation, and may not be representative of the full population of primary care safety net sites. While these results are promising and address an important gap in PCMH implementation, continued testing in more heterogeneous settings is warranted. Also, the PCMH-A was administered in the context of substantial training and technical assistance related to implementation of the change concepts, and further research will be necessary to determine whether our findings can be replicated in circumstances associated with less support. However, although the PCMH-A was designed specifically to closely align with an action-oriented framework to guide quality improvement efforts—the Change Concepts for Practice Transformation (Wagner et al. 2012)—emerging feedback from diverse practices suggests that the PCMH-A will be useful and relevant to any practice on journey to become a PCMH. We conclude that the PCMH-A, a newly developed instrument to stimulate and monitor efforts to accelerate medical home transformation, is a valid tool that is sensitive to adoption of key changes needed for meaningful practice transformation among primary care practices.