Selecting a change and evaluating its impact on the performance of a complex adaptive health care delivery system.

Complexity science suggests that our current health care delivery system acts as a complex adaptive system (CAS). Such systems represent a dynamic and flexible network of individuals who can coevolve with their ever changing environment. The CAS performance fluctuates and its members' interactions continuously change over time in response to the stress generated by its surrounding environment. This paper will review the challenges of intervening and introducing a planned change into a complex adaptive health care delivery system. We explore the role of the "reflective adaptive process" in developing delivery interventions and suggest different evaluation methodologies to study the impact of such interventions on the performance of the entire system. We finally describe the implementation of a new program, the Aging Brain Care Medical Home as a case study of our proposed evaluation process.

Introduction

In 2001, the Institute of Medicine (IOM) identified suboptimal health care quality, compromised patient safety, and waste in our current health care system.1 These quality issues are found among a wide range of chronic conditions, including dementia. In 2005, Americans spent an estimated $100 billion providing care for three million Americans and their family caregivers affected by dementia.2,3 By 2011, these costs are expected to double.4 Despite the high cost of this care among those with recognized dementia, only 20% to 30% of Americans with dementia are diagnosed with the condition5,6 and multiple other quality problems have been reported. For example, 50% of older adults with dementia are exposed to a potentially inappropriate medication; only 7% are prescribed cholinesterase inhibitors; more than 20% are prescribed neuroleptics that have not been approved for older adults with dementia; and 33% utilize acute care services at least once every six months.5–7

The IOM recommended new organizational approaches and systems frameworks to ensure that the 21st Century American health care system delivers safe, effective, patient-centered, timely, efficient, and equitable care.1 In this white paper, we use complex adaptive system (CAS) theory to provide the reader with a framework to select, implement, and evaluate new care delivery models that have the potential of fulfilling the IOM recommendations.

The health care delivery system as a complex adaptive system

A CAS is a dynamic network of semiautonomous, competing, and collaborating individuals who interact and coevolve in nonlinear ways with their surrounding environment.8–11 These interactions lead to various webs of relationships that influence the system’s performance.8–12 By receiving and storing lessons from previous experiences and through modification of the relationships among its members, a CAS displays emergent self-organized behaviors and horizontal controls8,9,11,13–15 (Figure 1).

Figure 1

The complex adaptive health care system.

Health care delivery organizations are considered complex adaptive systems.7,10,13,16–24 These organizations are composed of semiautonomous individuals who interact constantly in a nonlinear way while faced with external and internal stressors such as patients’ medical status, insurance requirements, regulations, new research findings, members’ turnover, and legal issues.7,10,13,16–24 Traditional conceptual models of the health care delivery system often portray the health care system as a machine with replaceable parts and predictable behaviors that can be changed and reproduced based on past performance data.10,13,18,21–26 This view assumes that stability is the natural state of these organizations; that they consist of functions and roles that are carried out by replaceable nurses and physicians; and that financial incentives and regulatory policies offer recipes for predictable improvements in the performance.10,13,18,21–26

However, a new wave of health care leaders argue that the assembly line conceptual model does not fit health care systems.10,11,13,15,18,21,22,24–28 Rather, they conceptualize their organizations as a CAS with local critical nonlinear relationships that produce unpredictable behavioral patterns.10,11,13,15,18,21,22,24–28 Attempts to rigidly control these CASs often worsen the targeted problems and lead to unintended negative consequences.10,11,13,15,18,21,22,24–31

The reflective adaptive process of selecting and implementing a change in a complex adaptive health care delivery system

A health care leader using the lens of CAS understands that the capability of the health care delivery organization to adapt to an ever-changing environment depends on the skills and adaptability of its individual employees, their relationships and interactions, and the organizational communication patterns with other organizations.8,10,11,13,18,19,32

The research to support such an innovative approach is already beginning to emerge.20,33 In a systematic evidence review of 32 intervention studies seeking to improve chronic diabetes mellitus care, Leykum et al found a positive linear association between the success of a specific delivery intervention and the use of a CAS theory to develop or implement the intervention.20 In another systematic review of factors affecting the success of various programs to disseminate innovations in health service delivery, Greenhalgh found that some of the factors influencing adoption of innovations were: 1) the adaptability of the innovation to the needs of the adopter; 2) the feasibility, workability, and ease of use of the innovation; 3) the harnessing of the influence of true opinion leaders; and 4) the support of informal interorganizational networks for the adoption of the innovations.33 These factors resemble the framework and the characteristics of a CAS.

While the CAS framework is attractive for understanding the aspects of a health care organization, the method by which the CAS framework is used to implement an intervention is not explicit. Crabtree et al developed the “Reflective Adaptive Process” (RAP)13,18,24,26,34 to apply complexity science principles to select, develop, and implement a change in health care delivery systems.13,18,24,26,34 The RAP was developed over a series of large descriptive and intervention studies of primary care practices funded by the National Institutes of Health (NIH) and the Agency for Healthcare Research and Quality.13,18,24,26,34 RAP has five guiding principles that offer focus to implementation efforts without prescribing specific actions.13,18,24,26,34

“Vision, mission, and shared values are fundamental in guiding ongoing change processes”.

“Creating time and space for learning and reflection is necessary”.

“Tension and discomfort are essential and normal during change”.

“Improvement teams should include a variety of systems agents with different perspectives of the system and its environment, including patients”.

“System change requires supportive leadership that is actively involved in the change process, ensuring full participation from all members and protecting time for reflection”.13,18,24,26,34

The RAP starts with forming a cross-functional team that begins to meet regularly. This RAP team uses iterative cycles to identify priority improvement opportunities, discuss potential solutions, pilot several changes, and reflect on the impact of changes. A facilitator helps the team develop the necessary skills of group process, conflict management, meeting management, team building, and reflection-action cycles.

The RAP is considered a new Quality Improvement (QI) method that compliments the previous success of the other QI initiatives by recognizing the interdependence of system members and allowing the system leaders to create an optimal matrix for both the system members and their surrounding environment to coevolve.13,18,24,26,34,35

The effectiveness of RAP has been evaluated by the National Heart, Lung, and Blood Institute (NHLBI)-funded ULTRA (Using Learning Teams for Reflective Adaptation) study. ULTRA is a five-year group randomized clinic trial of 60 primary care practices in New Jersey and Pennsylvania that use RAP to enhance the care delivery for multiple chronic conditions.26,34 We have used RAP to develop a computerized decision support system for hospitalized older adults with cognitive impairment that was funded by the National Institute on Aging (K23-AG-26770–01).16,17 We also used the RAP principles to successfully build the Aging Brain Care Medical Home (ABC-MedHome) as described in detail later.

Evaluating the impact of a planned change on the performance of a complex adaptive health care delivery system

Currently, the process of approving a drug, device or new care delivery model requires a randomized double blind controlled clinical trial (RCT) to produce unbiased evidence of efficacy or safety.36–39 Without the use of RCT design, evaluating the efficacy of any change would be susceptible to various forms of bias such as the placebo response, the effects of unknown confounders, the natural history of any acute or chronic illness, and patients’ and providers’ negative or positive expectations of any introduced intervention.38 At the same time, extrapolating the results produced by a RCT suffers from generalizability limitations at the local implementation phase.36–40 Such generalizability limitations are due to the local and unique characteristics of the complex adaptive health care delivery system and demand local modification of the planned change.27,38,39 Ironically, the efficacy of such a modified locally sensitive version of the intervention would technically need to be tested in a new RCT. This paradoxical cycle is the main source of challenge in evaluation within the complex adaptive health care delivery systems. We need a solution that balances the internal and the external validity of the experimental intervention with the realities of a CAS.

We suggest two evaluation designs to test an intervention introduced into a specific CAS (Table 1): the use of N-of-1 randomized controlled trial (N-of-1 RCT);14,41–43 or a standardized pre–post time series design.

Table 1

Selecting a change in a complex adaptive health care delivery system

Selecting an overall content that is based on a systematic evidence review of past research or guidelines. Develop a reflective adaptive process implementation team to – Localize the content – Localize and or invent the delivery process – Monitor the delivery process – Monitor the system’s members’ interactions – Detect emergent behaviors – Evaluate the impact of the selected change

The N-of-1 RCT design

Guyatt et al introduced N-of-1 RCTs to medicine in 1986.41 Essentially the N-of-1 RCT uses time series data for both independent and dependent variables to evaluate the effect of therapy on one person. N-of-1 RCTs consist of a random sequence of different interventions that may include placebo, usual care or any other control, administered in double-blind protocol with regular and standardized measurement of specific intervention effects (efficacy measure) and measurement of impact on the entire system (harmful or unexpected positive or negative impact on the entire system).14,41–43 Typical group-based RCTs inform us of the average magnitude of an effect in a group of CASs. However, health administrators, managing a specific CAS, are concerned with knowing what is going to happen in their own health care delivery system.10,23

After selecting a planned change and choosing N-of-1-RCT as the evaluation design, three sets of outcomes need to be monitored. The first outcome set monitors the performance of the entire complex adaptive health care delivery system such as overall quality, cost, clinicians’ satisfactions, overall occupancy, overall errors, and overall length of stay. The second set is specific to the direct impact of the selected intervention. The third set monitors and measures the processes that mediate the effect of the interventions.14,41–43 The statistical analysis can be the application of a “sign test” that is based on the binomial distribution; and assuming there is no treatment difference, the probability of three consecutive pairs of treatment periods favoring active treatment is one in eight or 0.125 (one-sided test).14,41–43

N-of-1 RCT might be appropriate when several of the following criteria coexist:14,41–43

There are significant doubts about an intervention’s effectiveness in a specific complex adaptive health care delivery system;

The intervention might have potential adverse effects on the complex adaptive health care delivery system;

The selected intervention is expensive and its impact needs to be confirmed to justify these expenses;

The intervention is targeting a chronic but stable poor performance problem within a specific complex adaptive health care delivery system;

The evaluation team has the capacity to monitor all three types of outcomes mentioned above.

A Standard Pre-Post Time Series design

In cases where the use of N-of-1 RCT is not possible then an alternative option could be a standardized pre-post time series analysis. This design includes the following elements:

Selection of specific outcomes that are targeted by the intervention and choose a reliable measurement method;

Measuring the performance of the entire system using already existing data such as overall cost, overall quality, admission rates, death rate, discharge rate, patient satisfaction, and other data;

Select a preintervention and intervention period to measure the system and intervention-related outcomes.

A case study, the aging brain care medical home (ABC-MedHome) program

We conclude this paper by providing the reader with an example of a quality improvement project that follows the steps that we discussed regarding selecting an intervention in a complex adaptive health care delivery system and evaluating its impact (Table 2).

Table 2

Using the RAP to translate the collaborative care model into the ABC-MedHome

General principle	Local application	Delivery process
Complex adaptive systems need a mission, shared value, or a vision to implement change	Implementing Collaborative Care Model based on minimum standard approach to dementia and depression care.	Early and multiple one hour meetings with local leadership and the implementation team to develop and agree upon the shared vision, the minimum care standard, and the evaluation matrix.The team received technical and acculturation training in collaborative care.
Complex adaptive systems need time and space to adapt and plan change	The implementation teams need support for regular meetings for interaction.	Weekly one hour face-to-face meetings during the translational phase (up to 12 months) and biweekly meetings during the evaluation phase (up to 12 months) with time provided by the local health care system.
Tension and discomfort are normal in implementing change within complex adaptive systems	The complex adaptive system theory provides a structure to facilitate discussion, feedback, and review.	Internal facilitator uses a group problem-solving activity called a “consultancy”. This is structured to enable a set of people with a variety of knowledge and expertise to provide support, new perspectives, and ideas to one another, particularly around an important or difficult challenge.
Implementation design must incorporate the diversity of people and program affected by the change	Implementation teams are comprised of a matrix of people with the relevant roles, expertise, skills, and perspectives.	The team included a primary care physician, a primary care practice manager, a geriatric psychiatrist, a mental health counselor, a mental health practice manager, a geriatrician, a dementia care coordinator, a geriatric practice manager, team facilitator, and medical informaticians.
System change requires supportive leadership	Wishard Health Services leadership is actively involved in the change process, ensuring full participation from all members and protecting time for reflection.	A quarterly update of the implementation process to the leadership and quarterly review of the evaluation matrix.

Abbreviations: RAP, reflective adaptive process; ABC-MedHome, Aging Brain Care Medical Home.

Although primary care practice does not have the resources to provide the appropriate assessment and the complex management required for patients with dementia and depression,5,6,7 redesigning this practice setting to accommodate the special needs of these patients and their caregivers can improve health outcomes as evidenced by recent randomized controlled trials.12,44–46

Using the RAP method, the Indiana University Aging Brain program adapted the collaborative care model into the locally sensitive ABC-MedHome program. The goal of this program is to find, assess, and manage the biospsychosocial needs of older patients suffering from dementia or depression, and their family caregivers who are receiving care within the primary care practice affiliated with Wishard Health Services (WHS).

Translating the collaborative care model into the ABC-MedHome

WHS is an urban integrated public health care system serving residents of Marion County in Indiana. Guided by the RAP method, the implementation team was provided with the necessary time, space, semiautonomous decision making capacity, diversity, and leadership support to work with the clinical providers involved in the care of older adults with dementia or depression to agree on a locally acceptable approach to implementing the collaborative care model (Table 2). The team was provided with the existing guidelines and protocols from the Collaborative Care studies to develop a locally sensitive and minimally standardized care approach including consensus on the specific choice of screening instruments, diagnostic criteria, and measures of severity that will be used to deliver care at ABC-MedHome. The team also specified the role of the ABC-MedHome coordinator and director and other providers, the criteria for co-management, and the criteria for referral to memory care specialists. Finally, the team attended to the need to integrate current guidelines for care among those patients who suffer from dementia and depression with particular attention to the co-management of cardiovascular diseases. Although caregivers are targeted as a vehicle to deliver care to the care recipient, the team developed a standardized approach to directly detecting and treating the caregiver for depression.

To accomplish the above goals, the team met on a weekly basis for approximately one year and worked closely via quarterly reports with the leadership of the target health care system to understand the opportunities and barriers to the implementation of the ABC-MedHome from a financial, human resource, and space perspective.

Planned evaluation of The ABC-MedHome

As mentioned earlier, implementation of change in a health care system often fails because organizers fail to recognize the complex adaptive system properties of this dynamic human system and design an inappropriate feedback process that monitors the impact of the change on the entire system. Using the conceptual model of CAS and the resources from National Institute of Mental Health (P30AG024967), we developed an evaluation platform that would provide continuous and timely data relevant to the performance of the ABC-MedHome for the leadership of WHS and would also support conducting time series analyses that could provide an adequate method to investigate any potential causal association between the ABC-MedHome and certain system outcomes.

The evaluation matrix included the following specifications:

Define the borders of the complex adaptive system by selecting the patients who would potentially interact in a direct or indirect way with the program.

Determine and measure the outcomes of the entire system using already existing data such as services utilization data, quality care, admission rate, death rate, discharge rate, patient satisfaction, and other data.

Select preintervention and postintervention periods to measure the above outcomes.

Select the length and the frequency of performance reports.

Based on the data from the three clinical trials that evaluated the impact of the collaborative care model, using the above criteria, and utilizing the capacity of the local health care system electronic medical record system, the implementation team has finalized the ABC-MedHome performance platform (See Table 3).

Table 3

The evaluation platform or the matrix of the ABC-MedHome performance

Selection of the patients cared for by the complex adaptive primary care delivery system prior to and during the implementation of the ABC-MedHome program – Older adults (age ≥ 65) with at least one visit to the target PCP within the defined period (annual or quarterly) AND – Carry any ICD-9 codes of dementia or depression; or receiving at least one prescription of antidementia or antidepressant medications (using both inpatient and outpatient eMR within three years prior to the visit) Evaluation period – Every three months (quarterly report); every 12 months (annual report) The domains of the evaluation matrix
Ambulatory services	Acute services	Quality indicators
– Number of patients* seen at least once at PCP. – Number of PCP visits.* – Number of patients* seen at least once at MCP – Number of MCP visits.* – Number of brain imagings. – Number of laboratory tests (comprehensive metabolic profile, blood count, thyroid function test, vitamin B12 and folate levels, lipid profile, hemoglobulin A1c, others).	– Number of patients* seen at least once at the ER. – Number of ER visits.* – Number of hospitalized patients. – Number of hospitalizations. – Median length of hospital stay.	– Among patients with ER visit, % seen at PCPC within 7 days; % return to ER within three days. – Among hospitalized patients, % seen at PCPC within 7 days of hospital discharge; % rehospitalized within 30 day of discharge. – Among all ABC-MedHome patients, % with at least one unfilled prescription; % with at least one order of definite anticholinergics. – Among patients with dementia, % with at least one order of neuroleptics. – Among Alzheimer disease or Lewy body dementia patients, % with at least one order of antidementia medications. – Among depression patients, % with at least one order of antidepressants. – Among patients receiving antidementia medications, % with at least one order of definite anticholinergics. – Among patients with hyperlipidemia, % of patients with at least one LDL order. – Among patients with hyperlipidemia, % with LDL < 130 – Among patients with diabetes, % with at least one hemoglobulin A1c order. – Among patients with diabetes, % with hemoglobulin A1c < 8. – Among patients with hypertension, % of patients with systolic blood pressure <160 during last PCP visit.

Notes:

Patients with dementia or depression;

Period is annually or quarterly.

Abbreviations: ABC-MedHome, Aging Brain Care Medical Home program; PCPC, Primary Care Practice; ICD, International Classification Diagnosis codes; eMR, Electronic Medical Record; pt, patient; MCP, Memory Care Practice; ER, Emergency Room.

The generalizability of the Abc-Medhome

The program was developed and is currently undergoing field-testing at WHS. There are three characteristic of WHS that might limit the generalizability of the ABC-MedHome. First, WHS provides care to a diverse urban and vulnerable population of older adults many of whom are low-income and subject to multiple socioeconomic stressors. Thus, generalizability to rural and solo primary care practices is limited. Second, WHS has been the site of the two randomized trials of the collaborative care model for both dementia and depression and thus, the buy-in of the leadership and the clinicians of the locally sensitive content of the ABC-MedHome had already been facilitated. Third, WHS is served by one of the nation’s oldest and most comprehensive electronic medical records that captures diagnostic studies, physician orders, drug dispensing, death certificates, encounter information, dictated reports, hospital admission and discharge diagnoses, dates and lengths of hospital stay, and discharge disposition. Thus, health care systems that do not have similar electronic systems may struggle in providing data for the evaluation platform of the ABC-MedHome. However, some of these features are shared by other health care systems. For example, the replication of ABC-MedHome is possible in systems such as the Veteran Affairs (VA) health care system and Kaiser Permanente health system. Both have a comprehensive electronic medical records system, are considered integrated health care systems, and have conducted numerous research activities related to the collaborative care model. Indeed both Kaiser and the VA were performance sites for the collaborative care model RCTs.44,45

Conclusion

The current research infrastructure and the CAS nature of our health care delivery organizations are not well designed to support the translational “discovery to delivery” step that will be required to reduce the current and future burden of chronic care epidemics facing our American society. The IOM and the NIH recognized this large gap in translating research innovations from discovery to delivery and recommended urgent “re-engineering of the clinical research enterprise”. Understanding our health care delivery system as a CAS might transform our health care system into a new integrated adaptive discovery network capable of continuously, instantly and efficiently improving the safety, quality, and cost utility of the American health care system. Such an understanding may enhance our capacity to bridge research findings and the daily dynamics of these organizations and thus develop a flexible and local process of a minimally standardized approach with continuous feedback loops to maintain the best probability of surviving the often unpredictable future challenges that will face these health care delivery systems.