Applying the RE-AIM framework to the Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions.

INTRODUCTION: Diabetes represents a major public health and health system burden. As part of the Alberta's Caring for Diabetes (ABCD) Project, two quality-improvement interventions are being piloted in four Primary Care Networks in Alberta. Gaps between health research, policy and practice have been documented and the need to evaluate the impact of public health interventions in real-world settings to inform decision-making and clinical practice is paramount. In this article, we describe the application of the RE-AIM framework to evaluate the interventions beyond effectiveness. METHODS AND ANALYSIS: Two quality-improvement interventions were implemented, based on previously proven effective models of care and are directed at improving the physical and mental health of patients with type-2 diabetes. Our goal is to adapt and apply the RE-AIM framework, using a mixed-methods approach, to understand the impact of the interventions to inform policy and clinical decision-making. We present the proposed measures, data sources and data management and analysis strategies used to evaluate the interventions by RE-AIM dimension. ETHICS AND DISSEMINATION: Ethics approval for the ABCD Project has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. The RE-AIM framework will be used to structure our dissemination activities by dimension.
RESULTS: It will be presented at relevant conferences and prepared for publication in peer-reviewed journals. Various products, such as presentations, briefing reports and webinars, will be developed to inform key stakeholders of the findings. Presentation of findings by RE-AIM dimension will facilitate discussion regarding the public health impact of the two interventions within the primary care context of Alberta and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. It will also promote the application of evaluation models to better assess the impact of community-based primary healthcare interventions through our dissemination activities.

Diabetes represents a public health and health system burden; primary care requires collaborative team interventions and self-management support.

Study protocol using a mixed-method approach for a comprehensive evaluation.

Evidence on effectiveness of public health interventions in real-world settings is needed to better inform decisions about practice and policy.

RE-AIM model provides a framework to elicit contextual information to better interpret the effectiveness of interventions.

Dissemination activities can be structured using the RE-AIM dimensions to identify target audiences, key messages and appropriate knowledge products.

Application of the RE-AIM framework and a mixed-methods approach allows for a comprehensive evaluation of the ABCD Project interventions beyond clinical effectiveness.

This represents the evaluation of a pilot study in four Primary Care Networks (PCNs), which may not be representative of other primary care models.

Introduction

Diabetes represents a major public health and health system burden. The Canadian National Diabetes Surveillance System has estimated that 6.2% of the population has diabetes.1 In Alberta, 206 000 people were living with diabetes in 2009, representing over 5.5% of the population.2 This signifies a doubling of affected individuals within the past decade. The majority (ie, >90%) of these individuals have type 2 diabetes. As the number of people with diabetes increases, the number of resulting complications and co-morbidities increases, creating a greater demand on healthcare resources.2 3

The Alberta's Caring for Diabetes (ABCD) Project, funded by the Alberta Health ministry as part of the provincial diabetes strategy, was developed to improve the quality and efficiency of care for diabetes in Alberta, Canada, with a focus on supporting Primary Care Networks (PCNs) in non-metro areas of Alberta. PCNs consist of a voluntary network of family physicians (hereby referred to as ‘member physicians’) and allied health professionals, who identify priorities and coordinate health services for patient populations.4 5 The PCN model is akin to the ‘patient-centered medical home’ model emerging in the USA.6 7

The ABCD team has worked with participating PCNs to implement a number of quality improvement interventions. This includes an ongoing, survey-based cohort study that seeks to understand why some people with type 2 diabetes develop complications while others do not. This study involves an annual survey of individuals with type 2 diabetes over 5 years, to collect data on lifestyle behaviours, self-management and patient-reported outcomes and linkage with administrative databases to assess healthcare utilisation and longer-term clinical outcomes. In addition, participating PCNs will implement pilot interventions including: (1) Healthy Eating and Active Living in Diabetes (HEALD-PCN), a pedometer-based walking programme;8 and (2) TeamCare-PCN, a collaborative team-based, depression case management intervention.9 Key features of HEALD-PCN include the provision of information in a group setting by an exercise specialist on increasing the amount and intensity of physical activity (ie, walking), the glycemic index and individual goal setting. The HEALD-PCN programme also provides opportunities for participants to implement lessons learned (ie, walking group sessions) through partnerships with community recreational facilities.10 Key features of TeamCare-PCN include coordinated care by a nurse care manager to direct active patient follow-up, treat-to-target principles and specialist (ie, psychiatrists and internists/endocrinologists) consultation.11

The efficacy of both pilot interventions has been proven in other settings,10–12 and the study protocols to determine the effectiveness of HEALD-PCN and TeamCare-PCN in the PCN environment in Alberta have been published.8 9 Our goal is to also assess the impact of the entire ABCD project activities, including how these different interventions were simultaneously implemented, in Alberta's PCN environment. The purpose of this paper is to describe the design of the evaluation for the different elements of the ABCD project, using the RE-AIM framework.13

Evaluating the ABCD pilot interventions using RE-AIM

The gaps between health research, policy and practice have been well documented.13–16 Evaluations of health interventions are often limited to efficacy studies rather than assessment of potential public health impact.17 Efficacy studies tend to focus on the internal validity of high-intensity health interventions with motivated and homogeneous populations in controlled settings.13 This narrow focus hinders the translation of research into practice and reduces the ability to generalise findings to similar settings.13 Evidence on the external validity of less-intensive interventions in real-world settings is needed to better inform decisions about practice.13

In this context, assessment of clinical effectiveness alone is not enough to inform decisions about a programme's broader public health impact. The RE-AIM evaluation framework was designed to assess health interventions beyond effectiveness to include multiple criteria to better identify effect and transferability.18 The framework consists of five dimensions: Reach into the target population; Effectiveness of the intervention; Adoption by target settings, institutions and staff; Implementation, including consistency and cost of delivery; and Maintenance of intervention effects over time.18

The RE-AIM model addresses two levels of assessment: individual (Reach, Effectiveness); organisation (Adoption and Implementation) or both (Maintenance).13 To fit our evaluation goals, we expanded the assessment level of ‘Reach’13 beyond the individual assessment level (ie, absolute number, proportion and representativeness of individuals willing to participate in an intervention) to include an organisation assessment level (ie, an organisation's ability to identify the entire target population) (table 1). An example of an organisational strategy to identify a population is the development and use of a patient registry.

Table 1

RE-AIM dimensions, definitions and assessment levels for evaluation of the ABCD pilot interventions

Dimension	Definition	Level of assessment
Reach	The ability to identify targeted population(s) at an organisational level and the absolute number, proportion and representativeness of individuals who are willing to participate in an intervention	Individual and organisational
Effectiveness	The impact of an intervention on important outcomes, including potential negative effects and quality of life	Individual
Adoption	The absolute number, proportion, and representativeness of settings and intervention agents (ie, people who deliver the programme) who are willing to initiate an intervention	Organisational
Implementation	At the individual level, implementation refers to clients’ use of the intervention strategies. At the setting level, implementation refers to the intervention agents’ fidelity to the various elements of an intervention's protocol, including consistency of delivery as intended, and the time and cost of the intervention	Individual and organisational
Maintenance	At the individual level, maintenance has been defined as the long-term effects of a programme on outcomes six or more months after the most recent intervention contact. At the setting level, maintenance refers to the extent to which a programme or policy becomes institutionalised or part of the routine organisational practices and policies	Individual and organisational

ABCD, Alberta's Caring for Diabetes.

In table 1, we provide the original definitions for each RE-AIM dimension. Italicised words or phrases indicate modifications made by the ABCD Project team to the original ‘Reach’ definition and assessment level.13 This table was compiled and adapted from several sources.13 17 18

While there are other evaluation frameworks, such as Procede-Proceed19 and Health Impact Assessment,20 we assert the RE-AIM model is well suited to evaluate the ABCD pilot interventions for two reasons. First, RE-AIM is considered more appropriate for evaluation of behavioural change interventions21 than other models. Second, the dimensions of the RE-AIM model are well matched to inform the specific needs of our audiences and interested parties including healthcare providers, PCN management, policy makers and funders.

Methods and analysis

We will employ a mixed-methods approach22 for our comprehensive evaluation of the ABCD pilot interventions. Using the RE-AIM model, our research team developed logic models and data matrices for both interventions in consultation with advisory committees (see online supplementary Appendix 1; web only file). The overarching questions guiding the evaluation for each intervention are: (1) Is the service delivery model effective in the context of Alberta's primary care setting and (2) What factors contribute to the effectiveness (or ineffectiveness) of the intervention? The more specific evaluation questions related to the RE-AIM framework that will direct the collection and analysis of data for both interventions include:

Reach: Is the intervention reaching the intended target population?

Adoption: Has the intervention been adopted by the PCNs and staff?

Implementation: Is the intervention being implemented as intended? Is it cost-effective?

Effectiveness: What are the immediate, intermediate and long-term impacts of the intervention?

Maintenance: Is the intervention sustainable in a cost-effective way?

Measurement by RE-AIM dimensions

In the following section, we outline the measures proposed for each dimension of RE-AIM to evaluate the ABCD project interventions. A detailed summary is provided in table 2.

Table 2

Measures, data sources and data collection timeline by RE-AIM dimension and assessment level

Assessment level(s)	Measures	Data sources	Timeline
Reach
Individual	▸ Eligibility criteria	▸ Patient-recruitment tracking system	▸ Ongoing
	▸ Demographic information	▸ Survey items	▸ HEALD-PCN specific:  baseline, 3–6 months▸ TeamCare-PCN specific: baseline, 612 months
	▸ Identified facilitators and barriers to recruitment▸ Identified recommendations for improvement	▸ Interview data (PCN staff and ABCD team)	▸ Baseline and midpoint
	▸ Patient characteristics (participants vs population)	▸ PCNs’ patient registry▸ AH/ADSS data	▸ Post-intervention
Organisation	▸ Ability to estimate and identify targeted patient populations	▸ Document review (standardised checklist)	▸ Baseline
	▸ Registry development and maintenance process issues, including identified facilitators and barriers▸ Identified recommendations for improvement	▸ Interview data (PCN staff and ABCD team)	▸ Baseline and midpoint
		▸ Document review (field notes)	▸ Ongoing
Effectiveness
Individual	Primary outcomes: A1c, blood pressure, total cholesterol, & BMI▸ HEALD-PCN specific: total # of steps▸ TeamCare-PCN specific: Composite of PHQ-9 Secondary outcomes: self-reported quality of life, quality of care, self-efficacy, & satisfaction with care▸ HEALD-PCN specific: nutritional behaviours & satisfaction with intervention▸ TeamCare-PCN specific: process care indictors including: # of visits with healthcare providers, referrals, psychotherapy sessions, medication adjustments, and adherence to treatment	▸ Clinical assessment recorded in patient outcome tracking systems▸ Survey items	▸ Ongoing▸ HEALD-PCN specific: baseline, 3–6 months▸ TeamCare-PCN specific:  baseline, 6–12-months
	▸ Perceptions of impact/ consequences (positive or negative)	▸ Interview data (PCN staff)	▸ Baseline, midpoint, and post-intervention
Adoption
Individual	▸ Total number of member physicians participating in ABCD project	▸ Document review (PCN and ABCD project documents)	▸ Post-intervention
Organisation	▸ Criteria for PCN participation in ABCD Project▸ PCN Board agreement to participate▸ Features of participating PCNs▸ Comparison of characteristics between participating and  non-participating PCNs, as possible▸ Description of usual care in the focus areas▸ Perception of extent to which ABCD Project has been adopted by PCNs and modified to fit their context(s)▸ Identified facilitators, barriers, and recommendations at  organisational level	▸ Document review (project and PCI/PCN documents –websites and business plans, availability of secondary data e.g., PCI evaluation)▸ Standardised checklist▸ Interview data (PCN staff)	▸ Baseline, midpoint, and post-intervention
Implementaton
Individual	▸ HEALD-PCN specific: # of steps in log and  self-reported physical activity▸ TeamCare-PCN specific: adherence to treatment  plan, including medications and behavioural modifications	▸ Patient outcome tracking systems▸ Survey items	▸ Post-intervention
Organisation	Development of:▸ Project materials: job descriptions for intervention staff,  recruitment and data collection protocols and forms▸ Training and resource materials: project binders, algorithms,  patient resources▸ Systems/processes: patient registries, patient recruitment &  outcome tracking systems	▸ Document review (PCN and ABCD Project  documents)	▸ Baseline
	▸ # and type of intervention staff hired by PCNs, including  turnover	▸ Document review (eg, contracts)	▸ Ongoing
	▸ Provision of and quality of training in ABCD Project and  interventions: # and type of staff trained, detailing sessions, and training materials provided; attendance in training sessions; assessment of change in knowledge and satisfaction	▸ Document review (ABCD Project documents)▸ Presurvey /postsurvey items▸ Interviews with PCN intervention staff	▸ Baseline, midpoint, and  post-intervention
	Service delivery: ▸ HEALD-PCN specific: # and type of group meetings and  patient resources distributed; level of attendance▸ TeamCare-PCN specific: # and type of screenings,  assessments, patient management plans, follow-up sessions, specialist consultations; time of service delivery; and QI assessment through monthly teleconferences	▸ Document review:(class attendance lists)▸ Patient outcome tracking systems	▸ Ongoing and post-intervention
	▸ Perceptions of implementation as intended▸ Identified facilitators and barriers to implementation▸ Identified recommendations for improvement	▸ Interviews with PCN staff	▸ Baseline, midpoint, and  post-intervention
		▸ Document review (field notes, communications,  meeting minutes)	▸ Ongoing
	▸ Economic Evaluation: Decrease in # of family physician and  ER visits; reduction in complications, co-morbidities, and  mortality; reduction in direct medical costs; and reduction in projected future healthcare costs	▸ Document review (budget and invoices)▸ AH/ADSS data	▸ Post-intervention
Maintenance
Individual	▸ Sustained awareness, knowledge, and management of type  2 diabetes and depression or lifestyle behaviours	▸ Survey items (ABCD Cohort Study) regarding  health behaviours and self-care	▸ Post-intervention & ongoing  (minimum 4-year follow-up)
		▸ Interviews with HEALD-PCN intervention group  participants	▸ Post-intervention
Organisation	▸ PCN level: integration of aspects of the model into usual  care; and/or incorporation of models into future business planning	▸ Interviews with PCN staff	▸ Post-intervention
	▸ More appropriate healthcare utilisation: decrease in # of  family physician and ER visits; reduction in complications, comorbidities, and mortality; reduction in direct medical costs; and reduction in projected future health care costs	▸ AH data	▸ Post-intervention

Reach

Evaluation of reach will be done at the individual (patient) and organisation (PCN) assessment levels to determine if the ABCD pilot interventions are reaching those in most need. At the individual assessment level, we will examine total recruitment into the interventions and usual care groups and compare their characteristics with respect to eligibility criteria, demographic information and other measures. As possible, we will compare characteristics between participants (ie, intervention and usual care groups) and non-participants using aggregate demographic information accessed through PCN patient registries and Alberta Diabetes Surveillance System data.23 Facilitators and barriers to individual patient recruitment and suggestions for improvement will be identified through interviews with PCN staff.

At the organisation assessment level, we will document usual care in the PCNs, including the ability to estimate and identify target patient populations in the focus areas (ie, type 2 diabetes management, depression management and lifestyle counseling) through completion of a standardised checklist. We will examine processes related to registry development and identify facilitators and barriers related to development, use and maintenance through interviews with PCN staff. In addition, we will elicit recommendations related to the PCNs’ ability to identify patient populations to actively offer targeted health services.

Effectiveness

Evaluation of effectiveness will be conducted at the individual assessment level to determine impact of the pilot interventions on important outcomes. The design and rationale for controlled evaluations of the effectiveness of the two ABCD pilot interventions have been described elsewhere.8 9 The primary outcome of HEALD-PCN is improvement in physical activity (ie, brisk walking), determined by step pedometers and self-report.8 For TeamCare-PCN, the primary outcome is improvement of depressive symptoms as measured by the Patient Health Questionnaire-9 (PHQ-9) items.9 We will also use a variety of measures to determine the effectiveness of both interventions on important outcomes at the individual assessment level including clinical measures (eg, improvements in glycemic control, blood pressure, lipid measurements and body mass index), self-reported health-related quality of life,24–26 self-efficacy,27 satisfaction with care,28 29 and process indicators. In addition, we will document unanticipated consequences (positive or negative), such as improved patient linkages with community health resources, to provide a richer understanding of effectiveness. Additional measures and data sources to assess effectiveness are provided in table 2.

Adoption

We will assess the adoption of the ABCD pilot interventions at the organisation level, including documentation of the criteria for PCN selection and participation in the ABCD Project and PCN Board approval. Also, we will document and compare the characteristics of the participating PCNs (eg, number of family physicians, number of patients served and governance structure) as well as usual care in the focus areas. Dependent on availability of secondary data, we will consider the representativeness of participating PCNs compared with non-participating PCNs. This will be accomplished through document review (eg, ABCD project documents, PCN websites, business plans), use of a standardised usual care checklist and interviews with PCN staff. In addition, perceptions related to the extent to which the ABCD pilot interventions have been adopted by the PCNs and modified to suit their contexts will be elicited through interviews with PCN staff. Identified facilitators and barriers to adoption of the interventions along with creative solutions or modifications will also be documented.

Implementation

Evaluation of implementation of the ABCD pilot interventions will be done at the individual and organisation assessment levels to determine patient adherence, consistency of implementation and costs of delivering the pilot interventions. To address implementation at an individual assessment level, participant adherence to the intervention models will be determined for both interventions. For HEALD-PCN, attendance at group sessions, participant step logs (ie, recording the number of steps over 3 days) and self-reported physical activity will be assessed. For TeamCare-PCN, adherence to treatment plans, including medication and behavioural modifications (eg, engaging in planned pleasant activities), will be assessed. These types of data will be derived from patient outcome tracking systems employed in each PCN and/or survey items.

At the organisation assessment level, consistency of implementation and the cost of delivering the ABCD pilot interventions will be evaluated to determine the practicality of the interventions. Actual versus intended implementation will be assessed through extensive documentation including development of project materials (eg, training and resource materials), presence of systems and processes (eg, patient registries), intervention staff recruited or hired by PCNs and provision and quality of training in the intervention models. Additional measures and data sources to assess consistent implementation are provided in table 2. Our implementation assessment will also include economic evaluations of the ABCD pilot interventions, which have been described in detail elsewhere.8 9

Maintenance

For both ABCD pilot interventions, maintenance will be evaluated at the individual and organisation assessment levels to measure continuation of intervention effects over time. We will use a previously developed conceptual framework that defines sustainability outcomes of health interventions.30 At the individual level, maintenance will be evaluated based on patient-reported health behaviours and self-care collected annually through the ABCD cohort study survey and interviews with a subsample of HEALD-PCN intervention group participants at 6-months post-intervention.

At the organisation assessment level, interviews with PCN staff will be conducted post-interventions to assess integration of intervention model components into practice (eg, continued use of patient registries or screening tools), enhanced organisational capacity (eg, maintaining partnerships) and continued focus on the interplay between diabetes, depression and lifestyle (eg, incorporation of the intervention models into future business plans). In addition, interviews with specialists participating in TeamCare-PCN will be conducted with a focus on sustainability of the model in the current primary care environment, including appropriate compensation and funding approaches and potential medicolegal liability issues.

Data management

Our comprehensive evaluation will involve the collection and management of a wide range and large volume of data. Primary data sources for the evaluation of the ABCD pilot interventions include: (1) clinical outcome measures; (2) patient-reported outcomes; (3) interviews (eg, with PCN staff, HEALD-PCN intervention group participants and specialists for TeamCare-PCN); (4) document review (eg, usual care checklists, project documents and field notes) and (5) administrative healthcare datasets.

Clinical outcomes and survey data captured in the patient outcome tracking systems or standardised case forms used in each PCN will be entered into centralised, web-accessible databases. These study databases will be housed on secure servers in the research offices at the University of Alberta. Once the pilot interventions are completed, all data will be exported and merged, based on individually assigned study ID numbers, to form an analysable dataset. Investigators, research assistants and analysts will be masked to allocation status at all times.

Semistructured interviews will take place at the PCN offices of the interviewees. Interviews with HEALD-PCN intervention group participants and TeamCare-PCN specialists will be conducted via telephone. Interviews will be facilitated through the use of interview guides. Interviews will be digitally recorded for subsequent analysis, transcribed verbatim by an independent transcriptionist and verified for accuracy.

Regarding document review, we will develop a standardised usual care checklist by adapting themes from the Change Process Capability Questionnaire31 and the Organisational Readiness to Change Scale32 to be validated by staff of the participating PCNs. Topic areas include: usual care for people with type-2 diabetes; existing PCN diabetes, depression and lifestyle programming and organisational factors and strategies related to PCN patient care. Also, we will document how the ABCD pilot interventions unfolded in each PCN through field notes, communications and meeting minutes. All qualitative data sources, including interview transcripts and documents, will be compiled and managed using Nvivo V. 9.0 software.

Patients enrolled in the pilot interventions and the ABCD Cohort study will be asked for permission to access their medical records by providing their personal health number, thus allowing linkage to provincial healthcare administrative data from Alberta Health for physician, hospital, and emergency department billing and pharmaceutical data (for patients 65 years and older). This linkage will allow healthcare utilisation and healthcare costs to be included in the evaluation.

Data analysis

We are undertaking a broad mixed-methods approach to analysis. In terms of quantitative data, the approach to power, sample size calculations, assessment and statistical modelling of clinical effectiveness have been previously detailed.8 9 In terms of qualitative data, we will take a general inductive approach33 with the evaluation questions related to the RE-AIM framework directing the analysis of data. Findings will be derived directly through a content analysis34 of the raw data without preconceived notions about specific findings.

Ethics and dissemination

Ethical considerations

Ethics approval for the entire ABCD Project and its associated interventions has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. However, the Board deemed this component of the ABCD Project as evaluation and not research; therefore, it did not require ethics review and approval. Regardless,  the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans35 will be followed.

Discussion and dissemination

The ABCD Project was developed to improve the quality and efficiency of diabetes care in non-metro Alberta. In order to address the gap between research, policy and practice, we have adapted and expanded the RE-AIM model to conduct a comprehensive evaluation of the ABCD pilot interventions. This will contribute to our knowledge of the broader impact of the two interventions within the evolving primary care context of Alberta beyond effectiveness, as outlined in the study trial designs.8 9 The purpose of this article was to present the proposed measures and data sources to be used to evaluate the interventions by RE-AIM dimension. Using the RE-AIM evaluation framework will allow us to systematically identify facilitators, challenges, opportunities and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. In addition, our application of the RE-AIM evaluation framework may encourage others to use similar models to determine the impact of community-based primary-healthcare interventions. The RE-AIM model will also be used to structure our dissemination activities. For example, each RE-AIM dimension will inform the development of products (such as academic manuscripts for peer-review publication, presentations at relevant conferences and workshops, and briefing reports) and identification of relevant target audiences.