Process and outcome evaluation of a CBME intervention guided by program theory.

RATIONALE: Competency-based medical education (CBME) has gained momentum as an improved training model, but literature on outcomes of CBME, including evaluation of implementation processes, is minimal. We present a case for the following: (a) the development of a program theory is essential prior to or in the initial stages of implementation of CBME; (b) the program theory should guide the strategies and methods for evaluation that will answer questions about anticipated and unintended outcomes; and (c) the iterative process of testing assumptions and hypotheses will lead to modifications to the program theory to inform best practices of implementing CBME.
METHODS: We use the Triple C Competency-based Curriculum as a worked example to illustrate how process and outcome evaluation, guided by a program theory, can lead to meaningful enhancement of CBME curriculum, assessment, and implementation strategies. Using a mixed methods design, the processes and outcomes of Triple C were explored through surveys, interviews, and historical document review, which captured the experiences of various stakeholders.
FINDINGS: The theory-led program evaluation process was able to identify areas that supported CBME implementation: the value of a strong nondirective national vertical core supporting the transformation in education, program autonomy, and adaptability to pre-existing local context. Areas in need of improvement included the need for ongoing support from College of Family Physicians of Canada (CFPC) and better planning for shifts in program leadership over time.
CONCLUSIONS: Deliberately pairing evaluation alongside change is an important activity and, when accomplished, yields valuable information from the experiences of those implementing and experiencing a program. Evaluation and the development of an updated program theory facilitate the introduction of new changes and theories that build on these findings, which also supports the desired goal of contributing toward cumulative science rather than "reinventing the wheel."

INTRODUCTION

Health care systems are a product of many factors, and in recent years, medical education has been an area of focus.1, 2, 3, 4 Some evidence suggests current training methods need improvement to support the development of physicians who are able to provide care that meets the needs of complex, diverse, and ever‐evolving communities.1, 2, 3, 4 Competency‐based medical education (CBME), originally introduced in 1978, has regained momentum over the past decade as an educational approach that can address some of the current issues in health care systems by transforming the way that health care providers are trained.5, 6 While many definitions and interpretations of CBME have been identified in the literature, our discussion of CBME is defined as:

[…] an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It deemphasizes time‐based training and promises greater accountability, flexibility, and learner‐centredness.7

Although multiple health professions education regulatory bodies around the world have mandated a shift to CBME,6, 8 CBME is not accepted across all stakeholders.8, 9, 10, 11 While the assumptions of CBME are based on established educational theory with supporting evidence,5, 12, 13, 14, 15 there have been critical debates in the health professions training literature regarding the potential impact of this education transformation. One of the key debates centres on outcomes: how can we know whether CBME will result in improved outcomes, and can we justify the human and financial costs associated with a transition to CBME when there is no certainty of a better outcome?16, 17, 18

How can we evaluate a complex education intervention, such as CBME?

Implementation is defined as the deliberate introduction of new, or modifying existing, patterns of collective action.19 These patterns of collective action are institutionally sanctioned, formally defined, consciously planned, and intended to lead to a changed outcome.19 Implementation of new education models is complex as it involves modifying how people think, act, and organize themselves and others to promote collective action, leading to desired outcomes.19 Several authors have highlighted the complexity involved in implementing CBME, given the multiple stakeholders and settings that must be involved, along with multiple levels of approvals creating obstacles affecting adoption of CBME.20, 21, 22, 23 Critics have argued that all too often, the complexity involved in implementing change is used as an excuse to justify why CBME cannot be rigorously evaluated.6, 11, 24, 25, 26 We propose that deliberate evaluation of both the process and outcome of CBME is needed and can be done rigorously, yielding useful information in support of ongoing CBME implementation and improvement.

Criticism in the literature about whether CBME results in “improved” graduates can be answered through use of outcome evaluation.8, 9, 10, 11, 16, 17, 18 Outcome evaluation examines the progress of the program and the status of accomplishing desired results and answers the questions such as unintended outcomes, return on investment, and changes in knowledge, attitudes, and behaviours.27, 28 While literary conversations about CBME have predominantly focused on outcomes in the form of the individual competence of graduates, understanding best practices in enacting the implementation of CBME is equally as important as measuring its outcomes. In other words, evaluating how CBME is implemented with an understanding of the context within which it is implemented will provide a more a fulsome understanding of why an outcome emerges.

Process evaluation explores social processes and mechanisms during the implementation of an organizational transformation and prospectively draws a bridge to the evaluation of outcomes associated with the change.29, 30 Process evaluation questions can be asked throughout the implementation of CBME with questions such as the following: what are the barriers/facilitators to implementing CBME activities; what has been accomplished; and who is being impacted by the CBME transformation.27 By understanding factors that influence outcomes, whether positive or negative, more information can be derived for use in the future.

The implementation of CBME is very context dependent. Adapting a CBME intervention that worked in one context may or may not be feasible in another. However, successful uptake of an innovation by others may be more likely if evaluation of both processes and outcomes are planned alongside implementation.29, 31 Without this type of evaluation, information about contextual mechanisms and processes are lacking, and therefore, the risk is that future implementers may fall into the “cargo cult” problem of organizational transformation whereby the process of implementation is reduced to adherence to a “checklist” protocol, but essential mechanisms and processes that supported the outcomes observed in the original context are lacking, resulting in a superficial imitation lacking specificity.32 Without knowing the “how” and “why” questions related to implementation, implementers may not have access to useful information about the necessary enablers or barriers that impact anticipated outcomes. They also may not know how to carry out implementation in their own contexts if influencing factors are not included within itemized protocols.29, 31, 33

Theory‐based evaluation

This paper proposes that theory‐based evaluation approaches, such as those used in evaluating complex community‐based social and health care initiatives,34, 35, 36, 37, 38 can be used for CBME. Theory‐based evaluation begins with the development of a program theory (also called a theory of change), which clearly defines a problem that a transformation or intervention is anticipated to address, and how this change (in this case the shift to CBME) is anticipated to be successful.39, 40, 41, 42 The program theory defines intended impacts of a change or intervention and then systematically maps factors that contribute to a chain of short‐ and long‐term outcomes that are expected to have impact.41, 43, 44, 45 Accompanying the program theory should be a logic model, illustrating the activities involved in CBME implementation including a description of short‐ and long‐term outcomes.46, 47, 48, 49 Contributory factors that influence the process and the outcome of implementation should be identified in the logic model including assumptions/hypotheses of the results of CBME.50, 51

The program theory acts as a guide for the development of a program evaluation plan that can intentionally explore inputs, processes, outcomes, and impact. The findings from a process and outcome evaluation, as defined in this paper, sets up a cyclical opportunity for ongoing improvement, enhancing the anticipated program theory and its process and outcomes.52, 53, 54 For those involved in CBME, this includes understanding elements influencing the process of evaluating the fidelity (ie, does the program as enacted look like the program as conceived?) of implementation, as well as its integrity (eg, do workplace‐based assessments provide quality feedback that are detailed and actionable?).29, 54, 55, 56 Findings from this type of evaluation can enhance our understanding of the best CBME implementation approaches and highlight factors for consideration that may impede or facilitate its success.

Worked example: Triple C Competency‐based Curriculum (Triple C)

The links between process and outcome evaluation and a clearly defined program theory can best be understood by looking at a worked example. Worked examples describe a problem and the problem‐solving approach to arrive at a final solution with a focus on both the outcome and the process.57, 58, 59, 60, 61 Process‐oriented worked examples centre on providing an explicit explanation of “how” and “why” certain steps were taken and/or can help explain how they may have contributed to an observed solution.57, 58, 59, 60, 61 Here, we present a worked example from Canadian family medicine residency training of a theory‐based program evaluation carried out concurrently with implementation. In addition to illustrating the connections between program theory and evaluation, this worked example shows how uncovering findings to support regular updating of a program theory can facilitate ongoing continuous program improvement. Finally, this worked example can act as a template for individuals looking to design program evaluation for their CBME innovations or other curriculum reforms.

Background

The College of Family Physicians of Canada (CFPC) is the accrediting and certifying body for the discipline of family medicine in Canada. In 2010, the CFPC transformed their national family medicine residency education curriculum and assessment guidelines to align with the core components of CBME.62 The CFPC's version of CBME is called the Triple C Competency‐based Curriculum (Triple C). A detailed description of Triple C can be found in the Triple C Competency‐based Curriculum Reports 1 and 2.63, 64

Summary of program evaluation approach

The development of the program theory of Triple C initially focused on identifying how family medicine residency education should be improved by using a social accountability approach to: “address the priority health concerns of the community, region, and/or nation (Canada).”65 The CFPC consulted with different stakeholders conducted data reviews, and through expert working groups, used a consensus building approach with the academic community to define the family medicine residency education reform to be implemented nationally in Canada from 2010 onwards.63 CFPC offered Triple C as a guide for use by residency programs as the CFPC had not yet incorporated Triple C into accreditation standards.63 It was anticipated that adaptation would occur and learnings would be generated that could inform the program theory captured in the program evaluation plan, which would further enhance family medicine's curriculum reform. Figure 1 illustrates the original program theory of Triple C, which is centred on improving the quality of medical education to reach a long‐term goal of increased access to and improved quality of care provided by family physicians.

Figure 1

Original Program Theory of Triple C (abridged)

Theory‐based evaluations use a logic model to describe key activities and include anticipated short‐ and long‐term outcomes specific to various CBME stakeholders. Figure 2 describes the CFPC's logic model with outcomes hypothesized for family medicine residents, graduates, faculty, the overall discipline, and the CFPC as the accrediting body. (Figure 2)

Figure 2

Logic model for the Triple C based on strategies, assumptions, and desired results from the program theory

The logic model (Figure 2) acted as a map to identify opportunities for data collection, which would be incorporated into the evaluation plan that included eight areas of focus evaluating both process and anticipated outcomes (Figure 3). Findings have been used to refine and revise the CFPC's next iterative cycle of curriculum renewal. Learned lessons enhance the original program theory based upon lived experience.

Figure 3

The Triple C Program Evaluation Plan focusing on implementation, process, and outcome evaluation

Program evaluation: Testing assumptions and revising the original program theory

In Table 1, we illustrate the process of a theory‐led program evaluation plan using the original Triple C program theory and subsequent updating based on data collected. The data sources are listed with references shared for those interested in more information. Assumptions from the original program theory and the data source(s) for program evaluation of those assumptions are presented below, along with some examples of evaluation findings and the subsequent refinements to the original program theory that resulted. We do wish to emphasize that this is an illustrative example focusing on specific short‐term outcomes, and not an exhaustive list of all areas of evaluation included in the full evaluation of the Triple C.

Table 1

Program evaluation process showing assumptions from the original program theory, data collection methods, and revisions of the program theory based on findings specific to short‐term outcomes

Original Program Theory of Triple C	Data Sources	Updated Program Theory of Triple C based on findings
Assumption1 : the CFPC's advancement of a CBME approach to family medicine residency education through policies (accreditation and certification standards) and support offered to residency programs will enable successful adoption of Triple C across Canada	Residency Program Implementation Profile (RPIP)a Qualitative Understanding and Evaluation Study of Triple C (QUEST) Studyb	The advancement of Triple C benefited from a nondirective vertical core approach which encouraged uptake from early adopters even before accreditation standards were implemented specifically for Triple C. Ongoing implementation support by the CFPC was felt to be needed for adopters at later stages. Collaborative co‐creation with stakeholders supports adoption. Effective communication with all program leaders is imperative.
Assumption 2 : uptake of Triple C by residency programs will vary depending upon external factors (eg, provincial policies related to family medicine education/practice and medical education culture) and internal factors (eg, faculty engagement, learner demographics, medical school/residency leadership, and infrastructure support)	Residency Program Implementation Profile (RPIP) Qualitative Understanding and Evaluation Study of Triple C (QUEST) Study   Family Medicine Longitudinal Survey (FMLS)c	Flexibility of strategies to implement core features of Triple C increased program autonomy and ownership, and this increased adoption. Collective sense of accountability to learners and patients supported timely implementation Protected resources for Triple C reduces strain of implementation processes. Sharing processes and successful strategies increases efficient and effective use of resources and motivation for change. Differing interpretations of concepts in Triple C challenges implementation.
Assumption3: if family medicine trainees experience Triple C, graduates will choose to practice comprehensive family medicine, will choose to work in diverse communities that may be traditionally underserved, and will be able to self‐assess and address ongoing learning needs.	Family Medicine Longitudinal Survey (FMLS)c p>/p> Pre‐Triple C National Physician Survey (NPS; 2010)d	Ongoing evaluation of processes and outcomes uncovered areas requiring action, such as the need for increased learning experiences in certain clinical domains and/or settings, which facilitates dynamic and rapid continuous quality improvement of Triple C. Program‐specific data provided to program directors can be used to undertake local continuous quality improvement.

Note: The full report: Hamza, DM., Oandasan, I., on behalf of the Program Evaluation Advisory Group. Triple C Competency‐based Curriculum: Findings Five Years Post‐Implementation.

Abbreviations: CBME, competency‐based medical education; CFPC, College of Family Physicians of Canada.

Residency Program Implementation Profile (RPIP, 2015): self‐report from programs of their triple C implementation.

Qualitative Understanding and Evaluation Study of Triple C (QUEST) Study: qualitative study carried out in 2016 that examined personal experiences of Program Directors, Department Chairs, administrative support personnel, Postgraduate Deans, and residents related to Triple C implementation.

Family Medicine Longitudinal Survey (FMLS, 2010‐2017): Self‐report survey administered to family medicine residents at entry to program and at graduation from program across 16 family medicine programs in Canada.

Pre‐Triple C National Physician Survey (NPS; 2010): Pan‐Canadian self‐report survey administered to physicians in practice. Questions specific to scope of practice of family physicians were used as pre‐Triple C controls.

The CFPC's advancement of a CBME approach to family medicine residency education through policies (accreditation and certification standards) and change management support offered to residency programs will enable successful adoption of Triple C across Canada.

Family Medicine Longitudinal Survey (FMLS)c p>/p>

Findings from the Residency Program Implementation Profile (RPIP, self‐report survey completed by program directors) indicated that all had implemented Triple C across their programs but timing varied.66 Some early adopters had most core elements of Triple C in place within a year or two of the introduction of this training model, while other programs took several more years to implement Triple C.66 Findings from the Qualitative Understanding and Evaluation Study of Triple C (QUEST, qualitative study carried out in 2016 examining experiences of Program Directors, Department Chairs, administrative support personnel, postgraduate Deans, and residents through semi‐structured interviews and focus groups) uncovered more information related to factors that influenced uptake of Triple C.67 Early adopters of Triple C reported that the CFPC was helpful in providing support for Triple C through the provision of guides, as well as facilitating opportunities for the sharing of program experiences and lessons learned.66, 68 The co‐creation approach taken by the CFPC, collaborating with family medicine program directors and chairs from 2000 to 2010,63, 66, 69 also helped increase uptake. While early adopters felt that the CFPC provided adequate support, late adopters, particularly with new program leaders who were not involved in the co‐creation of Triple C, felt that the CFPC should have provided more directive and on‐going support.66, 68

Updating the original program theory

In Assumption1, it was hypothesized that CFPC policies and change management support would facilitate Triple C uptake. However, the anticipation of accreditation changes even before the standards were released in 2017 was enough to enable change. The findings suggest that stakeholder involvement in the development of Triple C, especially program leadership, helped to facilitate implementation. The CFPC's guides and early sharing across programs were useful, but later adopters who were not as involved felt the CFPC could have done more. Effective communication and longitudinal change management strategies to support leadership shifts are a key learning for future use.

Uptake of Triple C by residency programs will vary depending upon external factors (eg, provincial policies related to family medicine education/practice and medical education culture) and internal factors (eg, faculty engagement, learner demographics, and medical school/residency leadership and infrastructure support).

Findings from the RPIP indicated that the timing of Triple C implementation varied greatly across the 17 university‐based programs situated across 10 provinces in Canada and across teaching sites.66 Although eventual uptake was identified, the time variability was a point of interest. Findings from the Family Medicine Longitudinal Survey (FMLS 2010‐2017, self‐report survey administered to family medicine residents at entry to and at exit from residency across 16 family medicine programs in Canada) identified that the majority of residents had experienced comprehensive and continuity of care experiences centred in family medicine as well as competency‐based programmatic assessment elements specific to Triple C.66, 70 The QUEST study reported that programs with supportive administrative infrastructures, effective communication networks, protected resources (eg, human and financial capital and time), and who had engaged in consultative approaches with stakeholders (eg, postgraduate deans and ministries of health) had smoother Triple C transitions.66, 68 Those who were simultaneously undergoing accreditation reviews and/or expansion of training sites were able to include changes needed to transition to Triple C, capitalizing on pre‐existing funding and leadership support.66, 68 Certain factors challenged implementation, such as differing interpretations or definitions of the core components of Triple C.66, 68 Consistent shared mental models amongst all those involved in advancing Triple C were developed through ongoing communication and collaboration between stakeholders such as administrators, program advisors, curriculum designers, and postgraduate deans. These shared mental models helped to drive and support implementation66, 68 and facilitated uptake. Additionally, faculty development for all preceptors coaching family medicine residents was identified unilaterally as an area requiring attention.66, 68

Findings from the RPIP and from the FMLS identified that programs had implemented Triple C. The second assumption noted that both external and internal factors would influence the uptake of Triple C but it did not offer many specifics. Although all programs implemented Triple C, the variability related to time was further understood from the QUEST study. This has prompted the CFPC to consider how best to understand how and if provincial governments and external policies influence residency education and residency education reform. Because of the lack of information that was gathered in the program evaluation about this issue, the CFPC is actively exploring how to study these external influences more fully. In terms of the internal factors influencing uptake, it was interesting to hear how programs that took advantage of existing reform processes, accreditation reviews, and new resources to embed Triple C transitioned more easily. This offers new insight on change facilitators.

If family medicine trainees experience Triple C, then graduates will choose to practice comprehensive family medicine; will choose to work in diverse communities that may be traditionally underserved; and will be able to self‐assess and address ongoing learning needs.

Findings from the FMLS: Residents reported an increase in learning and practice experiences reflecting the Triple C vision after completing their training program.66 Findings also indicated a few gaps in learning and practice experiences to be addressed by programs and the CFPC.66

Findings from the National Physician Survey (NPS 2010, Pan‐Canadian self‐report survey administered to physicians in practice. Questions specific to scope of practice of family physicians were used as pre‐Triple C controls): the NPS illustrated a significant increase in residents' intention to practice comprehensive care in multiple clinical domains and settings by comparing pre‐ and post‐Triple C cohorts.66, 70 Responses from residents also illustrated decreased intentions to practice in certain clinical domains and settings,66, 70 which has prompted the need for additional evaluation and research to understand factors that drive these findings.

The demonstration over time that sharing FMLS results back to programs was helpful in implementation of Triple C reinforced the need for program evaluation to run concurrently with implementation. While the initial program evaluation plan was developed to help the CFPC to understand what worked, what did not, and what needed to be changed about Triple C, it was evident early on that the program evaluation data were also immensely beneficial to programs in carrying out local improvement of their curriculum and assessment.

DISCUSSION

Using the Triple C Competency‐based Curriculum as a worked example of CBME implementation, we have demonstrated how to develop and update a program theory through theory‐based evaluation that looks at both processes and outcomes. This process begins by identifying the challenge or problem, determining the needs and assets of the community, strategies to counter the challenge or problem, and the assumptions that are held that link the strategies to the desired outcomes or results. Influential factors that may impact the problem and outcomes, such as societal needs, are also identified in the development of an original program theory. Once the program theory is developed, the features that should be included in a logic model become evident, such as inputs and activities to support implementation of CBME, process of enacting implementation, and how these initial investments are anticipated to lead to short‐ and long‐term outcomes. We have also illustrated how a program theory and logic model can guide the selection of evaluation methodologies and data collection methods to effectively answer questions about the process and outcomes of implementing new training models in medical education.

While we have presented some examples of the program evaluation process in our worked example, we have also explored how the findings that emerged from our study could be interpreted and used for further improvements for residency program implementation. The CFPC's use of a program evaluation alongside implementation, modelling “collection and action” that uses data on an ongoing basis to inform curriculum implementation and reform, has had multiple benefits to family medicine residency programs in Canada. It is clear that the process and outcomes of a well‐designed evaluation can play a significant role in driving ongoing change and ongoing residency education improvement. Modelling collection and action encouraged engagement in evaluation and research since participants (ie, faculty, teachers, administrators, and residents) had tangible evidence that their efforts supported the growth of Triple C. The outcomes of the program evaluation played a significant role in driving change to improve the future of this discipline.

CONCLUSION

Deliberately pairing evaluation alongside change—such as was done in this CBME education transformation—yields invaluable information from the experiences of those implementing and experiencing a new innovation. Findings “from the field” reinforce the notion that social processes and mechanisms play a significant role in the ways in which a program is experienced and implemented and in how outcomes may differ as a result of these otherwise hidden factors. In addition, findings from program evaluation can support the improvement of current processes, as well as guide future implementation by shedding light on lessons learned and supporting effective and efficient use of resources. Evaluation and the development of an updated program theory also facilitate the introduction of new changes and theories that build on findings, which supports the desired goal of contributing to cumulative science rather than “reinventing the wheel.”

Future studies may benefit from ongoing longitudinal evaluation to identify trends in factors that drive, support, and/or challenge change over time and if, how, and why these factors shift over time.

CONFLICT OF INTEREST

The authors declare no conflict of interest.