What Do Reinforcement and Confidence Have to Do with It? A Systematic Pathway Analysis of Knowledge, Competence, Confidence, and Intention to Change.

The outcomes model most applied in continuing education for the health professions evaluation is Moore and colleagues' conceptual framework. Examination of how the levels interact and the role of confidence and intention to change can help outcomes professionals understand better how to impact clinician practice and conductand report outcomes studies. The current study examined the relationships among knowledge and competence change, confidence change, and intention to change across 57 online oncology certified education programmes published from 2018 to 2020 on Medscape.org. Findings indicate that not only improvement in knowledge and competence but also reinforcement of knowledge and competence are significant predictors of changes in confidence. They also indicate that knowledge and competence influence intention to change through confidence.

Introduction

Social and behavioural psychology theories and research point to the importance of confidence in behaviour change. This is reflected in social cognitive theory [1], the transtheoretical model of change [2], and identity-based motivation theory [3]. In education, confidence is a target of intervention as well as building one’s knowledge and competence around a concept [3]. One may be equipped with the right information and know how to use that information, but without confidence in the ability to enact that information, the associated behaviour may not take place [4]. Moore and colleagues specified both knowledge and competence as important outcomes from continuing professional development (CPD) [5]. Specifically, knowledge is reflective of knowing and knowing how, and competence is showing but can also be assessed with intention to change which is thought to be a precursor to behaviour change. Confidence is not a part of the Moore and colleagues’ model, but based on the wealth of research supporting the importance of confidence in behaviour change, it is included in the current study.

In their seminal paper, Moore and colleagues specify “… a conceptual framework represents is a collection of variables and events that might interact in some way to produce something” [5]. The relationship between confidence and intention to change has been supported in CPD research [6]. Additionally, links between knowledge and competence and confidence have been found [7,8]. The goal of this research was two-fold: 1) to assess the relationship between knowledge/competence and confidence and 2) to examine the relationships among knowledge/competence, confidence, and intention to change. The focus is on oncology because it is an area where information moves at a rapid pace, and CPD is quite necessary in helping oncologists and hematologist/oncologists keep current in their practice [9].

Recently, Lucero and Dunn presented survey research supporting that education is not only meant to improve, but also reinforce knowledge/competence for behaviour change to occur [4]. Of 6105 physicians who participated in one of six online CPD activities, 87% reported experiencing reinforcement, and 65% agreed that reinforcement was important to their use of evidence-based practices [4]. The current study extends the prior research by examining the role of reinforcement vs. improvement of knowledge/competence on confidence and intention to change. It is hypothesised that both improvement and reinforcement are positively associated with confidence and intention to change post-CPD. In addition, confidence will mediate the relationship between knowledge/competence and intention to change.

Methods

Ethical Considerations

This study was exempt from institutional review board approval as it is educational research and is exempt under 45 CRF 46.104(d)(1) according to the US Department of Health and Human Services[https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts-2018/index.html#c2].

Sample

Data used in the current study are from all online oncology continuing medical education (CME)/CPD-certified programmes launched between 08/2018 and 07/2020 on Medscape.org/Oncology. Programmes range in length from 15 minutes to 1 hour. Other selection criteria include 1) being focused on breast cancer (including advanced and metastatic breast cancer), chronic lymphocytic leukaemia, or lung cancer, 2) having a focus on improving Oncologists’ and/or Haematologists/Oncologists’ (herein referenced as Hem/Oncs) knowledge, competence, and confidence in one of the three clinical areas, 3) having repeated outcomes assessment on knowledge and/or competence, confidence pre- and post-education. The focus was on summative assessment – the assessment of learning at the conclusion of an activity [10]. A total of 57 programmes (17 breast cancer programmes, 9 CLL programmes, and 31 lung cancer programmes) were included in the analysis.

A total of 29,806 Hem/Oncs were learners (viewed the content of the programme at the URL) in the 57 programmes; 8527 Hem/Oncs completed both pre- and post-outcomes assessments (29% response rate). This is the sample for the first research question. Of the 8527 Hem/Onc outcomes assessment completers, 4746 completed post-education evaluation which assessed programme satisfaction and planned practice changes. This is the sample for the second research question.

Measures

The following measures were included in the analysis of the current study.

Covariates

All analyses included clinical areas and years in practice as covariates to control for their effects in the models. Years in practice was treated as a continuous variable in all models (Mean = 28, SD = 17) and was computed by subtracting Hem/Oncs’ medical school graduation year from their Medscape member profile from the year they completed the outcomes assessment for the current study. Mean imputation was used to code missing values as 28 years for this variable. In addition, two dummy variables were created for CLL and lung cancer programmes treating breast cancer programmes as the reference group for learning topic for analysis.

Overall Knowledge/Competence Change Status

To assess the CME/CPD impact, three questions measuring knowledge and/or competence were asked both prior to and after the exposure to the education content for all 57 programmes included in analysis. For this analysis knowledge and competence questions were examined together as “knowledge/competence”. Overall knowledge/competence change statuses were calculated at the individual learner level for each of the activities (see Table 1 for definition). In analysis, two dummy variables were created for the improved (n = 3640) and reinforced (n = 4223) groups, and Hem/Oncs who did not answer any of the questions correctly after education were used as the reference group for knowledge/competence improvement status (n = 664) to allow for significance tests in the differences between the improved and reinforced groups with those who were unaffected by the programme.

Table 1.

Definition of knowledge/competence change status at the activity level for each individual learner.

Step 1. Defining knowledge/competence change status at the individual question level for each of the activities.
Status		Pre choice	Post choice
Improved		Incorrect	Correct
Reinforced		Correct	Correct
Step 2. Defining knowledge/competence change status at the activity level based on individual question change status.
Status	Definition
Improved	Improved on one or more questions
Reinforced	Did not improve on any of the questions but reinforced on one or more questions
Unaffected	Did not improve and were not reinforced on any of the questions

Confidence

All programmes had a question assessing global confidence related to one or more of the activity learning objectives repeated before and after education (e.g. How confident are you in identifying pneumonitis in a patient with stage III non-small cell lung cancer?). Responses were rated on a 5-point Likert-type scale ranging from 1 = not confident to 5 = very confident.

Post-Education

To test changes in confidence before and after education, a dummy variable was created where 0 = pre-education and 1 = post-education.

Intention to Change

For all programmes included in analysis, activity completers were asked to report whether they plan to make one or more changes to their practice (e.g. modification to treatment plans, changes to screening/prevention practices, etc.) and their level of commitment to making the change(s). As more than 90% of the Hem/Oncs reported that they were planning to change their practice, leaving very little variance to be explained in the models, we used responses to the evaluation question on commitment to change as a measure of intention to change in analysis to capture both intention to change as well as commitment to change. For analytical purposes, a dummy variable was created where 1 represents Hem/Oncs who not only planned to but were also very committed to make practice change(s) and 0 represents Hem/Oncs who did not plan to make a practice change or were somewhat or not very committed.

Statistical Analysis

All analyses in the current study were performed using IBM SPSS Statistics 25.

Multilevel regression analysis [11] was conducted to account for the correlations between the pre- and post-responses for each individual Hem/Onc.

A linear regression model was fitted as the initial step to examine the relationship between knowledge/competence status and post-education confidence. The mediating effect of post-education confidence in the relationship between knowledge/competence status and intention to change was then tested using a set of logistic regressions.

Results

Preliminary Analysis

Of the 8527 Hem/Oncs included in the analysis, 43% improved and 50% reinforced their knowledge/competence. Mean confidence was higher at post (Mean = 3.14, SD = 1.11) vs pre (Mean = 2.77, SD = 1.18) education. Results from the multilevel regression analysis indicated that controlling for Hem/Oncs’ years in practice and clinical areas of the programmes, increase in confidence remained statistically significant (b =.38, P < .001; see results from Model 1 in Table 2). Of the 4746 Hem/Oncs, 59% were very committed to making practice change(s).

Table 2.

Multilevel regression coefficients predicting confidence (n = 8527).

	Model 1		Model 2
	b	SE	b	SE
Intercept	2.779***	.021	2.239***	.047
Years in practice	−.005***	.001	−.005***	.001
CLL	.092**	.035	.118***	.034
Lung cancer	−.048	.026	−.029	.026
Post education	.378***	.009	.154***	.031
Improved			.428***	.047
Reinforced			.692***	.047
Improved*Post education			.367***	.034
Reinforced*Post education			.138***	.033

** P <.01

** P <.001

Changes in Confidence from Pre- to Post-education Differ Depending on Education’s Impact on Knowledge/competence (ie, Knowledge/competence Change Status)

Results from multilevel regression analysis testing the interaction effect between knowledge/competence and confidence change status revealed statistically significant differences in confidence change from pre- to post-education among Hem/Oncs with improved (b = .14, P < .001) and reinforced knowledge/competence (b = .37, P < .001) versus those who were unaffected by the programmes (see results from Model 2 in Table 2). Specifically, while confidence increase was significant for all three Hem/Onc groups, the changes in confidence were 90% and 238% higher for Hem/Oncs who reinforced and improved knowledge/competence, respectively, as compared to those who were unaffected (see Figure 1).

Figure 1.

Changes in mean levels of confidence by knowledge/competence change status controlling for years in practice and programme type.

Confidence Mediates the Relationship between Knowledge/Competence and Intention to Change Practice

A linear regression model was fitted to examine the relationship between knowledge/competence change status and post-education confidence controlling for years in practice and educational topic. It was found that Hem/Oncs with improved (b = .79, P < .001) and reinforced (b = .83, P < .001) knowledge/competence both had significantly higher levels of confidence after education compared with those who were not affected by the CPD. Specifically, mean levels of confidence after education were 32% and 34% higher among the improved and reinforced group compared with the unaffected group, respectively.

Results from logistic regression found that Hem/Oncs with improved (b = .49, P < .001) and reinforced (b = .42, P < .001) knowledge/competence are both significantly more likely to be very committed to making practice changes compared with those who were not affected (see Table 3). Specifically, Hem/Oncs with reinforced (adjusted probability: 61%) or improved (adjusted probability: 63%) knowledge/competence are 24% and 20% more likely than those who were not affected (adjusted probability: 51%) to be very committed to making practice changes. Follow-up analysis found no significant differences between the improved and reinforced groups in the likelihood of being highly committed to practice changes.

Table 3.

Logistic regression coefficients predicting intention to change (n = 4746).

	Model 1			Model 2
	b	SE	Exp(b)	b	SE	Exp(b)
Intercept	.029	.127	1.029	−.885	.148	.413
Years in practice	.003	.002	1.003	.006**	.002	1.006
CLL	−.048	.088	.954	−.084	.090	.920
Lung cancer	−.120	.067	.887	−.117	.068	.889
Improved	.490***	.127	1.632	.210	.131	1.234
Reinforced	.402***	.126	1.495	.106	.131	1.112
Post confidence				.373***	.030	1.452

** P <.01

** P <.001

Interestingly, when the impact of post confidence on intention to change was taken into account (see Table 3), the difference between Hem/Oncs with improved and reinforced knowledge/competence with those who were not affected became non-significant, and post-confidence significantly predicted intention to change even after controlling for change status of knowledge/competence (b = .37, P < .001). Specifically, Hem/Oncs who reported being very confident (i.e. who rated themselves as 5 on a scale of 1 to 5) are 97% more likely to be very committed to making changes in practice than those who reported being not confident (who rated themselves as 1 on a scale of 1 to 5). These findings along with the significant relationship between knowledge/competence with confidence suggest that confidence mediates the influence of knowledge/competence on intention to change practice. In other words, knowledge/competence has an impact on intention to change practice, specifically commitment to change, via its influence on confidence (see Figure 2).

Figure 2.

Confidence mediating the relation between knowledge/competence change status and intention to change.

Discussion

Overall, the results from the current study help specify the relationships among the Moore and colleagues’ conceptual framework [5], which furthers the ability to outline a theory of change for impacting clinical behaviour through education. The results from this study lend support to research that has shown that reinforcement of information through education does impact practice [4,12] by providing evidence that reinforcement results in increases in confidence which then results in commitment to change practice. Although improvement in knowledge/competence results in greater increases in confidence than reinforcement, confidence post-CPD for both groups was not significantly different. The impact of post-confidence on commitment to change for both groups was also not significantly different.

Building on the research that examined motivation to change, confidence (self-efficacy), and global intent to change and found evidence to suggest motivation to change mediates confidence’s impact on intention to change [8], the current study provides further evidence that confidence does contribute to practice change. This study also suggests that post-confidence differentiates being not or somewhat committed to change from being very committed to change. Research that further disentangles declaring an intention to change a practice and commitment to change practice as predictors of actual behaviour change will help provide further information on whether intention to change practice is a valuable outcome or that it is most valuable if combined with commitment to change.

Furthermore, real-world performance or practice was not an outcome considered in this study but is important to include in future research. Research outside of healthcare and CPD suggests that confidence is a predictor of behaviour [1,3], but it is particularly important to examine linkages between confidence and real-world behaviour resulting from online education given the shift to digital in CPD during the COVID-19 pandemic. Research has shown that commitment to change resulting from CPD (albeit not online) is associated with objective measurement (medical record documentation) of real-world change [13]. Conversely, other studies with a smaller sample size have not found evidence to support commitment to change and confidence predict self-reported change. More research is needed in this area. Investigating whether commitment to change mediates the relationship between confidence and real-world practice will also be an important relationship to examine to fully understand the mechanism(s) of change for clinician behaviour. Therefore, online CPD can be designed to focus on what it can most immediately impact and can be measured (i.e., knowledge, competence, confidence, and commitment to change) with some additional confidence that it is likely to impact practice.

One limitation of the current study is the data could not be parsed systematically into separate knowledge and competence outcomes – meaning we could not investigate the differential impact of knowledge and competence separately on confidence and intent to change. This also will be important for future research to examine – does reinforcement or improvement of knowledge play a different role than reinforcement or improvement of competence in predicting confidence? Another limitation is that the data are limited to those who completed the assessments. Finally, this study only focused on Hem/Oncs and outcomes from oncology and hematology/oncology programmes. However, there is a strong likelihood that the results are generalisable for online CPD activities that are 15 to 60 minutes in duration for this audience because the results represent 57 programmes and over 4000 Hem/Oncs. It is the largest single empirical study examining the impact of CPD published to date in this area.

Despite limitations, this study makes important contributions to our understanding of the relationships among knowledge/competence, confidence, and intent to change, specifically commitment to change. It also makes the important distinction between reinforcement and improvement in knowledge/competence and provides strong evidence that reinforcement is associated with significant increases in confidence which lead to a stronger commitment to change. Implications for the CPD community are: 1) evaluators should measure reinforcement of knowledge/competence and improved confidence if reflected in the learning objectives; global confidence related to a clinical practice assessed using a Likert-type scale of 1 to 5 has construct and criterion-related validity, and 3) funders and providers should consider learning objectives that seek to reinforce information or current practice and build confidence to be desirable goals of CPD (of course, assuming the practice gaps and educational needs point to these being important to target).