A Qualitative Evaluation of an Online Expert-Facilitated Course on Tobacco Dependence Treatment.

Qualitative evaluations of courses prove difficult due to low response rates. Online courses may permit the analysis of qualitative feedback provided by health care providers (HCPs) during and after the course is completed. This study describes the use of qualitative methods for an online continuing medical education (CME) course through the analysis of HCP feedback for the purpose of quality improvement. We used formative and summative feedback from HCPs about their self-reported experiences of completing an online expert-facilitated course on tobacco dependence treatment (the Training Enhancement in Applied Cessation Counselling and Health [TEACH] Project). Phenomenological, inductive, and deductive approaches were applied to develop themes. QSR NVivo 11 was used to analyze the themes derived from free-text comments and responses to open-ended questions. A total of 277 out of 287 participants (96.5%) completed the course evaluations and provided 690 comments focused on how to improve the program. Five themes emerged from the formative evaluations: overall quality, content, delivery method, support, and time. The majority of comments (22.6%) in the formative evaluation expressed satisfaction with overall course quality. Suggestions for improvement were mostly for course content and delivery method (20.4% and 17.8%, respectively). Five themes emerged from the summative evaluation: feedback related to learning objectives, interprofessional collaboration, future topics of relevance, overall modifications, and overall satisfaction. Comments on course content, website function, timing, and support were the identified areas for improvement. This study provides a model to evaluate the effectiveness of online educational interventions. Significantly, this constructive approach to evaluation allows CME providers to take rapid corrective action.

Introduction

Online learning was introduced with the advent of the Internet in the early 1990s and has been increasingly used in medical and health care education.[1] As the number of online courses continues to grow, the need for quality and accountability in course evaluations is also increasing.[2] A meta-analysis in health professions education concluded that online learning is associated with a large positive effect in outcomes such as satisfaction, knowledge, skills, behaviors, and impact to patients compared with no intervention, and effectiveness is similar to that of classroom-based learning methods.[1] While the results of online and in-person trainings are not significantly different, online evaluation response rates typically fall below that of classroom-based response rates.[3,4] Evaluating online learning remains a challenge, and there is a lack of general consensus on the best way to evaluate online learning and use the evaluation data to support quality improvement.[2,5]

Online courses might be an acceptable alternative to training a diverse workforce spread across great distances and time zones.[6] It is usually time intensive and cost prohibitive to offer in-person intensive training courses to health care providers[1] (HCPs) where there is a large “know-do” gap in some specific areas such as tobacco dependence treatment. Moreover, quality improvement of course offerings requires collecting and analyzing feedback from participants, and using this feedback to modify the experience and content for future learners.

The Training Enhancement in Applied Cessation Counselling and Health (TEACH) Project is the first university-accredited continuing medical education certificate program in Canada focused on tobacco dependence treatment. This project has been highly successful in developing an interprofessional, collaborative, and experiential education program, training more than 5000 HCPs to date. The TEACH curriculum consists of three courses: (1) an online introductory course on tobacco control pillars (10.5 hours), (2) a classroom-based or online Core Course on the fundamentals of tobacco dependence treatment (19.5 hours), and (3) a classroom-based or online Specialty Course focused on a variety of populations or topics where tobacco dependence treatment requires adaptation (13.5 hours). Course evaluation is a critical component to quality assurance and improvement. The TEACH Project’s comprehensive evaluation framework[7] includes course evaluations completed by learners following each course module (i.e., formative evaluation) in addition to a summative evaluation immediately administered post training. Learners also complete a follow-up evaluation at 3 and 6 months post training.

To create a learner-centered program that supports rapid experiential learning[8] and expands reach by building capacity among HCPs unable to travel for classroom-based training, TEACH courses have been adapted for the online environment, utilizing best practice principles in e-Learning. Despite administering comprehensive quantitative surveys, a qualitative evaluation was designed to inform quality improvement of the TEACH online Core Course.[7] Data obtained from written questionnaires with predefined answers may be less contextually rich than that obtained through qualitative approaches[9] (i.e., free-text or open-ended comments). With qualitative response options, participants can respond to the questions as they would like to answer them and the researcher can investigate the meaning of these responses. However, we recognized that the qualitative feedback may not be representative of the all HCPs who participated in the online Core Course, and therefore, quantitative data from an online survey were also obtained.

The goals of this article are to (1) introduce curriculum designers to an innovative use of qualitative methods in both formative and summative course evaluations and (2) describe the qualitative findings elicited from TEACH-trained HCPs following their participation in the TEACH online Core Course for program evaluation.

Methods

This article reports thematic analysis of HCPs’ opinions expressed in feedback data from four cohorts of the TEACH online Core Course. These data were collected in January, May, July, and October 2015 during and after the course was completed. HCPs were asked to provide free-text comments after each of 10 modules (i.e., formative) and respond to a series of open-ended questions following the end of the course (i.e., summative), in addition to quantitative questions. Demographic information, professional discipline, and number of years providing tobacco cessation interventions were also collected. At the time this study was designed, the Centre for Addiction and Mental Health Research Ethics Board deemed that formal review and approval was not required for our study.

In this phenomenological study, we used a hybrid process of inductive and deductive thematic analysis to interpret the free-text comments from the formative evaluations and open-ended questions from the summative evaluations. Inductive codes were based on prominent themes that emerged from HCPs’ free-text comments. Our deductive codes were based on concepts that we explicitly asked HCPs about (e.g., what overall modifications to the course do you suggest) through open-ended questions, in which we searched for attributes of Henri’s model.[10] This strategy allowed us to examine our study questions (i.e., through our questions in the summative phase and through deductive codes) while capturing other key themes that emerged from the data (i.e., free-text comments through inductive codes that allowed the data, rather than theory, to drive coding).[11] We used the Braun and Clarke model[12] for our thematic analysis as it is a flexible approach that can be used across a range of research questions applying the following six phases: (1) Free-text comments and responses to open-ended questions were read and reread to familiarize the researchers with the data; (2) a range of initial codes were generated to index common features across comments; (3) these initial codes were examined to determine prevalent themes; (4) themes were reviewed for internal homogeneity and external heterogeneity, to combine similar themes into overarching themes and draw coherent links or distinctions between them; (5) the overarching themes were firmly defined, organized in relation to their collated data extracts, and then analyzed for subthemes; and (6) the report was produced.

Several steps were taken to ensure methodological and analytical rigor in this investigation. Two independent raters (A.E.A. and M.B.) coded the transcripts. Before the analysis of the data, the raters met to discuss any discrepant codes until consensus was met in regard to the final codes. A quarter of the data were double-coded to check for consistency, for which there was >90% agreement between researchers (A.E.A. and M.B.). To minimize the risk of bias, the prominence of main themes and their respective subthemes was determined through discussion between two authors. These authors read the free-text comments and answers to open-ended questions as well as reviewed all coded quotations for each theme. Unanimity among the two coders was required. When unanimity was not achieved, comments were discussed until consensus was reached. Quotes are presented in the results to illustrate the themes and subthemes identified. To validate the findings, first hand quotes are presented word-for-word within the results in Table 2.[13] Also, further evidence of validity was obtained by comparing emergent finding with the literature. The qualitative software QSR NVivo 11 was used to perform content analysis of the themes derived from the comments.

Table 2.

Themes and Samples of 349 Comments Reported in Free-Text Comments in the TEACH Formative Evaluation.

Theme (cohort)	No. of comments (%)	Samples
Overall quality 88 (25.2)
Positive feedback	79 (22.6)
Cohort 1	41 (15.9)	The information is easily digestible and presented in a manner that is dynamic and engaging
Cohort 2	12 (8.1)	Overall constructive for me
Cohort 3	18 (10.5)	Clear and concise
Cohort 4	8 (11.0)	Having had no prior experience in tobacco cessation, I learned a great deal from this module
Suggestions for improvement	9 (2.6)
Cohort 2	3 (1.5)	Harder than I expected to really grasp
Cohort 3	1 (0.4)	I found the module rather unclear
Cohort 4	5 (9.2)	I have been an addiction counselor for 27 years; there is not that much difference although could be more intense
Content 120 (34.4)
Positive feedback	49 (14.0)
Cohort 1	28 (12.7)	There is no wasted information; it’s all important and I can see its relevance
Cohort 2	7 (11.4)	I am enjoying the multifaceted approach for assisting clients with smoking cessation
Cohort 3	6 (3.1)	Extremely helpful in starting my first client on pharm for smoking cessation
Cohort 4	8 (9.0)	Videos and chats in particularly were helpful
Suggestions for improvement	71 (20.4)
Cohort 1	29 (24.9)	Reading from text book had a lot of statistics that I found difficult to retain
Cohort 2	13 (28.7)	Repeat of a lot of the mandatory material that had to be taken before the course started
Cohort 3	20 (33.7)	Further development in terms of the content of this module is required to enhance the individuals’ learning experience
Cohort 4	9 (19.5)	Would have liked it to be longer-more case examples
Evaluation 42 (12.0)
Positive Feedback	1 (0.3)
Cohort 3	1 (2.0)	I appreciate the style of quiz you provide. For example, by giving a case study and then asking quiz questions, I believe you encourage critical thinking rather than simple recall of facts
Suggestions for improvement	41 (11.7)
Cohort 1	13 (14.2)	I was surprised at the length of the quiz
Cohort 2	8 (20.7)	I was very frustrated that I could not find the answer to question 1 in the modules and text book
Cohort 3	15 (19.1)	It seems as if material assessed in the quiz was not presented as part of the readings in the module
Cohort 4	5 (34.2)	I felt that the quiz did not reflect the content in the module
Timing 31 (8.9)
Positive feedback	2 (0.6)
Cohort 1	2 (0.6)	I enjoy working at my own pace and being able to review the materials over again
Suggestions for improvement	29 (8.3)
Cohort 1	14 (9.0)	I would have liked for the modules were opened sooner than at the end of each submission
Cohort 2	1 (0.6)	Could not review it during work hours
Cohort 3	14 (19.8)	Keep all modules open to be done at our own pace instead of limiting start and end times
Delivery method 68 (19.5)
Positive feedback	5 (1)
Cohort 1	4 (1.5)	The course content is very easy to navigate and I like that I can review over if needed
Cohort 2	1 (0.5)	Great facilitation as well! Thanks
Suggestions for improvement	63 (17.8)
Cohort 1)	29 (20.1)	I’m unsure who my faculty member is
Cohort 2	16 (26.6)	In the final screen the videos are teeny tiny
Cohort 3	9 (9.3)	Difficulties with video feed—kept cutting out
Cohort 4	9 (15.5)	The videos tend to be “Choppy” and hard to hear the entire video
Total	349 (100)

Note. Cohort 1: January 2015. Cohort 2: May 2015. Cohort 3: July 2015. Cohort 4: October 2015. TEACH = Training Enhancement in Applied Cessation Counselling and Health.

Results

In total, 96.5% of participants completed the course evaluations. A breakdown of the sample for each cohort appears in Table 1. Participants were asked to select their professional discipline. Overall, 44% (n = 124) of the participants were nurses.

Table 1.

Description of the Study Group.

Study group	Cohort 1January 2015	Cohort 2May 2015	Cohort 3July 2015	Cohort 4October 2015	Total No. (%)
	n.(%)	n.(%)	n.(%)	n.(%)
Clinical contact with clients
Yes	67 (81.7)	56 (69.1)	53 (84.1)	42 (87.5)	218 (78.7)
No	12 (14.6)	11 (13.5)	8 (12.7)	4 (8.3)	35 (12.6)
Did not answer	6 (3.7)	14 (17.4).	2 (3.2)	2 (4.2)	24 (8.7)
Years providing intervention
0-5	68 (80)	55 (67.9)	46 (73)	27 (56.3)	196 (70.8)
6-10	5 (5.9)	9 (11.1)	12 (19)	8 (16.7)	34 (12.3)
10+	7 (8.2)	3 (3.7)	3 (4.8)	4 (8.3)	17 (6.1)
Did not answer	5 (5.9)	14 (17.2)	2 (3.2)	9 (18.8)	30 (10.8)
Disciplines
Addictions counselor	4 (4.7)	9 (11.1)	2 (3.2)	2 (4.2)	17 (6.1)
Case manager or public health nurse	8 (9.4)	1 (1.2)	1 (1.6)	4 (8.3)	14 (5.0)
Community health worker	0 (0.0)	2 (2.4)	0 (0.0)	0 (0.0)	2 (0.9)
Dentist/dental hygienist	1 (1.2)	1 (1.2)	1 (1.6)	1 (2.1)	4 (1.4)
General practitioner/family physician	4 (4.7)	4 (4.9)	0 (0.0)	0 (0.0)	8 (2.8)
Health promoter/educator	7 (8.2)	9 (11.1)	4 (6.4)	2 (4.2)	22 (12)
Manager/coordinator	1 (1.2)	2 (2.5)	1 (1.6)	2 (4.2)	6 (2.1)
Nurse practitioner	4 (4.7)	4 (4.9)	5 (7.9)	5 (10.4)	18 (6.4)
Occupational therapist	2 (2.4)	2 (2.5)	0 (0.0)	2 (4.2)	6 (2.1)
Pharmacist	9 (10.6)	9 (11.1)	8 (12.7)	9 (18.8)	35 (12.6)
Registered nurse	27 (31.8)	22 (27.2)	25 (39.7)	11 (22.9)	85 (30.0)
Registered practical nurse	3 (3.5)	2 (2.5)	1 (1.6)	0 (0.0)	6 (2.1)
Respiratory therapist, clinical perfusionist, or asthma educator	6 (7.1)	4 (4.9)	2 (3.2)	1 (2.1)	13 (4.6)
Social worker	9 (10.6)	5 (6.2)	5 (7.9)	3 (6.3)	22 (7.9)
Researcher	0 (0.0)	1 (1.2)	1 (1.6)	0 (0.0)	2 (0.9)
Midwives	0 (0.0)	0 (0.0)	2 (3.2)	1 (2.1)	3 (0.1)
Dietitian/nutritionist	0 (0.0)	0 (0.0)	1 (1.6)	0 (0.0)	1 (0.1)
Aboriginal health worker	0 (0.0)	0 (0.0)	2 (3.2)	1 (2.1)	3 (0.1)
Advanced practice nurse/clinician	0 (0.0)	0 (0.0)	1 (16)	0 (0.0)	1 (0.1)
Treatment attendant	0 (0.0)	0 (0.0)	1 (1.6)	0 (0.0)	1 (0.1)
Kinesiologist	0 (0.0)	2 (2.5)	0 (0.0)	0 (0.0)	2 (0.9)
Other	0 (0.0)	2 (2.5)	0 (0.0)	4 (8.3)	6 (2.1)
Total number of participants registered	85	81	73	48	287 (100)
Total number of respondents (%)	85 (100)	81 (100)	63 (86.3)	48 (100)	277 (96.5)

Respondents (n = 277 out of 287 participants) provided 690 comments through free-text comments and answers to open-ended questions within the study questionnaires, describing their satisfaction with their learning experience and suggestions for improvement of the TEACH online Core Course.

Formative Evaluation

The five central themes identified from 349 comments within the data are presented with their subthemes alongside quotations as examples. Table 2 lists these themes in the following categories: positive feedback and suggestions for improvement. The overall focus of the study was on “suggestions for improvement,” and a high proportion of the comments (61% of comments) provided feedback to improve the following course areas: content of the course (20.4% of comments), delivery system (17.8% of comments), evaluation of the course (11.7% of comments), timing of the course (8.3% of comments), and overall quality (2.6% of comments).

More general positive comments emerging from the data (39% of comments) typically consisted of 1-word responses, such as “good” or “excellent.” Positive responses were grouped into the following themes: overall quality (22.6% of comments), content of the course (14% of comments), delivery method (1% of comments), timing of the course (0.6% of comments), and evaluation of the course (0.3% of comments). The majority of comments made by HCPs in the formative evaluation were positive, expressing satisfaction with overall course quality. Comments on course content and the delivery method were identified as key areas for improvement. HCPs also wanted the course designers to address problems with content flow, repetition, and relevance of concepts.

Summative Evaluation

Five themes emerged from 341 of the participants’ responses to open-ended questions in the summative evaluation administered at the end of each course. The themes, subthemes, and corresponding examples are shown in Table 3.

Table 3.

Themes and Samples of 341 Comments Reported in Open-Ended Questions in the TEACH Summative Evaluation.

Open-ended questions	Identified themes	Comments	Sample of health care providers’ quotes
	Subthemes	n (%)
Tell us about one thing that you have learned and will use in your clinical practice	Learning objective	48 (23.5)
	Motivational interviewing	6 (7.8)	I really enjoyed the module on motivational interviewing because it provided clear examples of different scenarios
	Pharmacological interventions	4 (4.2)	I feel much more confident with regard to pharmacotherapies used to treat tobacco addiction
	Screening and assessing tobacco use	9 (2.6)	I learned about the tools I can use to help with smoking
	Harm reduction	5 (2.0)	I now have the knowledge to provide various harm reduction strategies to clients
	Developing treatment plans	6 (1.7)	Quitting is a process and not a single event. Important to share this with patients as they begin or continue in the process
	The global impact of tobacco use	4 (1.6)	How many “bad” chemicals there are in one cigarette and all the systems that will be affected over the years
	Advocacy and system-level change	6 (1.2)	How media can affect clients in their attempts to quit
	Specific populations with high tobacco prevalence	4 (1.1)	Adapting treatment plans for high-risk populations
	Implementation	2 (0.8)	This is a very powerful module as the course draws to a close
	Facilitating smoking cessation groups	2 (0.5)	I will certainly use in my work as I will be facilitating groups at the center
Please give us your comments on how the interprofessional networking/collaboration was helpful or not helpful	Interprofessional collaboration and networking	11 (12.1)
	Positive feedback	5 (5.4)	Great opportunity to gain perspective from other disciplines, ie, pharmacist knowledge adjusting some psychiatric medications, caffeine with cessation
	Suggestions for improvement	6 (6.7)	There were very few posts from the group so I found it difficult to engage in relevant conversation—other than the facilitator’s input
What is future topics of relevance to you	Future topic of relevance	69 (8.2)
	Mental illness and substance use disorders	7 (2.3)	Tobacco use in mental health and substance abuse
	Tobacco dependence treatment	6 (1.2)	Customized treatment plans
	Tobacco cessation for women across the lifespan	7 (0.9)	Smoking cessation in pregnancy and postpartum
	Motivational interviewing	5 (0.6)	Motivational interviewing strategies
	Implementation	3 (0.4)	Strategies to promote cessation in specific high-risk populations such as CA and COPD clients
	Harm reduction	5 (0.4)	More info on e-cigarettes as it becomes available
	Advocacy and system-level change	4 (0.4)	I would like to learn more about the tobacco industry
	Pharmacological interventions	4 (0.4)	More coverage of pharmacotherapy available and how it works and potential side effects and so on
	Tobacco cessation for youth and young adults	7 (0.3)	Working with youth, youth with mental health disorders or youth currently pregnant
	Tobacco cessation for First Nations, Inuit, and Métis populations	5 (0.2)	Working with specific populations—eg, Aboriginal
	Tobacco cessation and chronic disease	3 (0.2)	Strategies to promote cessation in specific high-risk populations such as CA and COPD clients
	Others: tobacco cessation for specific populations such as immigrant populations, older adults, and so on	13 (0.9)	Smoking cessation and immigrant population
What overall modifications or improvements to the course would you suggest	Overall modification	141 (37.8)
	Content	50 (17.2)	I found that the mandatory evaluation after each module was too much and I don’t feel that a student’s passing a course should be dependent on whether or not they complete these evaluations
	Website function	52 (11.6)	Navigation into each module was a little cumbersome and it was not always clear how to access course content
	Timing	25 (6.0)	Would have preferred if you could do the course at your own pace and not have to complete 2 modules per week as some weeks I could have done more and other week I had more time constraints do to work, summer holidays, and so on
	Support	14 (3.0)	I was not sure how to contact the course instructor directly via the online course if I had a question
General comments about the course	Overall satisfaction	72 (18.4)
	Course usefulness	44 (14.0)	I thought the progression of the content of the course was good, starting with clinical information about nicotine and addiction, which grabbed my attention right from the start
	Course support	9 (2.4)	The support staff was very helpful when I had to delay beginning the course for a week. I really appreciated their help and understanding
	Web usability	16 (1.7)	I liked the course structure—this is the first time I completed an online course and it was easy to navigate
	Timing	3 (0.3)	It was really easy to set the time aside and was pretty relaxing when I got down into the information

Note. TEACH = Training Enhancement in Applied Cessation Counselling and Health.

Theme 1: Learning objectives—23.5% of comments

Participants were asked to comment on one knowledge or skill they learned the most about and were planning to use in their clinical practice. Qualitative responses included the following knowledge or skill areas: motivational interviewing, pharmacological interventions, screening and assessing tobacco use, developing treatment plans, harm reduction, the global impact of tobacco use, advocacy and system-level change, specific populations with high tobacco prevalence, implementation, and facilitating smoking cessation groups. The most commonly reported knowledge or skill areas were motivational interviewing, pharmacological interventions, and assessing tobacco use (7.8%, 4.2%, and 2.6% of comments, respectively). These are aligned with the learning objectives of the course.

Theme 2: Interprofessional collaboration and networking—12.1% of comments

Participants generally had positive comments in addition to suggestions for improvement related to opportunities for interprofessional collaboration and networking during the course. Overall, only 5.4% of the comments under this theme indicated participants were pleased to have opportunities to collaborate with fellow learners.

Theme 3: Future topics of relevance—8.2% of comments

When we asked participants about future cessation topics of relevance for professional development, they reported 11 content areas, respectively: (1) mental illness and substance use disorders; (2) tobacco dependence treatment; (3) tobacco cessation for women across the lifespan; (4) motivational interviewing; (5) implementation; (6) harm reduction; (7) advocacy and system-level change; (8) pharmacological interventions; (9) tobacco interventions for youth and young adults; (10) tobacco interventions for First Nations, Inuit, and Métis populations; and (11) chronic disease prevention. Also, other approaches to cessation related to working with specific populations such as immigrant populations and older adults were reported.

Theme 4: Overall modification—37.8% of comments

The participants’ perceptions regarding overall modifications to the course were categorized into four subthemes: course content, website function, course timing, and course support. A prominent subtheme was course content (17.2% of comments). This subtheme was attributed to less repetitive evaluation surveys following modules, more lengthy and knowledge-based quizzes, more videos, the incorporation of live webinars to connect with participants and faculty, and more case studies and interactive exercises to enhance skill development.

For the website function subtheme (11.6% of comments), participants reported some difficulties with loading the videos and navigating through the course content. Comments regarding course timing and scheduling (6% of comments) described the importance of revising the module timing to make it flexible for HCPs with busy schedules (e.g., having all modules open at the same time). Participants also requested continuous access to the course materials after the completion of the course so that they could refer back to content and use it as a reference and resource in the future. The final subtheme emerging under suggestions for improvement was course support (3% of comments). Some participants also commented that they were dissatisfied with the course registration process.

Theme 5: Overall satisfaction—18.4% of comments

When we asked participants about general comments regarding the training provided, they typically responded positively with 1-word responses, such as “good” or “excellent.” The central subthemes emerging from this section included course usefulness, course support, web usability, and timing.

Many participants described how the design and the delivery of the course content were useful which included both static content (e.g., resources and readings) and interactive content (e.g., videos, webinars, interprofessional collaboration, and quizzes). Course support also garnered positive feedback highlighting participants’ satisfaction with course administrators. Web usability, including course design, ease of navigation, and loading of webpages and videos, accounted for 1.7% of comments of overall satisfaction. For example, a very small number of participants (0.2% of comments) stated that they would have preferred taking this course in-person. Fewer comments for overall satisfaction could be found under the timing subtheme (mentioned in 0.3 % of comments).

TEACH online Core Course content is iteratively updated to incorporate the latest advances and evidence-based approaches in tobacco cessation, in addition to HCPs’ feedback. However, upon reviewing the results across the four cohorts in 2015, there were a number of changes that needed to be made to enhance the learner experience and align the course with best practice approaches in adult learning. The TEACH online Core Course has recently undergone a major revision incorporating knowledge-based and case-based quizzes, clinical video demonstrations, self-reflections, opportunities to practice clinical skills through case studies, and collaborative projects that require participants to work together on activities in the online environment. Course faculty are now required to provide individual and customized feedback on course assessments so that participants can increase their awareness of knowledge and skill areas for further improvement. Website design and functionality has been addressed by providing further instructions regarding course navigation, streamlining course content to reduce complex navigation requirements, and ensuring that the course content is functional across a number of website browsers. This revised course was launched in July 2016, and evaluation results are forthcoming.

Discussion

This study was initiated to demonstrate two goals and provides several important and novel findings. First, this phenomenological study, through thematic analysis of 690 comments and emerging 10 main themes, highlights the importance of employing a hybrid process of inductive and deductive thematic analysis for formative and summative qualitative evaluations, and also highlights the importance of collecting qualitative feedback (in addition to quantitative feedback) for program evaluation. This article provides a comprehensive understanding of the TEACH Core Course through qualitative analysis and evaluation. We found new learning needs emerged during the course implementation phase from the participants’ perspective. Using a strong qualitative evaluation for the TEACH online Core Course allows us to advance learning objectives focused on tobacco dependence treatment and implement research on education outcomes and the features that improve them, which has also been demonstrated in a meta-analysis for the effectiveness of online learning.[1]

Problem identification and needs assessment for the target audience should be administered prior to curriculum development; however, as learners might not fully know their needs until exposed to the subject matter, and their needs may evolve due to learner characteristics (e.g., discipline and years of experience), recurrent needs assessments should be built into course evaluations.[14-17] These important steps have been considered through our formative and summative course evaluations. Skeff et al[18] have also noted that individuals often do not fully appreciate their professional development needs until they have been exposed to the subject area.

The current qualitative study could be considered as a model for determining the success of educational programs and used as a learner needs assessment to guide iterative revisions to curriculum, the most important steps in the curriculum development and evaluation process.[17] Cronbach, who is one of the pioneers of the quantitative method, stated that a well-designed qualitative evaluation is important and emphasized the significance of its application to determine the success of educational programs.[19]

Second, the main themes emerging from this study are based on the feedback provided by HCPs across four course cohorts. Given the large sample size included in this qualitative research coupled with a high response rate (96.5%), a wide range of participant views were captured. This high response rate demonstrates that online training and evaluation can be deployed for HCPs without compromising learner overall satisfaction. However, many of the HCPs who responded to the qualitative evaluations may have had definite views regarding the course they completed (e.g., either a positive or negative learning experience), and it is possible that many of the participants who did not respond with qualitative comments either did not have suggestions for improvement or did not feel comfortable sharing negative feedback. This result also is consistent with past literature regarding evaluating online training for HCPs.[20]

A total of 349 comments came from the formative evaluations, and five themes emerged through an inductive approach: (1) overall quality, (2) content, (3) evaluation, (4) timing, and (5) delivery method. A total of 341 comments came from the summative evaluations and were related to the following five categories derived from a deductive approach: (1) learning objectives, (2) interprofessional collaboration and networking, (3) future topics of relevance, (4) overall modification, and (5) overall satisfaction.

Nearly all of the positive comments elicited from the formative evaluation were focused on the overall quality and content of the modules, and more than 75% of the overall satisfaction in the summative evaluation was related to the course content. Our findings are consistent with earlier research which indicated that learners give more feedback on the overall quality and content of the curriculum than on other course areas.[14]

Those participants who provided comments related to suggestions for improvement in the formative evaluations were looking for more knowledge-based quizzes and fewer repetitive module evaluations at the end of each module. However, completion of the module evaluation was not mandatory. Our results confirm that participants would like to have more videos and live webinars to increase interactivity, a technical solution for loading videos, more case studies and scenario-based quizzes to check knowledge retention and application, more opportunities for collaboration with fellow learners, feedback from facilitators regarding knowledge and skills, and for course designers to focus on troubleshooting issues associated with video loading. Our results are consistent with past literature indicating that a technology-mediated course that matched learners’ needs can facilitate understanding of the subject.[20-22] Several participants reported the usefulness of course administrative support, believing that this was helpful in addressing any technical or logistical issues in a respectful and timely manner. In addition, many participants felt that course discussion boards should be actively monitored by facilitators to encourage proactive conversation and provide further insight on the topics discussed by learners. This supports Boettcher’s best practices[23] for teaching online that emphasize the importance of creating a supportive online course community. In most online courses, the dialogue between faculty and participants is supported through weekly coaching and reminders. To enhance dialogue between participants in our online course, we have since added weekly live webinar sessions for enhanced interaction and problem solving and have added in additional discussion boards for reflective practice and coaching.

Participants also suggested that the course administrators should open all the modules at once, remove time constraints across the course (i.e., assignment deadlines), and continue to keep the course accessible after it has ended so that they can refer back to content and resources at a later date. Providing training to HCPs that will enhance practice skills can often be difficult given their competing clinical demands and priorities.[24] E-Learning can help address some of these challenges by offering more flexible timelines, eliminating travel, and offering a diverse learning environment without compromising leaner satisfaction.[20] A very small number of HCPs stated that they would have preferred taking this course in a classroom-based setting.

There are several limitations in this study. Participants were not followed up with to elicit more information about their comments. The analysis of free-text comments and open-ended responses in this context did not allow for the opportunity to probe respondents for further detail or explanation.[25] Our study did not collect demographic characteristics such as gender and ethnicity; therefore, we cannot be certain that our sample is reflective of the general Ontarian HCP population. Despite its limitations, there are some valuable conclusions to be drawn from this study. HCPs find TEACH to be a valuable program for tobacco dependence treatment training. This article adds a new perspective to the current literature regarding online courses in tobacco dependence treatment for HCPs, presents a model to evaluate effectiveness of online educational interventions, and provides tailored suggestions to improve online courses which can be applied in other contexts.[26] To complete the cycle of quality improvement, these findings will inform future course iterations’ development and revisions. Tracking evaluation outcomes and learner feedback by developing a strong evaluation component to online curricula allows educators to advance the science of education and increase evaluative research on education outcomes and the features that improve them.[1]

Conclusion

Using a combined technique of inductive and deductive thematic analysis, this formative and summative qualitative evaluation has highlighted an approach that demonstrates rigor within a quality improvement research study. This process made it possible to describe participants’ satisfaction with their learning experience. Areas of strength and needed improvements of the online course have been examined from the perspective of HCPs through qualitative feedback collected across four course cohorts. Curriculum designers should incorporate opportunities for monitoring and feedback by expert facilitators to foster a supportive online learning environment. CME providers should consider the importance of collecting qualitative feedback from their participants to elicit concrete and relevant suggestions to improve the experience for future learners. This qualitative evaluation approach can be replicated by other evaluation scholars and can assist them in evaluating the effectiveness of online educational interventions, in the context of CME and beyond.