Evaluation of a Statewide HIV-HCV-STD Online Clinical Education Program by Healthcare Providers - A Comparison of Nursing and Other Disciplines.

The New York State HIV-HCV-STD Clinical Education Initiative (CEI) has developed a large repository of online resources and disseminated them to a wide range of healthcare providers. To evaluate the CEI online education program and in particular to compare the self-reported measures by clinicians from different disciplines, we analyzed the data from 1,558 course completions in a study period of three months. The results have shown that the overall evaluations by the clinicians were very positive. Meanwhile, there were significant differences across the clinical disciplines. In particular, physicians and nurse practitioners were the most satisfied. In contrast, pharmacists and case/care managers recorded lower than average responses. Nurses and counselors had mixed results. Nurse practitioners' responses were very similar to physicians on most measures, but significantly different from nurses in many aspects. For more effective knowledge dissemination, online education programs should consider the unique needs by clinicians from specific disciplines.

Introduction

In the United States, the annual numbers of newly infected with HIV, HCV, and other STDs are estimated at 47,500 [1], 29,700 [2], and 20 million [3] respectively. The total infections are estimated at 1.2 million [1], 2.7 million [2], and 110 million [3]. Medical costs associated with diagnosis, treatment, and prevention could be as high as $16 billion a year [3]. With the many ongoing clinical trials on treatments, vaccines, and behavioral interventions and the frequent updates on practice guidelines based on the findings from research, effective dissemination of the latest clinical evidence to the community healthcare providers, who are working on the frontline to fight HIV, HCV, and other STDs, has become an essential requirement.

Comparing to the traditional classroom or clinic-based approaches to providing continuing professional education, online training is advocated as an efficient platform for rapid dissemination of knowledge to healthcare providers [4-5]. Built on a history of success for two decades to provide in-person training, the New York State (NYS) HIV-HCV-STD Clinical Education Initiative (CEI) [6] started its online education program in 2008. Over a period of seven years, CEI has developed 290 multimedia learning modules, 111 online CME/CNE courses, 14 interactive case simulation tools, and various other online resources [7]. These resources have been disseminated to tens of thousands healthcare providers from 170+ countries through web, mobile apps, email newsletters, and online social networks [8].

In previous publications, we reported the development of CEI online resources [7], effective dissemination of these resources [8], their actual usage by clinicians [9], and initial assessment on effectiveness and impact of the CEI online education program [10]. Here we report an evaluation study to further analyze the feedback from healthcare providers who have completed CEI online courses, focusing on a comparison of nursing and other clinical disciplines. The results from this analysis will provide important information to guide the future development of online education programs that can be custom-tailored to specific clinical disciplines to better serve their information needs.

Methods

We included the clinicians who successfully completed a CEI online course between April 1, 2015 and June 30, 2015 in this study. As a part of the process for course completion, each clinician was required to provide evaluations on the training. The evaluation measures included usefulness/relevance of information, easy comprehension, trainer’s knowledge, appropriateness of format, knowledge increase, intention to use the learned knowledge, and intention to change practice [10]. The entire process of course enrollment, completion, and evaluation by a clinician was through the CEI’s student portal. Within the student portal, a clinician’s personal information (for example, contact and demographics) and professional background (for example, discipline, employment setting, practice years, and patient case load) were collected and stored in the student profile [10]. Partial screenshots of the evaluation questionnaire and clinicians’ background from the CEI student portal are shown in Figure 1.

Figure 1

Partial screenshots of evaluation questionnaire and clinicians’ background from the student portal.

For data collection, we queried the CEI database to obtain the student background, course completions, and evaluations. For the Likert-scale measures, we reformulated the evaluation responses as binary variables (positive vs. non-positive). For the measure on knowledge increase, we first calculated the difference of a clinician’s self-reported knowledge levels (novice, not very knowledgeable, knowledgeable, very knowledgeable, or expert) before and after the training, and then formulated those with ≥1 level of increase as positive responses. For data analyses, we compared the proportions of the positive evaluation responses across the disciplines. We used the chi-squire test to examine the statistical significance of the differences.

Results

We recorded a total of 1,558 completions of 76 online courses during the study period. The clinical disciplines that logged the most course completions were: physician (301, 19.32%), nurse practitioner (292, 18.74%), nurse (182, 11.68%), pharmacist (117, 7.51%), case/care manager (101, 6.48%), and counselor (84, 5.39%). For the remaining fifteen disciplines, each with course completions less than 5% of the total number, we grouped them into a single category (481, 30.87%) for analyses.

Similar to our previous findings [10], the overall evaluations by the clinicians were very positive (usefulness and relevance 92.17%, easy comprehension 91.21%, knowledgeable trainer 92.49%, appropriate format 86.07%), and the clinicians’ self-reported impacts of training were significant (knowledge increase 41.21%, intention to use the learned knowledge 86.78%, intention to change practice 37.67%).

When analyzing the responses by disciplines, we found statistically significant differences (p<0.001) for all measures. In particular, physicians had the most positive evaluations in all but one (appropriate format) aspects when compared with the other disciplines. Similar to physicians, nurse practitioners had more positive feedback on all but one (knowledge increase) measures when compared with the average responses from all disciplines. In contrast, pharmacists and case/care managers had lower than average responses, while nurses and counselors had mixed results.

Comparing nurses with nurse practitioners, significant differences were found on the measures of easy comprehension (86.81% vs. 93.49%, p=0.014), knowledgeable trainer (90.11% vs. 94.86%, p=0.048), intention to use knowledge (77.47% vs. 94.52%, p<0.001), and intention to change practice (28.57% vs. 43.56%, p=0.030). No obvious differences were found between these two groups on the measures of usefulness and relevance (90.66% vs. 94.18%, p=0.149) and appropriate format, (92.86% vs. 88.10% p=0.089). Nurses had more positive responses than nurse practitioners on the measure of knowledge increase (45.60% vs. 33.90%, p=0.011).

The detailed evaluation data by measures and disciplines are shown in Table 1.

Table 1.

Number and percentage of positive evaluations by measures and disciplines.

Measures	Nurse	NP*	Physician	Pharm.*	CM*	Counselor	Other	All
usefulness andrelevance	165(90.66%)	275(94.18%)	293(97.34%)	102(87.18%)	87(86.14%)	66(78.57%)	448(93.14%)	1436(92.17%)
easycomprehension	158(86.81%)	273(93.49%)	288(95.68%)	104(88.89%)	87(86.14%)	67(79.76%)	444(92.31)	1421(91.21%)
knowledgeabletrainer	164(90.11%)	277(94.86%)	296(98.34%)	103(88.03%)	90(89.11%)	65(77.38%)	446(92.72%)	1441(92.49%)
appropriateformat	169(92.86%)	257(88.01%)	270(89.70%)	96(82.05%)	80(79.21%)	74(88.10%)	395(82.12%)	1341(86.07%)
knowledgeincrease	83(45.60%)	99(33.90%)	151(50.17%)	22(18.80%)	39(38.61%)	40(47.62%)	208(43.24%)	642(41.21%)
intention to useknowledge	141(77.47%)	276(94.52%)	285(94.68%)	99(84.62%)	86(85.15%)	64(76.19%)	401(83.37%)	1352(86.78%)
intention tochange practice	18(28.57%)	115(43.56%)	122(46.56%)	14(15.38%)	15(28.85%)	7(29.17%)	68(34.52%)	359(37.67%)
Total	182(100.00%)	292(100.00%)	301(100.00%)	117(100.00%)	101(100.00%)	84(100.00%)	481(100.00%)	1558(100.00%)

NP: nurse practitioner; Pharm.: pharmacist; CM: case/care manager

Discussion

Comparing to our previously reported preliminary results [10], the overall course evaluations by clinicians remained to be very positive (in fact, slightly better than the preliminary results on most measures). Nevertheless, the responses from the different clinical disciplines were not uniform. The analyses clearly indicated that physicians and nurse practitioners were the two groups most satisfied with the online program, followed by nurses. Meanwhile, pharmacists and case/care managers recorded lower than average responses (though need to note that they still had >80% positive feedback on most measures). One potential explanation is that the current curriculum development of the CEI online program focused more on physicians and nurses (the two largest groups of our audience). Our future program development, thus, should be strengthened to address the needs of pharmacists, case/care managers, and other members of HIV-HCV-STD care team.

Interestingly, we noted that nurse practitioners’ responses were very similar to physicians on most measures, but significantly different from nurses in many aspects. A potential explanation is that physicians and nurse practitioners had similar information needs, which were well satisfied by the significant number of our online courses addressing complex issues in patient management. Others had similar findings or assumptions in their evaluations of clinical education programs [11-12]. To our knowledge, this study is the first to report such findings on an online education program and with a focus in the clinical domains of HIV, HCV, and other STDs.

There are a few limitations in this study. First, we didn’t include the training courses as a variable in the analyses. The specific courses or topics are likely an important factor to influence clinicians’ evaluations. Given the significant number of the available CEI online courses, conducting such analyses will require a large sample of clinicians and course completions. This is a direction we would like to explore in the future. In addition to clinical discipline, other factors, such as employment setting, practice years, and patient case load, are also likely to influence a clinician’s evaluation. We have already collected clinicians’ professional background in the CEI student portal. We plan to include these variables in analyses for the next steps.

In conclusion, our evaluation of the CEI online education program has shown very positive overall feedback from the clinicians. Meanwhile, we have found significant differences across the clinical disciplines. For more effective knowledge dissemination, online education programs should consider the unique needs by healthcare providers from specific disciplines for more effectively learning.