EBM E-learning: Feasible and Effective for Occupational Physicians in Different Countries.

OBJECTIVES: Although evidence-based medicine (EBM) is a useful method for integrating evidence into the decision-making process of occupational physicians, occupational physicians lack EBM knowledge and skills, and do not have the time to learn the EBM method. In order to enable them to educate themselves at the time and place they prefer, we designed an electronic EBM course. We studied the feasibility and utility of the course as well as its effectiveness in increasing EBM knowledge, skills, and behaviour.
METHODS: Occupational physicians from various countries were included in a within-subjects study. Measurements were conducted on participants' EBM knowledge, skills, behaviour, and determinants of behaviour at baseline, directly after finishing the course and 2 months later (n = 36). The feasibility and utility of the course were evaluated directly after the course (n = 42).
RESULTS: The course is applicable as an introductory course on EBM for occupational physicians in various countries. The course is effective in enhancing EBM knowledge and self-efficacy in practising EBM. No significant effect was found on EBM skills, behaviour, and determinants of behaviour. After the course, more occupational physicians use the international journals to solve a case.
CONCLUSION: An electronic introductory EBM course is suitable for occupational physicians. Although it is an effective method for increasing EBM knowledge, it does not seem effective in improving skills and behaviour. We recommend integrating e-learning courses with blended learning, where it can be used side by side with other educational methods that are effective in changing behaviour.

Introduction

Decisions in occupational health care by occupational physicians (OPs) should be based on professional competences of the OP, workers' (patients'), and employers' preferences, and evidence from research information [1]. In order to accomplish this, OPs can benefit from integrating the evidence-based medicine (EBM) method into their daily practice. When applying EBM as described by Sackett et al. (1999) [2], OPs should start with formulating an answerable question, then searching and acquiring relevant literature, appraising it for quality, applying the findings, and evaluating the performance of the decision. In doing so, OPs can use evidence from occupational cohort and randomized controlled studies, for example, but they can also use evidence stemming from studies performed by a wide variety of medical and other disciplines such as occupational hygiene and psychology [3]. In contrast with the occupational setting, EBM is commonly used in the clinical setting and has been proven to enhance the quality of health care [4-6].

Over the last few years, there have been first attempts to promote EBM in the occupational and public health care setting. Associated evaluation studies showed that there are various opportunities to enhance the use of EBM [1,7-10]. As a result, we already know that OPs are often unaware of the high volume of relevant questions that can be derived from their own daily practice and of the opportunities to answer most questions when following EBM strategies [11]. Moreover, when answering their information demands, OPs rely rather on their own experience or on advice from colleagues and 'experts' instead of searching for evidence in the scientific literature themselves [12]. Interestingly, it has been shown that the advice OPs received from 'experts', including colleagues, do not reflect the best evidence from the literature [9].

An evaluation of an EBM course combined with recurrent peer group sessions showed that EBM knowledge, skills, and attitude of OPs improved significantly, as well as the quality of therapy advice of OPs in the short term [13,14]. The participating OPs of that study valued EBM as a useful method for enhancing professional performance, although they thought the combination of the course and the peer group sessions were intensive and time-consuming. In many other studies evaluating EBM strategies in settings other than the occupational health field, time constraints are also reported as a main barrier for the uptake of EBM [15-18].

In particular, to overcome this barrier of limited time and to promote EBM among OPs internationally, we designed an EBM e-learning introductory course in the English language. The aim was to enable OPs to train themselves in the basics of EBM at the time and place of their choice, determining their own pace and repeating the course whenever they choose [19]. However, although e-learning or web-based education is being used increasingly, little is known about the feasibility and the effectiveness of it when used by OPs in a wide variety of countries. One study on the preferences of OPs on distance learning reveals that they prefer online practice in addition to printed education material [20]. Another recent study shows that e-learning is just as effective as lecture-based learning in enhancing OPs' knowledge on mental health care [21]. One interesting question deals with the suitability of e-learning developed in one country or region, for the daily practice and learning demands of occupational professionals across various countries. We expect these professionals to have a large variety of professional activities and backgrounds in vocational training. Barriers to effective implementation might be expected in technical problems, such as access to the internet and availability of scientific literature, and in the wide variety of legislation, cultures, and languages [22,23]. All of these questions and uncertainties demonstrate a need for evaluation in order to learn for the future.

Therefore, in this study, we first decided to evaluate whether the EBM e-learning course is applicable for OPs in different countries, in terms of feasibility and utility. Next, we wanted to know whether the course enhances the EBM knowledge and skills of OPs. Finally, in order to determine whether the EBM e-learning course is capable of changing the EBM behaviour of OPs, we used the attitude, social influence, and self-efficacy (ASE) model [24]. This model is often used in the field of health education. Taken together, attitude, social influence, and self-efficacy determine the motivation for certain human behaviour, the intention. The ASE model implies that intention predicts behaviour [25]. In accordance with the ASE model, social influence, perceived self-efficacy, and the intention to practice EBM as determinants of actual EBM behaviour were included as variables in the study.

Materials and Methods

Study population and recruitment procedure

Study participants were OPs from various countries participating in a local postgraduate course in the field of occupational health care organized by experienced (postgraduate) education and training centres. To approach these centres, national and international contacts of our institute (Coronel Institute of Occupational Health) as well as a list of the members of the European Association of Schools of Occupational Medicine were used.

The education centres received information about our study via email in June and were invited to assist in our study in a four-month period from August until November 2007. If the centre agreed to assist, they were asked to provide the email addresses of OPs who were potential course participants of the centre. These potential participants received information on the EBM e-learning evaluation study as well as an informed consent form via email. Only those course participants who sent back their signed informed consent form were included in our study. Participants were requested to complete the EBM e-learning course in their leisure time. If they were by chance planning to attend an EBM course or a similar course offered by their educational centre, they were asked to follow our electronic EBM course first. Some educational centres offered our electronic EBM course as part of their own postgraduate programme. When this was the case, the electronic EBM course was provided at the start.

EBM e-learning course

The format of the EBM e-learning course (http://www.nspoh-on-line.nl./ebm-who/) was based on an effective e-learning course on mental health care for OPs, and its content was based on the recently published first international practical guide on EBM for occupational health [3]. The course is in English, lasts about one hour and a half, and comprises seven topics. The seven topics are: Basic principles of EBM (23 screens); types of questions (26 screens); the patient, intervention, comparison, outcome (PICO) method (10 screens); information sources and search strategies (20 screens); searching in MEDLINE with PubMed (29 screens); evaluating results (41 screens); answering questions (15 screens); in a total of 164 screens. Every topic starts with a screen on the issues that will be covered in that section (Appendix). Then, screens with information are alternated with screens with assignments. Feedback on the assignments is provided directly. Figs 1, 2, and 3 are examples of the screens. Furthermore, the course has an online library available where participants can find: direct access to PubMed; an overview of relevant keywords (search terms) for occupational health professionals; an overview of relevant databases; relevant articles and books; an overview of help files; examples of critically appraised topics; and links to websites that support professionals in the field of EBM.

Fig. 1

'Start' screen on the topic "Evaluating results" presents the issues covered.

Fig. 2

Information screen on 'statistical significance' in the section "Evaluating results."

Fig. 3

Example of an assignment in the section "the patient, intervention, comparison, outcome (PICO) method".

Study design and type and timing of measurements

A within-subjects study design with pre- and post-test evaluation was used. Pre- and post-test measurements were conducted by means of an electronic questionnaire on the participants' knowledge on, skills in, and attitude towards EBM, as well as EBM behaviour and determinants of that behaviour. Participants received the questionnaire at 3 time points. After providing their informed consent, they received a baseline questionnaire (T0) by email. As soon as the participants completed this questionnaire, the URL of the electronic EBM course and instructions on how to complete the course were provided. Upon completion of the course, the second questionnaire (T1) was automatically presented in a pop-up screen and the participants had to immediately fill in this questionnaire. Approximately 2 months after completion of the electronic EBM course and the second questionnaire, the last questionnaire (T2) was sent to the participants by email.

Outcome

In order to describe the study population, demographics of participants (country of residence, age, gender), experience as an OP, work setting, and previous training in research methods, epidemiology, statistics, or EBM were assessed at T0. The utility and feasibility of the electronic EBM course were assessed at T1, directly after completing the course. Knowledge, skills, attitude, behaviour, and determinants of behaviour related to EBM were assessed at all moments T0, T1, and T2. Evidence use was only assessed at T0 and T2, since T1 was too soon after T0 to expect any change on evidence use.

Feasibility and utility of the electronic EBM course

Feasibility was defined as the extent to which the EBM electronic course could be completed by the participants. Utility was defined as the extent to which the participants perceived the content of the EBM electronic course as useful for the improvement of their performance. This means that we evaluated to what extent the course was clear in both form and content, and thereby to what extent it was attuned to the prior knowledge and to the practice setting of the participants. The researchers formulated 14 statements to which the participants could agree on a 4-point Likert scale; scores ranged from 1 (strongly disagree) to 4 (strongly agree).

EBM knowledge

EBM knowledge was defined as the knowledge participants had on the basic principles of EBM, how to search in PubMed/MEDLINE, and how to appraise the literature critically. To measure the participants' knowledge, we used the critical appraisal part of the questionnaire of Taylor et al. [26,27], combined with questions designed by one co-author (F. V.) on the basic principles of knowledge on EBM and searching abilities in Pubmed. Our combined list consisted of 31 true/false/don't know questions. A correct, incorrect, and 'don't know' response yielded a score of +1, -1, and 0 points, respectively, and were added up to form an overall sum score. The sum scores were subsequently converted into scores between 0 and 100 in order to facilitate standardisation on all outcome measurements.

EBM skills

EBM skills were defined as the ability to perform the steps of EBM described by Sackett et al. [28] (1996): formulate a clear clinical question from a patient's problem; search the literature for relevant clinical articles; evaluate the evidence for validity and usefulness; and implement the useful findings in practice. For the measurement of EBM skills, we used the skills part of the Fresno test used by Schaafsma [14] and adapted the scenarios to occupational health-related situations familiar for professionals in many countries. The test consisted of eight open questions; we used the standardised grading system of the Fresno Test [6,29]. Again, the sum scores were converted into scores between 0 and 100.

EBM attitude, behaviour, and determinants of behaviour

EBM attitude was understood to represent the beliefs, feelings, and values of the participants towards EBM. In accordance with the ASE-model, we included the social influence related to EBM, the perceived self-efficacy to practice EBM, and the intention to practice EBM in the study [23]. The social context can be described as the processes whereby OPs' thoughts, feelings, and actions on EBM are directly or indirectly being influenced by others. Self-efficacy can be seen as the OPs' belief in his or her ability to perform EBM practice. The five constructs of the ASE model were measured by means of the questionnaire from Schaafsma et al. [12], including 22 statements. In the original Dutch questionnaire, there were five statements on the use of evidence-based national guidelines for OPs. Since guidelines are non-existent in some countries, three statements were removed from the original questionnaire and two statements were adapted into statements on the use of EBM in general. This resulted in: 1) 4 items measuring attitude towards EBM (e.g., "using EBM in daily practice improves the quality of the physician"); 2) 5 items measuring the social context of the OP, reflecting the support received from the social work environment such as from supervisors or colleagues; 3) 5 items measuring self-efficacy, including the feeling of self-confidence towards practising EBM; 4) 3 items measuring the intention to use EBM in the near future; and 5) 2 items measuring actual EBM practice. Our instrument used a 5-point Likert scale: strongly agreeing with a 'positive' statement and strongly disagreeing with a 'negative' statement scored '5' points, strongly disagreeing with a positive statement or strongly agreeing with a negative statement scored '1' point. For each of the 5 variables, a sum score was calculated to obtain an overall score. Again, the sum scores were converted into scores between 0 and 100 in order to facilitate standardisation on all outcome measurements.

Evidence use

We used a broad concept of evidence use, defined as the self-reported time spent on keeping up-to-date and solving a specific case and, in addition, the use of a number of sources by participants when doing so. To assess this, we used the 'Reading and evidence seeking behaviour' part of the questionnaire of Taylor et al. [27]. First, the OPs reported the number of hours spent on solving a specific case and on keeping up-to-date over the previous month. In addition, they reported the number of articles read to solve a specific case and the number of articles read to keep up-to-date during the previous month. Second, they were asked to report the proportion (%) of these articles that they read thoroughly, skimmed, or read only the abstract of. Finally, the OPs reported how often they used several kinds of sources to keep up-to-date and to solve their cases. The sources asked for were: review articles in international journals, original research reports in international journals, national journals, textbooks, Internet resources/computer databases or similar, guidelines, the Cochrane Library, and colleagues.

Statistical analysis

Proportions (means and standard deviation [SD]) of our measures of the utility and feasibility of the electronic EBM course were calculated. To determine if the electronic EBM course had an effect on EBM knowledge, skills, attitude, and determinants of EBM behaviour over time, the total scores on T0, T1, and T2 were calculated and analysed by means of general linear model for repeated measures. When trends were found, a paired t-test was used post-hoc to analyse differences in scores between T0 and T1. Mean scores (SD) and proportions of the items on evidence use were calculated. Statistical analyses were carried out using SPSS version 13.0 (SPSS Inc., Chicago, IL, USA).

Results

Baseline characteristics

Overall, 102 OPs from 16 countries returned the informed consent form and 84 OPs filled in a baseline questionnaire (T0). Directly after conducting the electronic EBM course (T1), 42 OPs returned a completed questionnaire. Approximately two months after completing the electronic EBM course, 36 OPs returned the last questionnaire (T2). Table 1 shows the personal characteristics of the participants at baseline and T2.

Table 1

Baseline characteristics of the participating OPs

Values are presented as number (%) or number only.

OP: occupational physician, T2: the last questionnaire returned approximately two months after completing the electronic EBM course, WHO: World Health Organization, OHS: occupational health service, GP: general physician, EBM: evidence-based medicine.

*Occupational and public health expert, occupational safety and health expert, studying occupational medicine; †more than one answer is possible.

The majority of participating OPs are from Europe, mainly Belgium (n = 13 at T0; n = 0 at T2), the Czech Republic (n = 9 at T0; n = 6 at T2), and Italy (n = 8 at T0; n = 8 at T2); Africa, mainly South Africa (n = 11 at T0; n = 5 at T2); and the West Pacific, mainly Japan (n = 16 at T0; n = 4 at T2). Almost 60% of the OPs are less than 40 years of age and about half of the OPs are still in training. The OPs mostly work at an occupational health service provider or work in a company, and almost 80% have had training in methodology, statistics, or EBM. No major differences of characteristics can be distinguished between the participants at baseline and the remaining group at T2.

Utility and feasibility of the electronic EBM course

Table 2 shows that the vast majority of the OPs agree with the positive statements on the utility and feasibility of the course. The highest agreement is on the statements that the course offered sufficient information for an introductory course (100%), that it helped them to better practice EBM (97.6%), and that it improved their quality of work (97.6%). The relatively smallest agreement is on the statements that the course matched their educational level and that they enjoyed taking the online course (both 79%). Ninety three percent of the participants would like to take more of this kind of online course. Overall, they appreciate the electronic EBM course with the mean figure of 7.6 on a 10-point scale.

Table 2

Utility and feasibility of the course: percentage of OPs that (strongly) agree with the statements (n = 42)

Values are presented as number (%).

OP: occupational physician, EBM: evidence-based medicine.

EBM knowledge, skills, attitude, behaviour, and determinants of behaviour

Of the participants who returned their questionnaire at T2 (n = 36), only 25 OPs filled in the skill part, 33 OPs the knowledge part, and 32 OPs the attitude, behaviour, and determinants of behaviour part of the questionnaire at all three time points. The electronic EBM course has an overall significant effect on the OPs' knowledge of EBM and their self-efficacy on practising it (p < 0.01 and p = 0.02, respectively). The overall effect of the intervention on the enhancement of EBM skills is not significant, but the skills enhance substantially at T1 and, although they decline, are still higher at T2 compared to T0. The initial high scores on attitude remain stable over time. Except for social context, all scores are higher after the course compared with before, and the increases sustain at T2, but are not significant (T0 versus T1 and T2, Table 3).

Table 3

OPs' EBM knowledge, skills, attitude, behaviour, and determinants of behaviour scores at T0, T1, and T2

Values are presented as mean (standard deviation).

*Overall tests on trends at time of the intervention.

†p < 0.05.

OP: occupational physician, EBM: evidence-based medicine, T0: baseline questionnaire, T1: the second questionnaire returned directly after conducting the electronic EBM course, T2: the last questionnaire returned approximately two months after completing the electronic EBM course.

Evidence use

Table 4 demonstrates that the number of articles read and hours spent reading decreases after conducting the course, particularly on keeping up-to-date. The number of journal articles that OPs looked at or read thoroughly is, both before and after conducting the course, higher for keeping up-to-date than for solving a case. In addition, the proportion of articles from which the OPs read the abstract only, skimmed, or thoroughly read stays the same.

Table 4

Evidence use of OPs (n = 36) before, and two months after conducting the electronic EBM course

Values are presented as mean ± standard deviation, number (%), or percent only.

OP: occupational physician, EBM: evidence-based medicine, T0: baseline questionnaire, T2: the last questionnaire returned approximately two months after completing the electronic EBM course.

At baseline, OPs particularly use the national and international journals to keep up-to-date; they use textbooks and guidelines to solve a case. After conducting the electronic EBM course, particularly the proportion of OPs using Internet resources, colleagues, review articles, and guidelines for keeping up to date, increases. For solving a case, particularly the proportion of OPs who use review articles, original research reports, national journals, and internet resources, increases.

Discussion

The electronic EBM course is applicable as an introductory course on EBM for OPs in a wide variety of countries. Ninety percent of the participants agreed that the online format was a good way to learn EBM, and almost all participants agreed that the course helps to improve the quality of their work. The course is effective in enhancing the OPs' knowledge on EBM and the self-efficacy of OPs in practising EBM. No overall significant effect was found on EBM skills, behaviour, and determinants of behaviour, although a substantial enhancement of EBM skills was noticeable directly after the course. OPs tend to read fewer articles and spend fewer hours on reading to keep up-to-date after the course, compared to before. After the course, international journals are particularly used more often by a larger proportion of OPs to solve a case.

To our knowledge, this study is the first research to test the applicability and effectiveness of an electronic EBM course on EBM knowledge, skills, and behaviour of OPs in a variety of countries. By means of the World Wide Web (WWW), we were able to include many OPs from all over the world in our study and test the electronic EBM course. However, we lost 57% of our participants after they completed the baseline questionnaire, which might have been the cause of the lack of change over time in parameters excluding EBM knowledge and self-efficacy. We also cannot exclude the fact that participants who were discontent with the utility and feasibility of the course were more likely to drop out.

Unfortunately, using WWW also raised some technical problems which presumably contributed to the high drop-out rate. First of all, the participants' computers had to be deactivated for pop-up blockers and spam filters in order to open the webpage of the course. Some participants informed us that they were not able to accomplish this, and we could not provide technical support because of the distance barrier. Secondly, an attempt was made to hack our electronic EBM course, and for safety reasons as well as to restore all conditions and materials, the course was unavailable for several days.

Another limitation of our study was the lack of a control group. The electronic EBM course has been developed as an activity embedded in the Global Network of World Health Organization (WHO) Collaborating Centers in Occupational Health and was freely accessible through the internet during the period of our study. This made it difficult to include a control group. Compared to reference groups from others studies, the scores on skills in our study at baseline were relatively low [29]. The OPs' mean score on knowledge at baseline was somewhat lower than the mean score of the control group of Taylor et al. [27]. This suggests that OPs in our study started with a relative lack of skills and knowledge in EBM. Attitude and determinants of behaviour at baseline were higher in this study compared to the randomized controlled trial (RCT) on the effectiveness of a course organised in the Netherlands [13,14]. At follow-up, the scores were still higher in our group compared to the scores of the Dutch intervention group. Presumably, only OPs who had a positive attitude towards EBM, participated in our study. The OPs in our study may be considered as 'innovators' in the adaption of EBM, which is still a relatively new phenomenon in the occupational health field. Considering these facts, together with the small sample size, the results of this study lack generalisability to all OPs in the countries involved.

Following other studies on this topic [27,30], evidence use was measured by self-assessment. As this method is susceptible to bias in terms of giving desirable answers by the participants, the increase of participants using international journals in solving cases might be an overestimation. Measurement of changes in evidence use and EBM behaviour in fact require assessment in the practice setting [31].

Agius and Bagnall [32] evaluated the demands and merits of the use of WWW in (postgraduate) learning occupational health care. The study indicated that it can be a useful learning resource, since students valued the flexibility, suitability, efficiency, and breadth of access to relevant information offered by WWW [32]. A recent RCT among Dutch OPs showed that e-learning was just as effective as didactic learning in enhancing knowledge [21]. There are no other studies on e-learning among postgraduate OPs, but we can learn from other clinical medical settings where e-learning appears to be at least as effective as traditional instructor-led methods such as lectures. An e-learning course can also be integrated in 'blended learning' - a combination of technology-based materials and face-to-face sessions used together. In addition, by offering blended learning to OPs, a maximum of learning approach preferences can be met in this way, resulting in better learning outcomes [32,33].

Since WWW is becoming more and more accessible for OPs all over the world, it is feasible to promote e-learning as a learning tool in occupational health care. E-learning technologies offer learners the control over learning sequence, pace of learning, time, and often media. Furthermore, e-learning offers educational centres the opportunity to reach OPs who are spread over a wide geographical area. On a larger scale, international collaboration can be achieved to develop more e-learning courses for OPs and other occupational health professionals, and evaluate the effectiveness of these courses on a larger scale.

In particular, since EBM is relatively new in the occupational health care field, an online introductory course can be useful for enhancing at least the EBM knowledge of OPs. For improving the EBM skills and EBM behaviour of OPs, more intensive education and training will be needed.

In conclusion, our electronic introductory course on EBM was applicable and successful in increasing OPs' EBM knowledge and self-efficacy in practising EBM. An improvement in EBM skills was found directly after the course, but did not sustain over time. The course was not effective in changing the already high attitude of OPs, the social influence, and EBM behaviour. It seems that e-learning courses can be integrated with blended learning, where it can be used next to other educational methods that are effective in changing behaviour.