Significant increase in factual knowledge with web-assisted problem-based learning as part of an undergraduate cardio-respiratory curriculum.

In recent years, increasing attention has been paid to web-based learning although the advantages of computer-aided instruction over traditional teaching formats still need to be confirmed. This study examined whether participation in an online module on the differential diagnosis of dyspnoea impacts on student performance in a multiple choice examination of factual knowledge in cardiology and pneumology. A virtual problem-based learning environment for medical students supervised by postgraduate teachers was created. Seventy-four out of 183 fourth-year medical students volunteered to use the online module while attending a 6-week cardio-respiratory curriculum in summer 2007. Of these, 40 were randomly selected to be included (intervention group); the remaining 34 served as an internal control group. Analysis of all written exams taken during the preceding term showed that both groups were comparable (86.4 ± 1.1 vs. 85.9 ± 1.1%; p = 0.751). Students in the intervention group scored significantly higher in the final course assessment than students allocated to the control group (84.8 ± 1.3 vs. 79.5 ± 1.4%; p = 0.006; effect size 0.67). Thus, additional problem-based learning with an online module as part of an undergraduate cardio-respiratory curriculum lead to higher students' scores in an exam testing factual knowledge. Whether using this teaching format increases overall student motivation to engage in the learning process needs to be further investigated.

RCT Entities:   Population Interventions Outcomes

Introduction

The use of computers was introduced in medical education more than 40 years ago (Lipkin et al. 1961). Although computer-assisted instruction (CAI) would naturally lend itself to a more complex presentation of content matter (Tegtmeyer et al. 2001), early attempts mainly aimed at a linear transmission of knowledge. Since then, the developmental focus has moved from mere technical aspects to the design of software solutions suited for specific teaching objectives (Valcke and De Wever 2006). In recent years, increasing attention has been paid to web-based learning (WBL) due to its flexible scheduling, economies of scale, ease in updating, and the potential for the delivery of individualized teaching using novel instructional methods (Cook 2006, 2007).

The combination of WBL with a problem-based teaching format results in a virtual learning environment which can be expected to foster the structured acquisition of both knowledge and skills related to clinical reasoning (Hammoud et al. 2006; Kamin et al. 2002). Although WBL cannot replace bed-side teaching, the case studies presented in a WBL module should resemble real clinical settings as far as possible (Glick and Moore 2001). A small number of studies suggest that collaborative learning via the internet is feasible and might yield advantages depending on the circumstances under which it is implemented (De Wever et al. 2008; Kamin et al. 2002; Nathoo et al. 2005; Stromso et al. 2004; Taradi et al. 2005; Wehrs et al. 2007). Regarding the acquisition of factual knowledge, previous studies did not detect an advantage of CAI over traditional teaching formats (Schwartz and Griffin 1993). However, the impact of participating in a problem-based WBL module on students’ gain of factual knowledge has not been studied in great detail.

This article describes students’ enrolment in and satisfaction with a newly set-up online module on the differential diagnosis of chronic shortness of breath. In addition, the students’ achievements in a multiple choice examination of factual knowledge were assessed with regard to their interest and participation in this online module.

Methods

Description of the online module

In summer 2007, 183 fourth-year medical students were given the opportunity to sign up for using and evaluating a collaborative online module while attending the six-week cardio-respiratory curriculum (blended learning) at Göttingen University Hospital (Germany). A web-based learning management system (Clix®, Information Multimedia Communication AG, Saarbrücken, Germany) was used in order to facilitate asynchronous group discussions and the exchange of documents as suggested by previous studies (Spinello and Fischbach 2004). The module was created according to criteria outlined by Hilty et al. (2006) and Greenhalgh (2001). Postgraduate teachers who were specifically trained to assist the students using the module served as online tutors. Course content and the didactic concept underlying the online module were developed by the first author of this manuscript and subsequently discussed with respiratory specialists and faculty staff as recommended (Hudson 2004); During the online module, students were asked to diagnose a 64-year old man who presented to his general practitioner with chronic shortness of breath. The differential diagnosis of shortness of breath is one of the learning objectives to be covered in the six-week cardio-respiratory curriculum. In order to find the correct diagnosis, students needed to have diagnostic procedures carried out on their virtual patient, the results of which were provided by their online tutors. In this regard, the online module did not represent true “problem-based learning” as defined by Barrows and Mitchell (1975). However, the module did comprise distinct steps towards the diagnosis: We expected students to spend on average two hours per week on the online module.

Information on the presenting complaint and past medical history of the patient was provided along with a table indicating costs for 46 standard diagnostic procedures in internal medicine. Students were asked to discuss the value of these procedures in the management of “their” patient and to choose three initial tests they thought most helpful for finding the correct diagnosis. Students were encouraged to retrieve information on standard diagnostic procedures from books and lectures embedded in the six-week cardio-respiratory curriculum.

Test results were uploaded by the online tutor. Students were now given the opportunity to order as many further test results (just one at a time) as they needed in order to reach a final diagnosis.

As soon as students indicated they were ready to make a diagnosis, they were asked to submit the relevant code according to the International Classification of Diseases (version 10) which was available on the platform.

Students received feedback on their “guess” and were given the correct diagnosis. In order to complete the module, they were asked to search the “Guidelines on diagnosis and treatment of pulmonary arterial hypertension” issued by the European Society of Cardiology in 2004 (Galie et al. 2004) and upload it on the online platform.

Student enrolment, data acquisition and analysis

In March 2007, an invitation to sign up for the supplemental online module was mailed to all 183 students. Due to a limitation of resources, participation in the module could only be offered to 40 students. Students who responded to the invitation were randomly assigned to either the intervention group (participation in the online module) or the control group (interest in the online module but no participation possible).

Students’ performance levels before entering the study were determined on the basis of scores achieved in written multiple choice exams taken in their previous semester (winter term 2006/07). For each student, an individual percent score (total number of correct answers/total number of questions asked in all written exams × 100) was computed. On the last day of the six-week cardio-respiratory curriculum, all 183 students took a summative examination containing 68 multiple-choice questions assessing factual knowledge. These questions covered the whole range of cardiology and pneumology while the online module was focused on the differential diagnosis of chronic shortness of breath. Student achievements in the final multiple choice examination were compared between students in the intervention and the control group.

Students who had been assigned a place in the online module were asked to complete online questionnaires (EvaSys®, ElectricPaper, Lüneburg, Germany) on subject characteristics and attitudes towards computer-based learning in general and the new online module in particular. Personal agreement with specific statements was assessed using six-point Likert scales anchored by 1 (“fully agree”) and 6 (“completely disagree”) on the first and the last day of the six-week curriculum.

Data are presented as median ± SE of the mean. Statistical analyses were carried out using SPSS 12.0.1 (SPSS Inc., Chicago, Illinois, USA); p values (significance level 5%) were derived from χ 2 tests and T-tests. Effect size was calculated according to Cohen (1992).

Results

Enrolment phase

Of 183 students entering their fourth year in summer 2007, 74 volunteered to participate in the supplemental online module. As is displayed in Table 1, these students had achieved higher scores in previous exams than those who did not sign up for the online module (86.2 ± 0.8 vs. 84.1 ± 0.6%; p = 0.035). However, there was no significant difference in previous exam scores between the 40 students randomly allocated to the intervention group and the 34 students who were assigned to participate as the internal control group (p = 0.751). Among those interested in the online module, the proportion of female students (66.2%) was significantly greater than in those who did not sign up for participation (51.4%; p = 0.046).

Table 1

Examination scores

	Students not signing up for the online module (n = 109) (%)	Students signing up for the online module (n = 74) (%)	p value	Students signing up for the online module (n = 74)
				Control group (n = 34) (%)	Intervention group (n = 40) (%)	p value
Percent score in previous exams (winter term 2006/07)	84.1 ± 0.6	86.2 ± 0.8	0.035	85.9 ± 1.1	86.4 ± 1.1	0.751
Percent score in the end-of-course multiple choice test	78.6 ± 1.0	82.4 ± 1.0	0.009	79.5 ± 1.4	84.8 ± 1.3	0.006

This table summarizes student achievement levels before entering the fourth year and after taking the six-week cardio-respiratory course. Percentages refer to the amount of points scored in relation to the maximum total score. p values are derived from T-Tests

Achievements in end-of-course examinations

Students who had signed up for the online module continued to score higher in the final course examination (82.4 ± 1.0%) than those not interested in using the online platform (78.6 ± 1.0%; p = 0.009). Although there had been no difference in summative exam scores between the intervention and the control group before entering the fourth year, students in the intervention group achieved significantly higher scores in the final multiple choice (84.8 ± 1.3%) test than students in the control group (79.5 ± 1.4%; p = 0.006, effect size 0.67). In fact, exam scores of students in the control group were no different from those of students who had not signed up for the online module at all (p = 0.636).

Student satisfaction, perceived learning outcome and evaluation of the online module

Before starting the online module, participants indicated that they were interested in CAI (2.11 ± 0.14 on a six-point Likert scale) and that they felt well-trained to use computers and the internet (2.24 ± 0.19). Perceived knowledge on the differential diagnosis of dyspnea (3.50 ± 0.12) and the components of a lung function test (3.66 ± 0.16) was low, and students did not feel competent to devise a diagnostic management plan for a patient complaining of dyspnoea (4.61 ± 0.09). In addition, they did not know the costs of standard diagnostic procedures in internal medicine (4.24 ± 0.15; Table 2).

Table 2

Students’ perceptions of knowledge and skills

Statement	Self-assessment
	Before the course (n = 38)	After the course (n = 30)	p value
“I know the differential diagnosis of dyspnea”	3.50 ± 0.12	1.80 ± 0.13	<0.001
“I know which diagnostic information can be derived from a cardiac ultrasound report”	3.24 ± 0.18	1.47 ± 0.10	<0.001
“I know the single components of a lung function test”	3.66 ± 0.16	1.93 ± 0.16	<0.001
“I can devise a diagnostic management plan for a patient complaining of dyspnea”	4.61 ± 0.09	2.97 ± 0.16	<0.001
“I can interpret the results of an arterial blood gas sample”	3.03 ± 0.17	2.07 ± 0.19	<0.001
“I can estimate costs associated with standard diagnostic procedures in cardiology and pneumology”	4.24 ± 0.15	2.27 ± 0.17	<0.001

This table displays students’ perceptions of their own knowledge and skills before and after attending the six-week course on cardio-respiratory diseases during which they participated in the online module. Personal agreement with specific statements was assessed using six-point Likert scales anchored by 1 (“fully agree”) and 6 (“completely disagree”). p values are derived from T-Tests

The second survey carried out at the end of the 6-week course yielded favourable ratings regarding technical aspects of the online module (average of 2 on a six-point Likert scale). On a six-point scale with 1 being the best possible mark, students’ overall rating of the online module was 2.2 ± 0.13. Students had spent an average of 2.7 ± 0.3 h per week using the learning platform. All participants stated that their knowledge on cardio-respiratory diseases had increased during the 6 weeks during which the module was online (Table 2).

Discussion

Although the acquisition of higher-order cognitive skills such as clinical reasoning can be viewed as the ultimate goal of medical education, most examinations at medical school as well as many high stakes exit examinations (Fischer et al. 2005) still contain a considerable amount of multiple choice questions assessing factual knowledge (Palmer and Devitt 2007). While problem-based learning is primarily effective in promoting skills related to clinical reasoning (Norman and Schmidt 1992), it is also capable of imparting factual knowledge as has been shown in large cohort studies (Van der Vleuten et al. 1996). The present study investigated whether this effect of the teaching format is preserved when problem-based learning is transferred to a virtual environment. The two main findings were that students on a higher achievement level were more willing to explore innovative teaching formats and that students randomly allocated to using the online module scored significantly higher in a final multiple choice test than those who signed up but could not be allocated due to a lack of resources. Importantly, there was no difference in achievement level between the two groups before entering the study.

A recent meta-analysis reviewing a total of 201 published articles on the effectiveness of internet-based learning for students, postgraduate trainees and practitioners in a profession directly related to human or animal health concluded that, when compared to no intervention, internet-based interventions have large and significant effects on the acquisition of knowledge, skills and behaviours (Cook et al. 2008). The effect size of 0.67 between achievements in the intervention and the control group found in our study population is well within the range of studies included in this systematic review. The primary outcome of most of these studies was matched to the original learning objective underlying the teaching intervention (Kern et al. 1998); for examples see (Pederson et al. 2006; Sisson et al. 2006; Walsh et al. 2007); thus the finding of a significant effect of internet-based learning over no intervention is not surprising. However, in our study we found a significant effect in knowledge increase even though final exam questions were not closely matched to the content of the online module, i.e., the module did not specifically prepare students for the final exam. Instead, it might have increased students’ motivation to learn about cardiology and pneumology in general since shortness of breath is a common presenting complaint in both disciplines.

Interestingly, motivation to use innovative teaching and learning formats has been identified as an important confounder in non-randomized trials (Hallgren et al. 2002). All 74 students signing up for the online module expressed their motivation by responding to the mailed invitation to participate. Thus it can be assumed that ‘baseline motivation’ was comparable in the two groups. Although motivation was not formally assessed, it can be hypothesized that participation in the online module further increased student motivation to learn. This notion is corroborated by the fact that students in the intervention group spent an extra amount of time (almost 3 h per week) on the online module. This increase in learning time is within the range of findings from comparable WBL projects (Nathoo et al. 2005). The impact of web-assisted problem-based learning on overall learning motivation has not been studied in great detail and should be addressed in future trials as this aspect was not covered in the systematic review mentioned above (Cook et al. 2008).

Students’ self-assessment of knowledge and skills related to the management of dyspnea increased significantly over time; due to the lack of a completely randomized design with mutual exclusion of either web-based or contact teaching, the contribution of the online module to this increase cannot be measured. We cannot exclude an improvement in students’ rating on their self assessment for knowledge and diagnostic skills in the control group similar to the one observed in the intervention group since we only obtained self-assessments from members of the latter. In any case, the true impact of WBL on any of the learning objectives pursued by this teaching format (e.g., factual knowledge gain, clinical reasoning) can only be determined in prospective randomized trials (Letterie 2003; Raupach et al. 2009). However, media-comparative research is being heavily criticized due to methodological limitations (Cook 2005) one of which has once again become apparent in this study: It might well be that only motivated students will benefit from using WBL modules. Thus, medical educators and students could face an ever-increasing demand for individualization regarding the teaching formats used to arrive at the desired outcome.

During the enrolment phase, female students showed greater interest in using the web-based learning module. This observation is surprising in view of the fact that women have previously been reported to be more hesitant regarding the use of computers (Broos 2005). However, this aspect needs to be addressed in future research since the present study was not designed to specifically assess gender aspects.

Conclusion

This study examined the effect of virtual collaborative learning in respiratory medicine on student achievements in a multiple choice test covering the whole range of cardiology and pneumology. Students randomly assigned to using the problem-based online module on the differential diagnosis of chronic shortness of breath scored higher in a test of factual knowledge than students not included in the study despite comparable achievement levels before entering the study. The online module is likely to have increased students’ motivation to learn, and subsequent learning was not restricted to the content of the online module. However, it remains to be elucidated whether all students would benefit from using innovative teaching and learning methods regardless of prior motivation. In addition, future research will have to further assess gender aspects related to computer-assisted learning.