Acceptance of medical training cases as supplement to lectures.

INTRODUCTION: Medical training cases (virtual patients) are in widespread use for student education. Most publications report about development and experiences in one course with training cases. In this paper we compare the acceptance of different training case courses with different usages deployed as supplement to lectures of the medical faculty of Wuerzburg university during a period of three semesters.
METHODS: The training cases were developed with the authoring tool CaseTrain and are available for students via the Moodle-based eLearning platform WueCampus at Wuerzburg university. Various data about usage and acceptance is automatically collected.
RESULTS: From WS (winter semester) 08/09 till WS 09/10 19 courses with about 200 cases were available. In each semester, about 550 different medical students from Würzburg and 50 students from other universities processed about 12000 training cases and filled in about 2000 evaluation forms. In different courses, the usage varied between less than 50 and more than 5000 processed cases. DISCUSSION: Although students demand training cases as supplement to all lectures, the data show that the usage does not primarily depend on the quality of the available training cases. Instead, the training cases of nearly all case collections were processed extremely often shortly before the examination. It shows that the degree of usage depends primarily on the perceived relevance of the training cases for the examination.

Introduction

Medical training cases are now in widespread use. Reports of successful use in different domains can be found e.g. in neurology [11] rheumatology [14], [12], hematology [7], pediatrics [6] internal medicine [1], general medicine [10], etc. The publications are based, however, usually only on one domain. Various surveys show that students want training cases for all courses. Our main question is: it is sufficient to provide the students with good training cases and does the usage rate mainly depend on the perceived quality of training cases? In this paper, we compare the acceptance of courses with training cases, which were used in addition to various lectures at the Medical Faculty of Wuerzburg University, to identify success factors, e.g. whether the usage rate depends on the quality of the cases, the degree of difficulty, the ease of the user interface, the processing time, the examination relevance or the variety of training cases per course. The conditions in different courses are comparable: Since 2007 a uniform infrastructure for easy development and deployment of training cases has been created through a cross-faculty Blended Learning Project at the University of Wuerzburg funded by student fees. In addition, limited resources were available for the content development for all interested subjects. After a year, medical training cases were developed in 19 courses and made available to the medical students in their clinical years. To avoid random fluctuations in one semester, this study uses data from a period of two to three consecutive semesters.

Methods

A typical training case consists of a sequence of consecutive information and question sections with a case discussion at the end. In information sections textual and/or multimedia patient data are presented, in question sections questions about diagnoses, tests, treatments, image interpretations, and general background knowledge are asked. Different question formats such as multiple-choice, long-menu, word, number, and essay questions are available for this purpose. The answers are graded automatically to give the students as users direct feedback (with essay questions, instead of an assessment only the correct solution is shown so that the students can assess themselves). While normally the case presentation is linear, it may differ if students have to order relevant facts by only showing the patient data the student actually requested.

The training cases were created with the authoring system CaseTrain [4]. Based on experience with the previous systems D3Trainer [13] and d3web.Train [5] as well as Casus [2] Campus [8], [3], Docs and Drugs [9] and others, special emphasis was put on a simple case editor component, which teachers could understand without special training, and an intuitive end user interface for the students (see [15]). For authoring a case the word processing system WORD is used, in which the cases are entered in a table structure and then uploaded, parsed, reviewed and approved via a web application (CaseTrain Manager), making them accessible for the Moodle-based Wuerzburg learning platform Wue¬Campus. The user interface for the students is programmed in Adobe Flash, to avoid problems with different browsers. The screen is divided in three parts: on the left side is a large information section and on the right side a question with the question text and their alternative answers (see Figure 1 (Fig. 1)). A detailed description of CaseTrain is given in [4].

Figure 1

Screenshot from a training case in CaseTrain.

Training cases were created with CaseTrain for 19 clinical subjects. Because a text system (WORD) is used for case input, the typical approach was that a lecturer outlined the content and a student assistant formatted the text including pictures or videos and - after review by the lecturer – uploaded it via a web interface. Some courses were funded by the Virtual University of Bavaria (VHB); in this case the cases were usually created by a physician. The cases were provided to students via the Wuerzburg WueCampus learning platform, which also allows uploading slides and other teaching materials for the lecture and provides good communication facilities for teachers and students. The teacher can decide whether the processing statistics are recorded anonymous or personalized. Most teachers selected personalized coverage, as this allows students to view personal statistics on their success in case processing. For each case date, processing time and the answers to the questions are recorded automatically. If not all questions were answered, the case in Table 1 (Tab. 1) is counted as an "uncompleted case processing". For complete processing the success is calculated on the basis of the automatic evaluation of responses with a score. In personalized processed cases it is possible to differentiate between the successes of the initial and subsequent processing of a user, so the proportion of the percentage of initial successful processing to all initial processing is calculated as an indicator of the degree of difficulty of the cases. In addition to these automatically logged data, the users are asked three evaluation questions at the end of each case: a grade for case content and ease of use as well as the possibility of a free-text comment. The evaluation was deliberately kept very simple in order to achieve a high response rate.

Table 1

Usage data of training cases in different courses and the results of a small evaluation questionnaire, which appears at the end of each case. The figures are based on an average semester, which was calculated from the average of three (or from two with "*") semesters during the period from WS 08/09 to WS 09/10. The courses have been sorted by the number of complete processed cases and are divided in 5 acceptance classes: more than 1000 complete processed cases per semester, 200-1000, 100-200, 50-100 and less than 50 processed cases. For anonymous courses no initial processing and therefore no difficulty levels could be determined. In SS 09 and WS 09/10 the course of Clinical Immunology / Rheumatology was also made available for students of other Bavarian universities via the vhb with approximately 700 complete processed cases by about 50 students per semester and similar characteristics in this subgroup (the scores on "content" are a little better and the ones on “user interface” a little worse than with the students in Wuerzburg and the processing time is a little longer). In further courses such as Medical Terminology or Clinical Chemistry independent questions instead of training cases were offered and therefore they are not included in table 1.

Results

Table 1 (Tab. 1) shows utilization data of the various medical courses with the following information: number of offered training cases, number of individual users, total number of complete and partial processed cases as well as the number of complete processed cases per available case, average duration of complete processing in minutes, number of completed evaluation questionnaires with average grades for content and user interface, the average level of difficulty as a percentage of complete, not successful initial processing of all cases and whether processing of cases was required for the course or not. In 13 courses, the data are averaged over three courses in subsequent semesters (winter semester 2008/2009, summer semester 2009 and winter 2009/2010), while in 6 with "*" marked courses the data are averaged over only two courses (usually starting in SS 09). Through the formation of an average course of several semesters one can abstract from random fluctuations in one semester, so data are more meaningful than if only the data from one semester are evaluated. The summary of several courses to acceptance classes on the basis of the usage frequency is also used to compensate random fluctuations in individual courses by forming averages.

We measure the acceptance of cases by the number of complete processed cases and have divided the courses into five acceptance classes (> 1000, 200-1000, 100-200, 50-100, <50 case edits) to check if the acceptance classes correlate with case or course properties. In the two upper acceptance classes not only the total number of processed cases but also the number of workings per case is highest (with exception of the Geriatrics course which has only one case). It turns out that the acceptance classes 1 to 4 have relatively uniform scores for user interface and content from 1.9 to 2.1 and only the very small acceptance class 5 has much better grades. However, there is a relationship between the difficulty level of the cases and the acceptance classes because the two upper classes have a lower difficulty level than the three lower classes. The cases of the classes 2 and 4 have an approximately twice as long processing time as the cases of the classes 1 and 3. The number of cases per course correlates slightly with the frequency of case use because courses for acceptance class 1 provide the highest number of cases. For the remaining classes 2 to 5 no clear correlation between the number of cases and the frequency of use can be seen.

Furthermore, we studied the temporal usage curve for each course. This shows a very striking pattern in all courses which has one outstanding peak per semester, namely the day before the exam is written (or - if the exam takes place in the afternoon - the day before and the day of the exam). We show two curves as examples: one for a busy and one for a less busy course (Infectious Diseases in Figure 2 (Fig. 2), Geriatrics in Figure 3 (Fig. 3)), whereby we only show the data of 2 semesters (SS 09 and WS 09/10). The number of complete processed cases in Infectious Diseases in the week before the exam in SS 09 (16.7 - 23.7.) is with 2420 almost three times as large as in the rest of the summer semester together (827, from 15.4 - 15.7.). In the WS 09/10, the corresponding figures are 2083 complete processed cases in the examination week compared to 485 in the rest of the winter semester. The same pattern, only on a much lower level, is also reflected in Geriatrics, in which in the respective exam week nearly three times as many cases are fully processed as in the rest of semester.

Figure 2

Daily number of complete processed cases in the course Infectious Diseases in SS 09 and WS 09/10.

Figure 3

Daily number of complete processed cases in the course Geriatrics in SS 09 and WS 09/10.

Only in a course of Clinical Immunology / Rheumatology, a different pattern (see Figure 4 (Fig. 4)) shows up. Here, the relation of total processed cases in the exam week to the remaining term is approximately 1:1 (in SS 09 3739 to 3562 processed cases in WS 09/10 2767 to 2701 processed cases). Another special feature of this course is that the successful processing of 10 - 20 training cases is mandatory (see last column in Table 1 (Tab. 1)), whereupon repeated processing of a case is allowed (up to SS 09 20 training cases, from WS 09/10 only 10 training cases with this condition tested immediately before the exam). One hypothesis is that the students fulfill their duty during the course and then repeat the cases the week before the exam, which could explain the specific processing pattern. To test alternatively, whether the cases were processed only due to obligation, we have examined how many students fall into the pattern of "minimalist" and solve just as many cases as they have to. In WS 08/09, there were 36, in SS 09 only 8 and in WS 09/10 16 minimalists who had solved at most 2 cases more than they had to, which means an average of about 15% minimalists. The remaining 85% have mainly solved all of the available cases, many even multiple times. A further indication that the high acceptance of processed cases does not depend primarily on the obligation to solve the cases is that the number of processed cases in WS 08/09 and WS 09/10 was about the same, although in WS 09/10 only 10 instead of 20 cases were mandatory. The peak values in the week before the exam also show that students perceive the case processing as a very good preparation for the exam.

Figure 4

Daily number of complete processed cases in the course "Clinical Immunology/ Rheumatology" in SS 09 and WS 09/10.

Discussion

The initial question whether offering good training cases to the students suffices for high acceptance, assuming that the usage rate mainly depends on the perceived quality of the training cases, must be answered "no" based on the data. There is no correlation between the average evaluation score for content and user interface of the cases and the frequency of case processing. The fifth acceptance class with the smallest number of cases processed even gets the best ratings, but only bears little significance due to the small number of users and evaluations. The length of processing time is not an indicator for the frequency of case use either. Of course, one cannot conclude from that, that the quality of the cases has basically no effect on processing frequency, but we rather interpret the results as following: the quality of the cases from the perspective of the students was perceived as good in virtually all courses and therefore cannot serve as an explanation for the difference in the frequency of use. The difficulty level of the cases, on the other hand, seems to have an impact on the case usage: if the percentage of cases which the students cannot solve at the first try is relatively high, the frequency of use is rather low on average. One explanation could be that the students had not been taught the knowledge needed for solving the cases in the lecture which means that the cases are not well matched to the course content.

Although it is widely known that medical students prepare for exams in a very targeted way and therefore the case processing rate increases before exams, we were still surprised by the definiteness and strength of the relationship between the frequency of case processing and the examination date. In Infectious Diseases and most other courses the case processing rate in the week before the exam is three times higher than in the entire rest of the semester which lasts about 14 weeks. This means that the frequency of use in this week is higher by about a factor of 40 than in an average week. This pattern is characteristic of all classes of acceptance, i.e. in courses with high and a low utilization. Only the course Clinical Immunology / Rheumatology is an exception. Since this is the only course in which the processing of a portion of the cases was mandatory and it was also shown that about 85% of the students solve significantly more cases than they have to, you could see the obligation as an incentive to solve the cases already well before the exam (for the first time).

In our data, we couldn’t find any other parameter than the difficulty level to explain the difference in the frequency of use of training cases. In addition to possibly varying interests of students in the subjects, an obvious hypothesis is that cases are processed more frequently if their perceived relevance for the exam is higher. This would also explain why difficult cases are processed less often if the conjecture is true, that this is a sign for a reduced match between case content and course resp. examination content. The perceived relevance for the exam may depend on different factors, for example on how much the teachers or fellow students recommend solving training cases in preparation for the exam, how well the above-mentioned congruence of lectures, examination and case content is, what learning alternatives the students have and how attractive they are, how well the various learning alternatives complement, how relevant the subjects are for earning a medical degree, what overall learning and exam preparation strategies the students have, etc. A more detailed investigation of these possible factors and the coherence with the features of training cases would be very interesting, but beyond the scope of this work. This would require much more data about the learning and exam preparation behavior of the students.

In summary it can be said, that it is not sufficient to offer easy to use and good quality training cases to the students in order to achieve a high utilization rate. Training cases are dealt with mainly just before the exam, so the perceived relevance for the exam seems to be crucial for the usage rate.

Competing interests

The authors declare that they have no competing interests.