Case-oriented computer-based-training in radiology: concept, implementation and evaluation.

BACKGROUND: Providing high-quality clinical cases is important for teaching radiology. We developed, implemented and evaluated a program for a university hospital to support this task.
METHODS: The system was built with Intranet technology and connected to the Picture Archiving and Communications System (PACS). It contains cases for every user group from students to attendants and is structured according to the ACR-code (American College of Radiology) 2. Each department member was given an individual account, could gather his teaching cases and put the completed cases into the common database.
RESULTS: During 18 months 583 cases containing 4136 images involving all radiological techniques were compiled and 350 cases put into the common case repository. Workflow integration as well as individual interest influenced the personal efforts to participate but an increasing number of cases and minor modifications of the program improved user acceptance continuously. 101 students went through an evaluation which showed a high level of acceptance and a special interest in elaborate documentation.
CONCLUSION: Electronic access to reference cases for all department members anytime anywhere is feasible. Critical success factors are workflow integration, reliability, efficient retrieval strategies and incentives for case authoring.

Background

Access to radiological expert knowledge for a broad spectrum of users – from students to senior radiologists – is a continuous challenge in a routine clinical setting of a university hospital. A balance between established and new teaching methods is needed to provide high quality, peer-reviewed content.

The knowledge transfer normally takes place during teaching sessions in small groups, lectures or in the context of a specific case during daily routine by personal interaction e.g. between attendant physicians and residents, house officers and students. For this reason all members of the radiologic department are involved in a general teaching file.

In this context we developed a new software tool to collect and document relevant cases – according to all different levels of radiologic knowledge – from daily routine in an integrated manner by including all types of image sources. Access to this Intranet-based Teaching File – the "Lehrarchiv" – should be provided from any workstation within the department.

The design of this teaching file should – in the long run – integrate all department members both as case authors and normal users. At the same time a high, homogeneous quality level should be guaranteed from the beginning.

By a differentiated authorization system two opposite prerequisites had to be fulfilled: On the one hand the necessary technical infrastructure to provide access to the system should be ensured, i.e. access to the Intranet should be possible anytime anywhere within the department. On the other hand clinical data with patient information and images must be protected against unauthorized use.

Another critical issue concerning the infrastructure is acquisition and archiving of image data, i.e. the pathway of the image from its generation to the teaching file. All radiological techniques should be included, especially CT, MR, conventional X-ray, angiography and ultrasound. In this context our objective was to answer the following questions:

• Is a teaching file with access anytime anywhere within the radiologic department and smooth integration into routine workflow feasible from a technical point of view?

• What format and data structure is appropriate for a radiologic case description?

• What kind of retrieval functions are required?

• Is an electronic radiologic teaching file accepted by students and physicians?

Materials and Methods

Image acquisition

In general there are three methods for image acquisition for a teaching file:

1. Secondary digitization of a primary analogous image e.g. a conventional radiographic film and subsequent electronic transfer transfer to the teaching file. This method is rather time-consuming and implies loss of image quality due to digitization, but is typically required when external radiographs or slides are included; the additional effort must be weighed against the relevance for the case presentation on an individual basis.

2. Transfer of a digital image from an external data source into the teaching file. This may be a digital image primarily stored in an electronic archive or digitized for other reasons.

3. Direct transfer of primary digitally generated, recent patient images (CT, MR etc.) into the teaching file.

All three methods have been implemented for our teaching file. The problems associated with different media are very common at present and characterize the current change from classical film-based to computerized radiology.

The direct digital transfer is – with respect to workflow integration and image quality – obviously the most attractive method and is the preferred way in the context of the establishment of PACS. This method becomes even more relevant, because the number of images per case is increasing continuously due to sophisticated radiological techniques. For this reason manual transfer of single images becomes more and more impractical.

The technical implementation was realized with a DICOM (Digital Imaging and Communications in Medicine) [1] interface – an international multi-vendor standard -, which enables the transfer of selected images to the teaching file in a very simplistic manner, similar to printing images. In order to preserve confidentiality the patient's identification is eliminated when a case is published.

An important success factor for a radiologic teaching file consists in efficient retrieval mechanisms, especially when searching for similar cases. Similarity is referring both to localization and pathology of a radiological finding. For this reason the internationally established ACR-Code [2] was applied, which allows a precise classification of cases according to these dimensions.

Software concept

The teaching file system acts basically as a DICOM receiver, i.e. the case repository behaves like a printing device.

By means of a web frontend and individual user accounts each author can access and edit his own cases on the teaching file server. Authorized users can release cases for the public case repository.

We applied an iterative software design approach with rapid prototyping, i.e. we built prototypes, tested and continuously refined them; altogether about 10 iteration cycles were required. The data structure consists of two tables (cases and images) with altogether 23 items.

Hard- and Software

The technical concept is based on established Internet tools. An Apache [4] Webserver (version 1.4.2) on a Linux [5] machine (Distribution SuSE 6.3) provides PERL[6] (version 5.005) programs accessing a relational database [7]. The department network consists of a Gigabit-Backbone and NT-workstations with 100 MBit connections.

Evaluation Method

To quantify user acceptance we applied a paper based survey. Categorized items were recorded on a six point Lickert scale, from 1=absolutely correct to 6=completely wrong. Scores are given as mean +/- standard deviation.

Results

Case authoring

After a program development phase of approximately one year and 18 months of clinical routine use there are now approx. 350 released cases accessible (283 are work in progress); 31 users from our radiologic department are registered, but only eight of them contributed to more than 80% of cases.

The radiologist can select relevant images on a DICOM workstation during his routine work and send these to the teaching file through the DICOM-Send procedure in analogy to printing. At a later time he can connect to his teaching file account and edit the case. In addition he can upload secondary digitized images.

The textual description (free text) consists of four sections:

• Diagnosis, which is mandatory,

• medical history / clinical presentation / laboratory,

• radiologic findings and

• comment.

An efficient retrieval strategy is required to find "similar" cases with both high precision and recall. For this reason a systematic code is necessary. We developed a specific coding tool to enter the ACR-code in both dimensions (localization / pathology) for each case. If several findings are present, up to three ACR code-pairs can be stored. Due to the hierarchical design of the ACR code (e.g. localization Gastrointestinal System – Stomach – Pylorus) similar cases – both in terms of localization or pathology – can be found easily. In addition for each case administrative information (e.g. author ID, date of release) is stored automatically.

Case retrieval

Retrieval of cases in the teaching file, which is shown in Fig. 1, is possible based on diagnosis, ACR-code, keyword or any combination. Due to the hierarchical structure of the ACR code similar pathologies or localizations can be retrieved.

Figure 1

Retrieval of cases [translated to English] in the teaching file is possible by diagnosis, ACR-code or a combination of both. A list of matched cases for the term "aneurysm" with ACR-location code 5 (Heart and Great Vessels) is displayed. In addition, specific case series can be selected from a pulldown menu. Authorized users can aggregate single cases to a case series, typically on a specific teaching issue (e.g. selected lung cases)

Fig. 2 presents the browsing mode of the teaching file. All information on a particular case is presented. Using the navigation bar, any text information can be excluded; by this means the user can assess his radiological knowledge and diagnostic abilites.

Figure 2

Browsing mode of the teaching file [translated to English]. All information to a particular case is presented. Using the navigation bar, any text information can be excluded; by this means the user can assess his radiological knowledge and diagnostic abilites.

Fig. 3 provides a detailed view of a single image in the system. By means of navigation links all images associated with a case can be viewed in high resolution.

Figure 3

Detailed view of a single image in the teaching file [translated to English]. Using the navigation bar, all images of a case can be viewed sequentially in high resolution.

In addition to the retrieval function users can select specific case series which are aggregated by authorized radiologists, typically on a specific teaching issue (e.g. selected lung cases). This provides added value for the teaching file: it can be used not only for retrieval of single cases, but also for presentations e.g. in scientific workshops. Individual teaching sessions on specific topics, e.g. for students, are also provided with this tool.

Data analysis

The system has now been in routine use for approximately 18 months. Fig. 4 shows the development of the total number of cases in the radiologic teaching file over time differentiated by case status (created / released). The interval between creation and release of a case is 73 +/- 87 days (mean +/- 3.0.).

Figure 4

Development of the total number of cases in the radiologic teaching file during the first 18 months of operation. The upper line represents the total number of cases, the lower one corresponds to the released cases.

After 18 months there were 583 records in the table 'Cases' consisting of 19 items each, and 4136 records in the table 'Images' with 4 items each.

Evaluation

The system was evaluated by 101 medical students (3rd year) who studied two case series (conventional thorax and liver CT/MR) in small groups (max. 9 persons each, assisted by an instructor) during their mandatory radiological course. We included all students from a semester to avoid a selection bias with respect to individual computer expertise. The learners themselves judged their computer knowledge as average (2,92 +/-1,06).

A student questionnaire showed a high level of acceptance (see Fig. 5) according to the content of the system (comprehensible 2,51 +/- 0,96; appropriate description 2,50 +/- 1,07; subjective learning effect 2,88 +/- 1,33; recommendation of the system to colleagues 2,31 +/- 1,24; alternative to handbooks 1,73 +/- 0,90) and image quality (brightness 2,05 +/- 0,86; contrast 2,32 +/- 0,95; size 2,24 +/- 1,10); however, we did not evaluate the gain in knowledge of the students. The software quality of the teaching file was honoured in terms of speed, reliability and case design.

Figure 5

Student questionnaire indicating a high level of acceptance. (results presented as boxplots ;6 point Lickert scale; n = 101)

We also asked the students for the optimal number of images to be contained in a "typical case": 10 +/- 5 (mean +/- S.D.). This is influenced by the selection of cases with a focus on CT and MR – probably less images were appropriate for conventional chest X-rays.

To improve the system the students suggested to apply interactive pointers or similar mechanisms to highlight specific radiological findings. From a technical point of view this can be implemented easily, but the authors argued this would increase the amount of authoring work substantially.

The results from the questionnaire are consistent with the subjective impression gained during the teaching sessions.

We also did an evaluation with residents (n = 15) on a case series consisting of more complicated cases selected by a senior attendant radiologist. There was again a high level of acceptance (comprehensible cases 1,47 +/- 0,74; recommendation of the system to colleagues 1,60 +/- 0,83).

Discussion

Computer-Based-Training (CBT) in Medicine has proven its clinical potential in several settings ([8],[9],[10],[11]). Compared with the situation 10 years ago [12], technological problems such as insufficient computer performance, lack of storage capacity or inadequate display devices can now be solved.

Despite these advances electronic peer-reviewed and up-to-date teaching files in radiology which can be accessed anywhere in the department are still very rare. Recently, reports of successful CBT in the field of radiology have been published ([13],[14],[15],[16]), but typically the radiological content of the systems is very limited (both in terms of cases and in terms of authors).

Lessons learned

The first objective of this study concerns technical feasibility of a teaching file with access anytime anywhere within the radiologic department and smooth integration into routine workflow. From our experience – after 18 months of routine operation – an Intranet-based system with PACS-interface (i.e. DICOM-compatible) can fulfill this task; it is very important that images can be sent directly from the radiological workstation to the case repository.

There is a wealth of CD-ROMs and other resources ([17],[18],[19]) providing 'snapshots' of radiological knowledge, but a collection of routine clinical reference cases for a specific department and its particular devices is needed. Due to the sophistication of imaging methods a clinically relevant teaching file must be updated continuously. This is only feasible by minimizing the effort to create and maintain this database.

The second objective of this study concerns the ideal format and data structure for a radiologic case description. Obviously from an academic point of view a detailed description of the case using controlled vocabularies, including access to a complete electronic patient record would be desirable. In our setting this was not feasible because it is too time-intensive for the authors. Therefore we decided to apply a compact approach with free text data for medical history, radiological findings, diagnosis and comment. Only location and pathology were entered in a coded manner, using the internationally established ACR code.

The third objective of this study addresses retrieval functions, which obviously depend on the data available in the system. From our experience both systematic and free text retrieval mechanisms are important. We applied the ACR code, which is characterized by a high level of granularity, to provide a systematic access to the database. By this means 'similar' cases – concerning both localization or pathology – can be found easily. We also considered to apply a controlled vocabulary for diagnosis, but in our setting the effort for systematic coding would have been too high.

Case series arranged by senior radiologists are another method to organize the case collection and facilitate access.

The fourth objective concerns acceptance of the system by students and physicians. We did not measure the learning success of students, but the questionnaire indicates a high level of acceptance which is in line with observations during the course. The constant growth of peer-reviewed cases over time provides good evidence for acceptance by the physicians.

From our experience, integration into daily routine work is the key success factor for a teaching file. Secondary digitization of images is laborious and implies a loss of quality. Media conversions are such an obstacle that only in a filmless setting with PACS and smooth integration by DICOM-interfaces relevant cases can be collected during daily routine.

Knowledge management

To maintain a high level of quality in patient care is a major challenge in the context of labour turnover and continous sophistication of radiological methods. The high degree of specialization makes knowledge transfer an important success factor for the department as a whole. There are many recent publications according to the relevance of knowledge management ([20],[21]) within organizations.

A technical system like our teaching file is a small, but important building block of a department-wide knowledge management strategy, because it provides access to relevant teaching material anytime anywhere. Our evaluation provides evidence that electronic teaching files are well accepted if quality of content and technical reliability is assured. From a medical informatics point of view, such case repositories offer the opportunity of building decision support systems in the future. Whilst a PACS archive is a huge collection of images, a teaching file is a knowledge database. Due to the systematic coding, an automatic search for similar cases is possible. Intelligent image analysis programs in the past mostly failed because a sufficient knowledge base could not be gathered. Meanwhile sophisticated methods for knowledge discovery and data mining ([22],[23]) have been developed, but these techniques require both high-volume and high-quality reference datasets.

By national and international cooperation the vision of peer-reviewed, up-to-date and comprehensive teaching files in radiology could become reality.

Pre-publication history

The pre-publication history for this paper can be accessed here: