Critical care procedure logging using handheld computers.

INTRODUCTION: We conducted this study to evaluate the feasibility of implementing an internet-linked handheld computer procedure logging system in a critical care training program.
METHODS: Subspecialty trainees in the Interdepartmental Division of Critical Care at the University of Toronto received and were trained in the use of Palm handheld computers loaded with a customized program for logging critical care procedures. The procedures were entered into the handheld device using checkboxes and drop-down lists, and data were uploaded to a central database via the internet. To evaluate the feasibility of this system, we tracked the utilization of this data collection system. Benefits and disadvantages were assessed through surveys.
RESULTS: All 11 trainees successfully uploaded data to the central database, but only six (55%) continued to upload data on a regular basis. The most common reason cited for not using the system pertained to initial technical problems with data uploading. From 1 July 2002 to 30 June 2003, a total of 914 procedures were logged. Significant variability was noted in the number of procedures logged by individual trainees (range 13-242). The database generated by regular users provided potentially useful information to the training program director regarding the scope and location of procedural training among the different rotations and hospitals.
CONCLUSION: A handheld computer procedure logging system can be effectively used in a critical care training program. However, user acceptance was not uniform, and continued training and support are required to increase user acceptance. Such a procedure database may provide valuable information that may be used to optimize trainees' educational experience and to document clinical training experience for licensing and accreditation.

Introduction

Handheld computers, or personal digital assistants (PDAs), are becoming increasingly used in medicine for a variety of functions [1]. From an educational perspective, handheld computers have been used to track trainees' educational experience and generate procedural reports in family medicine [2], emergency medicine [3,4], surgery [5], obstetrics [6], and anesthesia [7]. An advantage of using handheld computers to document procedural experience is that data can be entered directly into the database immediately after the procedure has been performed, preventing data loss and avoiding the need for duplicate entry [1]. In many jurisdictions, regulatory agencies require documentation of procedural experience for licensing or hospital privileges.

We describe the implementation of an internet-linked handheld computer based procedure logging system in a critical care training program. The feasibility of this system, as well as its perceived advantages and barriers, were evaluated.

Methods

Setting

Eleven trainees (seven first year, four second year), who enrolled in the academic year July 2002 to June 2003 in the 2-year critical care training program at the University of Toronto, were provided with handheld computers. The trainees ranged from postgraduate years 4 to 6 and rotated through six academic hospitals.

Hardware and software

The Palm Vx handheld device (Palm Inc., Santa Clara, CA, USA) was provided, but trainees were allowed to use their own Palm operating system devices. A customized software program for logging critical care procedures (IqLog Critical Care; Infiniq Inc., Mississauga, Ontario, Canada) was developed, allowing easy data entry using drop-down lists. Data fields included the procedure date, the trainees' role (i.e. whether the trainee was directly supervised, not directly supervised, or acted in the role of supervising junior staff), hospital, and supervising attending staff physician. An optional numeric medical record number could be entered. The procedure list included 63 options in six main categories (Table 1): airway, chest, lines, gastrointestinal/genitourinary, diagnostic and other procedures. Checkboxes and drop-down lists were used for all data entry, with an option to enter text for patient identifier and personal notes. The entered procedures could be reviewed and edited on the handheld computer screen.

Table 1

Categorization and procedures available from the handheld procedure logging drop-down lists

Category	Procedure	Examples
Airway	Intubation	Direct laryngoscopyBougieCricothyroidotomyPercutaneous tracheLaryngeal maskFiberoptic oral intubationNasotracheal intubationJet vent via angiocathTracheostomy change
	Bronchoscopy	Awake oralAwake nasalVia endotracheal tubeRigid
	Mechanical ventilation	ConventionalNoninvasiveNonconventionalNitric oxideProne positioning
Chest	Chest tubes	InsertionRemoval
	ThoracocentesisPericardiocentesis
Gastrointestinal/genitourinary	Bladder catheter	FoleySuprapubic catheter
	Continuous renal replacement therapy	HemodialysisHemofiltration
	Peritoneal tap	Diagnostic peritoneal lavagePercutaneous drain
	Nasogastric tube	Simple nasogastricNaso-jejunal
	Blakemore-Sengstaken
Lines	Central venous	SubclavianFemoralJugular
	Arterial	RadialFemoralPedal
	Dialysis catheter	FemoralSubclavianJugular
	Intra-aortic balloon pump	InsertionRemoval
	Pulmonary artery catheter
Diagnostic		Joint aspirationLumbar punctureBone marrow aspirationMuscle biopsySkin biopsyUrine microscopyBlood film reviewIntracranial pressure monitoring
Other procedure	Transvenous pacer	ECG guidedBalloon flotation
	Transthoracic pacer
	Opening of surgical wound	Reopen sternotomyOther surgical site
	Patient transport	InterhospitalIntrahospital
	Family conference	RoutineWithdrawal of careOrgan donation

Communication software (IqSync; Infiniq Inc.) allowed transmission of data from the handheld device to a central data repository via the internet. This could be achieved by synchronizing with a desktop computer with internet access and required the installation of communication software. Because of initial difficulties experienced by some users in setting up the communication software, an alternative system was developed. Infrared-enabled modems were set up at three hospital sites; these allowed transmission of data via infrared to the modem, which connected to the internet via an analog telephone line (Fig. 1). Each trainee was provided with a username and password, and was able to view only his or her procedures via a secure website. The Program Director of Critical Care was able to access individual data from each of the trainees using the same website.

Figure 1

Outline of the procedure logging system. Data are entered into (a) the handheld device, transferred via (b) an internet connected computer or (c) an infrared telephone modem to the internet server, and is accessible via a (d) secure internet website.

To provide additional benefit to the trainees, applications with medical content relevant to critical care were installed on the handheld devices. The program iSilo (iSilo v. 3.05; ) was used as a reader for text documents. The medical calculator MedCalc was installed (MedCalc version 4.3, ), and trainees were encouraged to install the pharmacopeia ePocrates on their devices.

All trainees participated in a 2-hour training session to familiarize them with the handheld devices, procedure logging, and the medical reference software. The Palm OS Emulator (Palm Inc.) was used to provide an interactive presentation. The trainees were given a support contact e-mail address and telephone number for trouble-shooting. Reminder e-mails and requests for feedback were frequently sent to all participants.

To avoid breaches in patient confidentiality, identifying patient information was limited to the last four digits of the medical record number. Data transfer to the central database incorporated 128-bit encryption. The software was programmed to allow deletion of procedures stored on the handheld device after they had been uploaded to the central database.

Outcome measures

Outcome measures were targeted at identifying feasibility, acceptance, benefits, and disadvantages of this computerized critical care procedure logging system, and to review the scope of experience in clinical procedures that our trainees received.

Feasibility and acceptance of the system were assessed by tracking utilization, trouble-shooting calls, and complaints or suggestions from users. Trainees were defined as 'regular users' if they uploaded procedures to the central database at least once a month for 6 months. The database generated by trainees was analyzed at the end of the academic year to evaluate the scope of their experience.

The benefits and disadvantages were assessed through a survey distributed 3 months after implementation of the system. This survey also evaluated previous computer and handheld experience, as well as prior methods (if any at all) used for procedure logging. This survey explored usability and satisfaction with the procedure logging system.

Results

Although all 11 trainees initially used the system, marked variation was noted in the number of procedures logged by individual trainees during the academic year (Fig. 2). Two out of four senior fellows (50%) and four out of seven junior fellows (57.1%) rapidly adopted the technology and became regular users. Tracking over time revealed a progressive decrease in the number of procedures logged (Fig. 3).

Figure 2

Procedures logged by individual trainees during the 2002/2003 academic year. Trainees marked with an asterisk met criteria for 'regular users' (i.e. they uploaded data at least once a month for 6 months).

Figure 3

Tracking of total procedures logged per month by the 11 critical care trainees.

During the academic year, a total of 914 procedures were entered into the database (mean 83.0 per trainee, range 13–242). First-year trainees logged more procedures (mean 98.5) than did second year trainees (mean 56). The most common procedural categories were 'lines' and 'airway' procedures. No significant difference was noted in the number of procedures performed in each of the program's teaching hospitals. Variability was found in the number of procedures when analyzed by supervising attending physician (Fig. 4). However, the attending physician was not identified on 32.7% of procedures logged. The majority (67%) of procedures performed by trainees were not directly supervised.

Figure 4

Procedures logged by trainees, according to trainee role and supervising attending staff. These data represent the 68.3% of procedures for which the attending staff were identified.

All trainees completed the survey, which indicates that they all owned a home computer with internet access. Nine (82%) had previous experience using handheld computers. Only one trainee tracked procedures prior to this program, by keeping a handwritten log. The procedure logging program was described as either very useful or somewhat useful by seven (64%) of trainees. The most common reason cited for not logging procedures was related to initial problems with the data uploading process. Other reasons included being in clinical rotations in which procedures were not performed (such as research or outpatient respirology) and a perceived lack of need to collect procedural data.

The support service was predominantly utilized by those trainees who chose to install the data transmission software at home. The handheld interface was found to be intuitive, and few trouble-shooting requests were received in this regard. The most common hardware problem encountered with the handheld device was battery failure, occurring only in infrequent users of the system who did not use their handheld for periods greater than 2 weeks. Battery failure was associated with loss of data on at least three occasions. Suggestions for additional procedures to be added to the software were addressed.

Discussion

We implemented and evaluated a handheld computer procedure logging system, with internet-based data transfer to a central data repository. The system was found to be technically feasible, although initial problems were encountered related to the internet uploading process. All users successfully documented procedures on their devices and uploaded them to the central database. The database was a potentially valuable resource and it provided the Program Director with insight into the scope of procedural training experienced as well as the sites and clinical teachers involved. It should be noted that the procedures were entirely self-reported; we made no attempt to evaluate the accuracy of this information.

This procedure logging system has the potential advantage over other handheld systems [2-7] in that it combines mobile data entry on the handheld with centralized data storage on an internet-based server. The centralized data storage allows access to the database in real time, allowing continual evaluation of trainees. However, the most common technical problem encountered was related to installation and setup of this communication software. The ability to upload data from home was considered a useful feature but required additional technical support

We found that only 55% of our small group of trainees used the logging system on a regular basis, with a decrease in procedures logged over time. The 914 procedures logged therefore represent only a proportion of the procedures performed by our trainees during the academic year. Of note, procedure logging was optional; mandatory use of the system may be an important consideration if training requirements change to mandate a procedure log. Other studies have reported variable compliance with similar systems. Garvin and coworkers [2] found that 88% of their family medicine residents collected data on their handheld computer and 73% of them reported daily use. We previously reported a 38% regular use rate 5 months after the introduction of the procedure logging program in a general surgery program of 69 trainees [5]. Others have reported difficulties in acceptance when introducing handheld computing technology, especially among the subset of staff/faculty physicians [8]. This may partially be related to user seniority or age. Handheld computer use by physicians is increasing, particularly in younger age groups, in which utilization is greater than 50% [9]. Compliance with such procedure logging systems may improve in the coming years as this younger cohort moves into senior positions. As technology improves based on lessons learned from experiences such as that gained in this study, increased acceptance is likely. Training in the use of the handheld device and software is critical [10], and although we provided an initial training session and follow-up support, this may not have been adequate.

Although logging of procedures may not be required by all licensing authorities and hospitals, there are clearly benefits to having these data available [11]. At the present time, documentation of procedural experience is not a requirement for critical care trainees in Canada, although the Program Director is required to ensure that trainees are competent in certain core procedures. In the future, such documentation may become increasingly important. Given current concerns over medical errors [12] and the fact that many of these errors may be occurring in the critical care environment, documentation of procedure performance in training and during maintenance of competency programs is likely to gain importance. As we face a shortage of critical care medicine practitioners, it may be necessary to better define those multidisciplinary practitioners who are able to function in this capacity. Procedure logging in some form may be a valuable component of such an effort.

Conclusion

This electronic procedure logging system was successfully implemented and generated a large database of trainees' procedural experience. However, the system was used on a regular basis by just over half of the trainees. Problems identified in the areas of training and data transmission are now being addressed. This system has the potential to provide valuable information for the individual trainee as well as for program directors and governing bodies.

Key messages

• We implemented an internet-linked handheld computer procedure logging system in our Critical Care training programme. Although effective, user acceptance was not uniform and required continued training and support.

• The database generated may be useful to document the training experience of individual users and to provide information to evaluate and optimize the training programme.

Competing interests

None declared.