Accuracy and Efficiency of Recording Pediatric Early Warning Scores Using an Electronic Physiological Surveillance System Compared With Traditional Paper-Based Documentation.

Pediatric Early Warning Scores are advocated to assist health professionals to identify early signs of serious illness or deterioration in hospitalized children. Scores are derived from the weighting applied to recorded vital signs and clinical observations reflecting deviation from a predetermined "norm." Higher aggregate scores trigger an escalation in care aimed at preventing critical deterioration. Process errors made while recording these data, including plotting or calculation errors, have the potential to impede the reliability of the score. To test this hypothesis, we conducted a controlled study of documentation using five clinical vignettes. We measured the accuracy of vital sign recording, score calculation, and time taken to complete documentation using a handheld electronic physiological surveillance system, VitalPAC Pediatric, compared with traditional paper-based charts. We explored the user acceptability of both methods using a Web-based survey. Twenty-three staff participated in the controlled study. The electronic physiological surveillance system improved the accuracy of vital sign recording, 98.5% versus 85.6%, P < .02, Pediatric Early Warning Score calculation, 94.6% versus 55.7%, P < .02, and saved time, 68 versus 98 seconds, compared with paper-based documentation, P < .002. Twenty-nine staff completed the Web-based survey. They perceived that the electronic physiological surveillance system offered safety benefits by reducing human error while providing instant visibility of recorded data to the entire clinical team.

Recording of vital signs and clinical assessments are core activities undertaken by health professionals to provide oversight of patient well-being. The Pediatric Early Warning Score (PEWS) has been advocated[1,2] as an adjunct to assist staff in the early identification of serious illness or deterioration in hospitalized children. Scores are derived reflecting the degree of abnormality in recorded vital signs, where higher scores signify increased risk of deterioration. The Pediatric Early Warning Score aims to reduce the incidence of critical deterioration events (CDEs) by flagging patients “at risk” so that care can be escalated. Critical deterioration events include respiratory or cardiac arrest,[3] urgent calls to the resuscitation team,[4,5] and emergency transfer from inpatient wards to the high-dependency unit[6] or the pediatric ICU,[6-8] or unexpected death.[9]

The history of PEWS development is short[10] but complex. Age-specific PEWS risk models are required across the age range because of the physiological variation that exists in respiratory rate, heart rate,[11-13] and blood pressure[14] from birth to adulthood. Many “unknowns” remain. There is no widespread agreement from clinicians or researchers regarding the following:

the essential components that should be monitored within a PEWS model or the frequency with which they should be recorded,

the optimum thresholds for clinical concern,

the weighting applied to individual score components to reflect the deviation from the predetermined “norm,” and

the escalation required at a given PEWS score, so that a CDE could be averted.

Despite these limitations, PEWS is increasingly being used internationally,[6,9,15-19] most commonly incorporated within paper-based charts. In the United Kingdom, PEWS is in widespread use (90% in tertiary hospitals, 78% in district general hospitals),[20] following endorsement in national guidance[1,2] and by the National Health Service Litigation Authority.[21]

Variations in monitoring and recording vital signs in hospitalized children are well described.[8,22,23] Incomplete, inaccurate, or illegible documentation; imprecise plotting; or miscalculation of PEWS has the potential to impede the delivery of safe clinical care.[24] The impact that errors in paper-based documentation of vital signs have on the accuracy of PEWS have been alluded to.[23,25] However, there is a paucity of evidence on fidelity assessment of implemented PEWS,[26] which is important in determining their reliability. Failure to measure process errors could mean a PEWS model is disregarded as poorly performing, when poor compliance and human factor errors are confounding that analysis.[27-29]

There is progression toward electronic documentation of vital signs and clinical assessments worldwide. This has the potential for improved accuracy in documentation of a patient’s physiological state[30] with obvious benefits to clinical care. Collection of large data sets of physiological signs and analysis of the recorded data[13,31,32] could also develop the PEWS evidence base.

In hospitalized adults, the evidence base for early warning scores (EWSs) has also been difficult to ascertain.[33] This changed with the utilization of an electronic physiological surveillance system[34] (EPSS). An EPSS is described as follows[35]:

especially designed software that prompts nurses to record a complete set of vital signs at the patient’s bedside at appropriate intervals on handheld computing devices.

The early warning (EWS) risk model was embedded within the software, with automated calculation of the EWSs when vital signs were entered. Instant bedside decision support based on the score was provided, including recommendations for reassessing the patient and escalation to senior staff. Documented vital signs and EWSs were instantly visible to the entire clinical team. Built-in process control provided reports on the chronology of CDE with compliance reporting of the agreed safety process. Significant reductions in mortality were achieved in two large UK hospitals using this technology, with a correlation with improved clinical monitoring.[35] In addition to the patient safety improvements, this technology enabled large-scale modeling of data to occur[34,35] providing evidence that contributed to development of a National Early Warning score[36,37] for screening the risk of deterioration in hospitalized adults. There are no published studies of comparable systems robustly tested for use in the pediatric hospital population.

Funding from the Small Research Business Initiative supported this 6-month study to develop and test proof of concept for an EPSS prototype suitable for pediatric patients. Small Research Business Initiative grants are awarded for feasibility or pilot studies to determine proof of concept for technical innovations prior to larger-scale research.

METHODS

In August 2014, a mixed-methods prospective study explored how an EPSS (VitalPAC Pediatric, SystemC Healthcare Ltd, Maidstone, Kent, England) compared with traditional paper-based documentation of vital signs, clinical observations, and calculating PEWS. A Web-based survey was used to assess the user acceptability of using both methods. Local research ethics committee approval was obtained. Participants volunteered to be involved, with no financial incentives offered to participants in either the controlled documentation exercise or the Web-based survey. The participants were a purposive sample of RNs, student nurses, healthcare assistants, and medical students working on the ward where pilot testing of the EPSS prototype was being conducted.

Controlled Documentation Exercise

This took place in a nonclinical environment, at a tertiary pediatric hospital in Northwest England. Five vignettes of vital signs and clinical observations were used, reflecting a range of pediatric severity of illness (Table 1).

Table 1

Vignette Overview

The key foci were the following:

accuracy of data recording;

accuracy of calculation of age-specific PEWS; and

time taken to document vital signs, clinical observations, and PEWS (efficiency).

The method for recording data was randomized so that alternate participants commenced with either paper-based or EPSS documentation. Participants completed all five vignettes using the same documentation method before progressing to record the same vignettes using the alternative method. The exercise was undertaken one person at a time, to limit distractions posed by others. The time taken to record documentations was measured using a stopwatch on an iPod Touch (Apple, Cupertino, CA) (G.S.). Timing commenced when the participant indicated he/she was ready and concluded when he/she indicated he/she had completed the exercise.

Paper-Based Documentation

The existing paper-based ward observation chart was used, which incorporated age-specific PEWS thresholds for respiratory rate, heart rate, and systolic blood pressure (Figure 1). There were five charts covering ages 0 to 1, 1 to 2, 2 to 7, 7 to 13, and older than 13 years (note: the child moved to the next chart on the day following his/her birthday). This age grouping was derived from a meta-analysis of respiratory and heart rates,[11] with ages with similar physiological values grouped together. The various charts were differentiated by color theme, age-group labeling, and an age-specific image in the top right hand corner.

FIGURE 1

Alder Hey age-specific PEWS paper-based observation chart.

The participants were required to select the correct chart based on the date of birth for each vignette. The sequence for documenting data followed the Advanced Pediatric Life Support mantra of airway, breathing, circulation, disability (ABCD)[38] to focus thinking on the possibility of serious illness or deterioration. The charts required the user to record absolute values, as well as plotting of graphical trends. The graphs for respiratory rate and heart rate utilized color banding to visually differentiate when vital signs were outside the accepted “norm.”[11] Data plotted in the white band scored 0, yellow band (90th and 10th centile) scored 1, and orange band (99th and 1st centile) scored 2. The PEWS was manually calculated from the aggregate sum of scores from nine components: respiratory rate, effort of breathing, oxygen saturation (SpO2), oxygen requirement, heart rate, systolic blood pressure, capillary refill time, rapid neurological assessment alert, voice, pain, unresponsive (AVPU), and nurse concern. Two “red flag” vital signs associated with increased risk of a CDE generated a score of 10 each, reflecting their significance and prompting an immediate medical review. These were as follows:

a systolic blood pressure at or below the fifth centile for age and

being unresponsive on AVPU assessment.[39]

All participants had training and experience of using the paper-based observation charts for a minimum of 3 weeks.

Electronic Physiological Surveillance System Documentation

The EPSS was configured to run on Apple hardware. For this pilot study, the EPSS was tested using iPod Touch fourth generation loaded with the VitalPAC Pediatric prototype, underpinned by the hospital’s age-specific PEWS risk model (Figure 2). All participants had training and experience of using the EPSS for a minimum of 3 weeks. The software interfaced wirelessly with the hospital patient information system (Meditech 5; Meditech, Westwood, MA), so that all patients admitted to that ward area are automatically loaded to the device.

FIGURE 2

VitalPAC Pediatric screen shots showing recorded vital signs and the calculated age-specific PEWS, with breakdown of the contributing data.

For the controlled documentation exercise, a virtual ward was created containing 10 fictitious patients, five of whom matched the vignettes to be recorded. The participant was required to confirm the “patient” identity using the barcode on the hospital identity band. The correct age-specific “chart” automatically loaded based on the registered date of birth. The VitalPAC Pediatric sequence for recording data followed the APLS process ABCD.[38] Absolute values for vital signs were manually entered and automatically plotted to display graphical trends. The EPSS had built-in features of disallowed values to reduce the erroneous input of clinically inappropriate values, for example, greater than 100% for SpO2. If an extreme value was entered, a clinical prompt clarified if that value was correct. If the user re-entered the same value, it was accepted. Automated calculation of the PEWS occurred with escalation advice based on the score.

Survey of Acceptability

The acceptability of both documentation methods was assessed separately from the controlled documentation exercise, using an anonymized Web-based survey accessible on a tablet in the ward clinical area. The survey consisted of 10 questions focusing on individual perceptions, opinions, and experience of using each documentation method. The format included statements to gauge opinion using Likert scale and free text responses to open questions. The survey took less than 3 minutes to complete.

ANALYSIS

Statistical analysis was performed using IBM SPSS Statistics for Windows, version 22.0 (IBM, Armonk, NY) (released 2013). The accuracies of documentation of vital signs, error rates, and PEWS were expressed as percentages. Because the same participant undertook both paper and electronic assessments for each vignette, the participant acted as his/her own control. The data were measured on a continuous scale, which cannot be assumed to be normally distributed. The assessments were matched pairs; therefore, the Wilcoxon signed rank test was used to test the null hypothesis that the population median of differences between the matched pairs was zero. P < .05 was considered significant.

RESULTS

Twenty-three clinical staff from wards testing the EPSS volunteered to participate, and informed consent was gained. Participants included 15 nurses, one healthcare assistant, and seven medical students. Complete data were recorded for 115 observation sets on paper and 111 observation sets using the EPSS (Table 2). One participant undertaking EPSS documentation had to leave unexpectedly. Only completed observation sets were included in the analysis.

Table 2

Documentation Accuracy and Time Taken to Document Vital Signs/Clinical Observations

Accuracy of Vital Sign Documentation

An overall accuracy of documentation score was applied to each vignette based on the proportion of correctly recorded data points. The accuracy score was superior with EPSS documentation compared with traditional paper-based recording, 98.5% versus 85.6%, P < .02.

Paper-based documentation required recording of absolute values and graphical plotting of trend, which provided 21 to 25 potential opportunities for error, depending on the complexity of the vignette recorded. Eight cases (7%) were erroneously documented on incorrectly selected age-specific charts. The remaining errors reflected 3.3% documenting and 4.2% plotting mistakes. In comparison, the EPSS incorporated automated plotting of recorded values so that the graphical trend was displayed. This reduced the potential opportunities for error in documentation from 14 to 16, dependent on the vignette complexity.

Accuracy of Pediatric Early Warning Score Calculation

The accuracy of PEWS calculation was superior using EPSS documentation compared with traditional paper-based recording, 94.6% versus 55.7%, P < .001 (Table 2).

The eight cases (7%) recorded on incorrect age-specific paper-based charts contributed to inaccurate PEWS because the underpinning age-specific PEWS risk model was inappropriate for that “patient.” Five other cases recorded on paper-based charts did not have the PEWS calculated, despite having the required information to do so.

During EPSS documentation, the automated PEWS calculation displayed six errors due to erroneous data input. Two cases involved documentation of “effort of breathing” and AVPU, which led to PEWS underscoring. Four cases had incorrectly entered “oxygen requirement” and “nurse concern,” which led to PEWS overscoring, but did not alter the escalation advice.

Efficiency: Time Taken to Record

Time taken to record vital signs and clinical observation and calculate PEWS was faster using the EPSS compared with traditional paper-based documentation: 68 versus 98 seconds, P = .001 (Table 2). The EPSS utilized automated plotting of graphical trends based on entered values and automatically calculated PEWS, which saved time for the user.

The Web-Based Electronic Survey of User Acceptability

The Web-based electronic survey of user acceptability was completed by 29 staff: 15 nurses, three healthcare assistants, and 11 medical students (Table 3). The survey was anonymized, so it was not possible to identify which respondents also participated in the controlled exercise of documentation of vital signs. Ninety-three percent of respondents had prior experience with Apple hardware. Fifty-five percent of respondents preferred data entry using the EPSS, but 25% of respondents remained undecided.

Table 3

Web Based Survey

DISCUSSION

A primary duty for health professionals is to uphold safety for the patients in their care using appropriate monitoring[40] to identify those in the early stage of serious illness or deterioration.[41] If this process is poorly applied, all subsequent opportunities to intervene are delayed and may have an impact on patient outcome.[42]

Pediatric patients rely on healthcare professionals or other adults to notice the subtle signs of becoming more unwell. The Pediatric Early Warning Score, when correctly applied, provides an adjunct to assist professionals of varying expertise to identify the patients at risk of deterioration. Human error is implicated as a prominent cause of avoidable adverse events[43] and can contribute to suboptimal PEWS assessment. Understanding the component parts contributing to failure within complex systems is essential so that safe processes can be designed to reduce preventable errors.

Errors in Documentation and Pediatric Early Warning Score Calculation

In this study, the overall error rate for paper-based documentation of vital signs was 14.5%. This included 7% of cases where the incorrect age-specific chart was selected. This was a significant error with potential to miss deterioration due to plotting of data on a chart with the wrong underpinning age-specific PEWS risk model for that “patient.” The remaining 7.5% errors were for inaccurate documentation of values (3.3%) and plotting of trends (4.2%). Documentation errors have been reported at 7.3% in adults.[44] The higher error rate in pediatric patients may reflect the increased complexity of charts[45] including the requirement for age-specific thresholds.[11]

The accuracy of PEWS calculated manually on the paper-based charts was 55.7%. This was mainly due to the errors in documentation and plotting of vital signs. However, in seven cases (30%) where all data points were correctly recorded, the PEWS calculation was incorrect. There were also five cases where the PEWS calculation was completely omitted, despite having the necessary data to undertake this. Twenty-five percent of all the recorded PEWS calculations were underscored. These occurred at low PEWS, and the clinical action prompt would have been unchanged, even if correctly calculated.

The level of inaccuracy was similar to findings published in adults.[46] It has been suggested that mental math skills are inadequate in nurses,[47] increasing the risk of calculation error, but this may also reflect the complexity of chart design[48] and the cognitive load of mentally processing information.[49,50] In adults, the accuracy of EWSs reduced when the clinical environment was busier[49] or when the calculation was more complex (sicker patient).[46]

In comparison, the error rate using the EPSS prototype was low (1.5%). Despite built-in safety features of disallowed values to reduce data input errors, a small number still occurred, for example, fraction of inspired oxygen (FiO2) of 0.35 delivered via head box was entered erroneously as 35 L/min oxygen via nasal specs in two cases, which generated a higher PEWS value. Further minor adjustments could be made to the EPSS prototype to reduce these errors.

The accuracy of PEWS was high (94.6%). The automated calculation of the PEWS using the EPSS had high reliability due to the low incidence of data entry and plotting errors. Calculation error and omission of PEWS recording were avoided by automation and standardization of process.

Efficiency: Time Required to Document

In this study, EPSS documentation of vital signs saved 30 seconds per patient per set of observations. In a 20-bed ward where children have a minimum of four hourly recordings of vital signs, an EPSS could save 60 minutes every 24 hours. That time would become available for other direct clinical care.

User Acceptability

Twenty-nine staff completed the Web-based survey (Table 3). Increased survey completion may reflect the ease at which it could be undertaken without leaving the ward area. Ninety-three percent of the respondents had prior experience with Apple hardware, which may have influenced their preference for the EPSS. High user acceptability of EPSS was demonstrated during early implementation in another study[46] but dropped marginally after 4 weeks. A follow-up survey was not possible in this study because of time restraints.

LIMITATIONS

This was a small sample size in a time-limited study restricted by the grant timeline to develop and test proof of concept of an EPSS prototype prior to undertaking a large-scale study.

This was a controlled study of documentation undertaken in a nonclinical area, so it does not accurately reflect the reality of performance in a clinical setting. Participants were not challenged by interruption to the documentation process or by the competing demands of several patients and families. The controlled exercise occurred in daylight hours, so visibility of the paper-based charts and EPSS was good. Working in dimmed lighting may contribute to further errors in documentation.

CONCLUSION

Smart technology has become integral to daily life for organization of professional and personal communication and commitments. Smart technology including EPSS has not yet been robustly evaluated to determine its effectiveness at improving clinical safety in a pediatric hospital population. In a controlled study of documentation, the VitalPAC Pediatric prototype demonstrated superior accuracy and efficiency compared with traditional paper-based charts for recording vital signs, clinical observations, and calculating PEWS. Process errors, documentation errors, calculation errors, and errors of omission were eradicated. Minor errors in EPSS data input persisted but could be reduced by further amendments of “disallowed values.” Electronic physiological surveillance systems have the potential to standardize the surveillance of sick children in hospital and improve clinical safety. Further large-scale scientific evaluation of EPSS screening of sick children for risk of a CDE is required.