Meta-analysis of the Diagnostic Test Accuracy of Pediatric Inpatient Fall Risk Assessment Scales.

PURPOSE: This study was conducted to obtain data for the development of an effective fall risk assessment tool for pediatric inpatients through a systematic review and meta-analysis of the diagnostic test accuracy of existing scales.
METHODS: A literature search using Medline, Science Direct, CINAHL, EMBASE, and the Cochrane Library was performed between March 1 and 31, 2018. Of 890 identified papers, 10 were selected for review. Nine were used in the meta-analysis. Stata version 14.0 was used to create forest plots of sensitivity and specificity. A summary receiver operating characteristic curve was used to compare all diagnostic test accuracies.
RESULTS: Four studies used the Humpty Dumpty Falls Scale. The most common items included the patient's diagnoses, use of sedative medications, and mobility. The pooled sensitivity and specificity of the nine studies were .79 and .36, respectively.
CONCLUSION: Considering the low specificity of the pediatric fall risk assessment scales currently available, there is a need to subdivide scoring categories and to minimize items that are evaluated using nurses' subjective judgment alone. Fall risk assessment scales should be incorporated into the electronic medical record system and an automated scoring system should be developed.

INTRODUCTION

Patient falls in hospitals are considered to be a major problem associated with morbidity, mortality, and prolonged hospitalizations in the adult and elderly inpatient population [1]. Falls in hospitalized children are almost equally prevalent and serious. However, little focus has been placed on pediatric inpatient falls [2].

A survey of pediatric inpatient falls reported that, within a 6-month period, fall rates ranged from .4 to 3.8 cases per 1,000 patients at 26 participating institutions in the US, with 68% of these cases occurring when one or more medical staff or family members were present [3]. A survey of pediatric inpatient falls reported that 48 patients experienced falls at one institution in Seoul during a 44-month period, with 82.2% of these cases occurring when family members were present [4]. The findings of that survey also indicated that the majority of medical institutions had a fall prevention program in place. Nevertheless, fall rates had not decreased, and the ineffectiveness of these fall preventive programs has led researchers to question the sensitivity and specificity of fall risk assessment scales. In addition, the problem of inconsistency in reporting and publicly available data on falls has been pointed out by several researchers [5]. Sensitivity is defined as the ratio of people with a disease to people with a positive test result. It is used to evaluate diagnostic tests, symptoms, or signs for the diagnosis of a disease. If a certain symptom, sign, or test has a high sensitivity, certain diseases can be ruled out when the result is negative. Specificity is defined as the ratio of people without a disease to people with a negative test result. Like sensitivity, it is used to evaluate diagnostic tests, symptoms, or signs for the diagnosis of a disease. If a certain symptom, sign, or test has a high specificity, the probability of having a certain disease is high when the result is positive. The diagnostic odds ratio (DOR) reflects the values of sensitivity and specificity for a diagnostic test. It is defined as the ratio of the odds of the test being positive in subjects with the disease to the odds of the test being positive in subjects without the disease [6].

In the USA, 89.7% of reviewed institutions used a pediatric fall risk assessment tool to identify patients at risk of falling. It was reported by 81% of these institutions that the tool used was developed internally, with only 22% indicating that a validated tool was used. In summary, fall risk assessment tools and prevention protocols in major pediatric hospitals have not been standardized [7].

Fall risk screening tools are a very effective intervention strategy for maintaining and improving fall prevention programs. These tools for pediatric inpatients must be evaluated in depth to prevent falls in this population, and the developed tools must be validated for their sensitivity and specificity. The aim of this study was to perform a systematic review and meta-analysis of the published literature on pediatric inpatient fall risk assessment diagnostic test accuracy (DTA) in order to provide useful data for the development of an effective fall risk assessment tool and fall prevention programs for the pediatric inpatient population.

METHODS

1. Study Design

This study was a systematic review and meta-analysis of studies on the DTA of pediatric inpatient fall risk assessment tools.

2. Procedure

1) Literature search

A literature search using PubMed (including Medline), Science Direct, CINAHL, EMBASE, and the Cochrane Library was performed between March 1 and 31, 2018, for articles published.

2) Search terms

Combinations of the following words were used:(child or pediatric or paediatric or children) and (inpatient or hospitalized) and (fall or fall down or accident fall or fall risk) and (scale or tool or assessment) and (validity or sensitivity or specificity). The time limit was set from the earliest year of publication available in each database to the date of the literature search in 2018. The search terms were selected in a way that reflected the query tree structure of each database.

3) Literature selection procedure

(1) Selection and exclusion criteria

All primary studies that evaluated the accuracy of pediatric inpatient fall risk assessment tools, including sensitivity, specificity, odds ratios, true positive values, and true negative values, obtained from the literature search using the search terms were included.

Studies were excluded if they met the following exclusion criteria:

• Duplicates of a study extracted from another database

• Studies that did not assess the effectiveness of fall risk assessment tools

• Studies on pediatric falls in a home or community setting and not an in-hospital setting.

(2) Final selection of studies

According to the above selection and exclusion criteria, two expert-level researchers selected the final sample of studies for analysis by cross-checking. Any disagreements were resolved by consulting a third reviewer until a consensus was reached. All procedures were performed according to the Cochrane guideline [8]. A more detailed description of the procedures is shown in Figure 1.

Figure 1.

Flow diagram for study selection.

(3) Assessment of study quality

Quality assessment of the selected articles was conducted using questions from the Quality Assessment of Diagnostic Accuracy Studies [6]. The risk of bias was assessed for domain-patient selection, the index test, the reference standard, and flow and timing. Three studies had a high risk of bias of patient selection and all 10 studies were evaluated as having a low risk of bias in the other domains.

3. Ethical Considerations

This study was carried out with the approval of the Institutional Review Board (1040782-180228-HR-03-22) for data collection.

4. Statistical Analysis

Data were analyzed using Stata version 14.0 (StataCorp, College Station, TX, USA). First, the metandi command was used to combine the true positive, false positive, true negative, and false negative values of each study, and forest plots and 95% confidence intervals were generated for the sensitivity and specificity of individual studies. Next, summary receiver operating characteristic (SROC) curves of DTA were formulated to increase the statistical power. The SROC model of meta-analysis provides a summary estimate for sensitivity and specificity together. Lastly, bivariate random-effect models from the midas command were used to assess heterogeneity across the studies. The bivariate box plot describes the degree of interdependence, including the central location and identification of any outliers [9].

RESULTS

1. Characteristics of Included Studies

The included studies were published between 2006 and 2017 (Table 1). Of the 10 studies reviewed, four used the Humpty Dumpty Fall Scale (HDFS). The sample sizes ranged from 100 to 652 participants, and the studies employed retrospective research methods. The number of items used in each scale ranged from 4 to 10.

Table 1.

Characteristics of Included Studies (N=10)

Author (year)	Subjects	Tool (range)	Tool items	Cut-off	tp	fp	fn	tn	Sensitivity	Specificity
Razmus, Wilson, Smith, & Newman (2006)	Fall: 100	Morse (0~125)	History of falling, secondary diagnosis, ambulatory aid, IV/IV access, gait, mental status (6 items)	45	62	54	38	46	62.0	46.0
	Control: 100
	Retrospective study
Hill-Rodriguez et al. (2007)	Fall: 153	HDFS (7~23)	Age, sex, diagnosis, cognitive impairments, environmental factor, response to operation, sedation education (usage) (7 items)	12	128	115	22	37	85.0	24.0
	Control: 153
	Retrospective study
Graf (2008)	Fall: 100	GRAF-PIF (0~10)	Length of stay, orthopedic diagnosis, physical therapy, seizure medications, IV (5 items)	2	76	24	25	76	75.0	76.0
	Control: 100
	Retrospective study
Neiman et al. (2011)	Fall: 59	I'm SAFE (0~8)	Impairment, medications, sedation, admitting diagnosis, anesthesia within the previous 24 hours, fall history, environment of care (7 items)	NA	Odds ratio=1.21 (1.14~1.28)
	Control: 177
	Retrospective study
Zhu et al. (2012)	Fall: 50	HDFS		12	73	72	3	9	96.3	11.1
	Control: 50
	Retrospective study
Ryan-Weinger & Dufek (2012)	Fall: 326	PFRA (0~30)	Weakness, physical disability, selected medications, risk for seizure, other conditions, age, confusion, previous fall, hazardous medication. equipment, voiding pattern (10 items)	5	81	98	82	65	49.7	39.9
	Control: 326 (1 month to 21 years)
	Retrospective study
Shin, Kim, Kim, Son, & Bang (2014)	Fall: 48	Pediatric fall risk assessment tool	Age, diagnosis, activity capacity, delay of physical development, number of hazardous medications (above 3) (5 items)	7	44	119	3	23	93.6	16.2
	Control: 149 (0~18 years)
	Retrospective study
Messmer, Williams, & Williams (2013)	Fall: 74	HDFS		12	42	148	17	94	57.0	39.0
	Control 242
	Retrospective study
Pauley et al. (2014)	Fall: 37	HDFS (7~23)		12	36	71	1	3	97.3	4.05
	Control: 74
	Retrospective study
Franck et al. (2017)	Low-risk: 105	Little Shumidy (0~7)	Age, mobility, elimination, current medications (4 items)	3	19	2	44	61	30.0	97.0
	High-risk: 21 (＜25 years)
	Retrospective study

tp=True positive; fp=False positive; fn=False negative; tn=True negative; IV=Intravenous; HDFS=Humpty dumpty fall scale; GRAF-PIF=General risk assessment for pediatric inpatient falls; PFRA=Pediatric fall risk assessment.

2. Items in Fall Risk Assessment Scales

The results of the analysis of the items in the fall scales used in the studies are shown in Table 2. There was a total of 7 scales included mediation items. The detailed contents of each medication item were not consistent across the scales, and various criteria were used for identifying hazardous medications. Four of the seven scales included a diagnosis item. A score was assigned when a certain diagnosis was given. Age, ambulatory aids, and mental status items were included in four of the seven scales. The other items included are shown in Table 2.

Table 2.

Items Included in Pediatric Inpatient Fall Scales (N=7)

Item	Detailed contents	n
Medication	Sedative medications, seizure medications, hazardous medications, current medications, number of hazardous medications	7
Diagnosis	Secondary diagnosis, diagnosis, orthopedic diagnosis, admission diagnosis	4
Age		4
Ambulatory aids	Impairment, weakness, physical disability, ambulatory aids	4
Mental status	Mental status, cognitive impairment, confusion, seizure	4
Activity capacity	Gait, activity capacity, mobility	3
History of falling		3
Others	Voiding pattern, elimination	2
	Environmental factors	2
	Response to operation	2
	Length of stay	1
	Other condition	1
	Delay of physical development	1
	Sex	1

3. Pooled Sensitivity and Specificity Forest Plot

Of the 10 studies reviewed, nine were included in the meta-analysis. A study by Neiman et al. [2] was excluded due to insufficient data. The meta-analysis showed that the nine studies had a pooled sensitivity of .79 (.59~.91) and a pooled specificity of .36 (.16~.63) (Figure 2). The pooled DOR was 2.15 (1.19~3.87) and the area under the curve (AUC) was .65 (.60~.69). The SROC curves of the pediatric inpatient fall assessment scales are shown in Figure 3, with a sensitivity close to 1 and specificity close to 0. The pooled figure also shows scattered points from the center line (Figure 3-A).

Figure 2.

Forest plot of the included studies.

Figure 3-A.

Hierarchical summary receiver operating characteristic (HSROC) curves for pediatric inpatient fall scales.

Figure 3-B.

Bivariate box plot comparing each pediatric inpatient fall risk scale.

4. Heterogeneity Assessment

An assessment of between-study heterogeneity showed negative bivariate correlation (logits) of -.89 and a hierarchical summary receiver operating characteristic (HSROC) beta of .14 (p=.541), indicating that the studies were considered substantial heterogeneity (Figure 3-A). In addition, there were only two outliers in the bivariate box plot comparing each scale (Figure 3-B).

DISCUSSION

We performed a systematic review and meta-analysis of pediatric inpatient fall risk assessments. Of the 890 studies identified, 10 studies were selected for review. Of those 10 studies, nine were included in the meta-analysis. Four of the 10 studies used the HDFS, while a different scale was used in the other 6 studies. In these studies, the items used in the scales varied in number and content. The most common items were the patient's diagnosis, use of sedative medications, and mobility. Age was included in 3 studies, and sex was only included in the HDFS. It has been reported that risk factors for pediatric inpatient falls include being between 3 and 5 years of age, male sex, cognitive development, and clinical conditions, such as surgery and use of sedatives [10]. In addition, some studies have suggested the need to incorporate more risk factors, such as the use of assistive devices, polypharmacy, history of falls, neurological diagnosis, and toileting, that are included in adult fall risk assessment scales [11]. Of the scales analyzed, the General Risk Assessment for Pediatric Inpatient Falls (GRAF-PIF) is the most commonly used tool, and is considered useful for pediatric patients. However, it requires computerization and the problem of length of stay has not been resolved. Use of the CHAMPS pediatric fall risk assessment tool is recommended in clinical practice, but its sensitivity and specificity are difficult to measure, and its accuracy has been questioned [12]. The HDFS is also widely used in clinical practice, and four of the 10 included studies used this scale. However, it has been reported that 80% of the children assessed using the HDFS were categorized as having a high risk of falling, indicating a very low specificity [13]. A similar result was found in our meta-analysis, as the pooled sensitivity of the inpatient fall scales was high (.79), while the pooled specificity was low (.36). Fall prevention programs for pediatric inpatients must effectively screen for children at a high risk of falling. However, the results from this study indicate that the pediatric inpatient fall risk assessment scales currently used in hospitals have failed to do so. As such, fall prevention programs for high-risk children have not been effective [14], and the fall risk scales currently available are not adequate for screening children at a high risk of falling.

The studies with highest sensitivity were by Pauley et al. [15] and Zhu, Xu, and Yang [16], with sensitivity values greater than .95. The HDFS was used in both these studies. The HDFS contains seven items, all of which may be checked from the patient's chart. This scale considers the age and sex of the child, as these two factors are most closely associated with an increased risk of falls. However, the scores given for each age category (4 points for children aged 3 years or younger and 3 points for children aged 3~7 years) do not seem adequate, as falls are reported most frequently in children aged 3~5 years. Furthermore, a history of falls and using beds are given the highest score of 4, while using a crib is given a score of 3 and is classified as an item relating to an environmental factor. As such, some items appear to be grouped without a rational basis. Such items may potentially give rise to confusion among nurses when evaluating a child. There is also a concern regarding giving high scores on ambiguous items. The sensitivity of the studies that used the HDFS was generally high (.11). In contrast, the specificity was very low (.04). This may be partly explained by the tendency of nurses to give high scores for items that are ambiguous and difficult to measure. This is because the likelihood of receiving a neglect of duty evaluation is higher for nurses when falls occur in children classified as low-risk than when no falls occur in children classified as high-risk. Moreover, a study by Myers and Nikoletti [17] showed that the sensitivity of nurses' clinical judgment was high, while the specificity and positive predictive value were low. This suggests that when using clinical judgment, nurses were not able to distinguish between high-risk and low-risk children. The scale used in the study by Franck et al. [18] had low sensitivity and contained four items, with a cutoff score of 3. This scale, however, had a very high specificity of .97, and 97% of children who were identified as having a high risk of falling using this scale experienced a fall. The 4 items included in this scale are age, mobility, elimination, and current medication. The simple set of items in this scale and its effectiveness in screening for children at high risk must be considered.

Considering the low sensitivity of the pediatric fall risk assessment scales currently available, there is a need to subdivide scoring categories and to minimize the items evaluated solely by a nurse's individual judgment. It is necessary to incorporate fall risk assessment scales into the electronic medical record system and to develop an automated scoring system.

Our review of pediatric inpatient fall risk assessment scales showed that research in this area is still at an early stage. As pointed out in a number of studies, the accuracy of these scales is poor, requiring more evaluation and validation studies [5,7,12-14,19,20]. Effort should be made to revise and improve the sensitivity and specificity of existing fall scales and to develop a new scale with better predictability through continued research. Moreover, it may be necessary to consider all cases of falls in order to increase predictability, since falls that do not lead to injury have been excluded from reports [21]. Thus, a more effective fall risk assessment scale must be developed based on the accumulated study results.

CONCLUSION

We aimed to provide useful data through a systematic review and meta-analysis of previous studies on the DTA of pediatric inpatient fall risk assessment scales, as it was evident that there is a need for an effective pediatric inpatient fall risk assessment tool. The findings from the meta-analysis of this study showed that the currently-available inpatient pediatric fall risk assessment scales have high sensitivity and low specificity, suggesting that there is a need to develop a scale with a greater number of subdivided categories and no items that require the subjective judgment of nurses. To this end, continued research is needed on the development of fall risk assessment scales and on the effectiveness of such scales in clinical practice.