Do pressure ulcer risk assessment scales improve clinical practice?

Standardized assessment instruments are deemed important for estimating pressure ulcer risk. Today, more than 40 so-called pressure ulcer risk assessment scales are available but still there is an ongoing debate about their usefulness. From a measurement point of view pressure ulcer (PU) risk assessment scales have serious limitations. Empirical evidence supporting the validity of PU risk assessment scale scores is weak and obtained scores contain varying amounts of measurement error. The concept of pressure ulcer risk is strongly related to the general health status and severity of illness. A clinical impact due do the application of these scales could also not be demonstrated. It is questionable whether completion of standardized pressure ulcer risk scales in clinical practice is really needed.

Introduction

Pressure ulcers (PUs) are significant health problems typically occurring in the context of severe illness and high care dependency. In their newly developed clinical practice guideline, the National Pressure Ulcer Advisory Panel (NPUAP) of the USA and the European Pressure Ulcer Advisory Panel (EPUAP) define a PU as “… localized injury to the skin and/or underlying tissue usually over a bony prominence, as a result of pressure, or pressure in combination with shear. A number of contributing or confounding factors are also associated with pressure ulcers; the significance as these factors is yet to be elucidated”.1 This definition clearly indicates that the understanding of this complex phenomenon is still limited. There is an ongoing debate about what PUs really are, how they develop, and how these lesions should be classified accurately.2–4 Even the recent international collaboration between NPUAP and EPUAP was unable to find a consensus about PU classification.1

Irrespective of these conceptual difficulties, PUs cause serious functional limitations, emotional burden, pain, and impairments for persons affected.5,6 Furthermore, the development of PUs in healthcare institutions is regarded as an outcome indicator for the quality of care provided.7 PU treatment is expensive and legal issues around PUs have become more and more important. Therefore, effective PU prevention plays an important role in everyday clinical practice. Recommended strategies include frequent repositioning, use of special support surfaces, or providing nutritional support.1

Since the susceptibility to PUs is different from person to person, determination of the individual PU risk is important. So-called PU risk scales aim to support practitioners in determining the individual PU risk. The structure of most of these standardized assessment instruments is comparable: they include a number of factors that are deemed important for PU development. These factors are operationalized in the form of items, which are scored; item scores are then summed, resulting in total scores indicating the individual risk level. Cut-off points are used to distinguish between “at risk” and “not at risk” or among risk levels. Chosen cut-off points or risk categories are used to trigger and plan prevention strategies. The very first PU risk scale was published by Norton et al in 1962.8 Other commonly used PU risk scales are the Waterlow9 and Braden10 scales. Today, there are more than 40 of these instruments available and new scales are being developed.

PU risk assessment scales are widely used and there are a large number of empirical studies about them. However, there is an ongoing debate about the usefulness and benefit of PU risk scales in clinical practice. PU risk scales are criticized for their poor psychometric properties and their inability to improve patient outcomes.11–13 In contrast to these arguments the latest international clinical practice guideline provided by the NPUAP and EPUAP states, “Risk assessment scales are the foundation of risk assessment practice”(p. 24).1 How can this contradiction be explained? Why is the use of PU risk scales debatable? To answer these questions this paper provides an overview and discussion of PU risk scale research and explores other issues of risk scale application.

Quality of pressure ulcer risk scales

PU risk scales aim to measure and quantify pressure ulcer risk. In general, measurement means the assignment of numbers to represent the amount of an object, attribute, or trait using specified rules. Determining the quality of these measurements usually involves evaluation of validity and reliability.

Validity

Validity is the degree to which accumulated evidence and theory support specific interpretations of test scores entailed by proposed uses of a test.14 Authors often distinguish between content, criterion, and construct validity but one has to keep in mind that these concepts are not clearly separable from each other.15 In PU risk scale research all these validation approaches were taken into account, but studies on diagnostic accuracy that are related to criterion validity were most often published.

Content validation

If one wants to measure PU risk, the scale should comprise all factors and items that are relevant (content coverage). Fulfilling this requirement is challenging. First, there are more than a hundred PU risk factors described in the literature. Taking both etiological knowledge2,16 and epidemiological evidence17–19 into account, factors directly causing enhanced exposure to pressure or shearing forces, particular restricted mobility, appear to play the most important role for PU development. On the other hand, in a population where the majority of patients suffer from limited mobility (eg, geriatric or intensive care patients) this single factor may not be discriminative enough to identify patients at increased PU risk. Unfortunately, there is no clear-cut evidence on the role of further intrinsic (eg, nutrition), iatrogenic (eg, specific medications or medical procedures), or behavioral factors (eg, nicotine intake) in specific populations.20–22 Available results are inconsistent and depend on the population under investigation as well on methodological issues of respective studies. Second, the specific importance of single risk factors is not adequately taken into account as long as all items are equally weighted.23 PU risk scales considering weights in their scores are available, but most often these weights are arbitrary, eg, in the Waterlow9 or in the Glamorgan scale.24 Today, there are only a few PU scales containing weights that were developed on a more rational basis, eg, the mul-tifactorial approach taken by Nonnemacher et al.19

Diagnostic accuracy

Results of diagnostic tests or scores of diagnostic tools must be able to identify a condition correctly. Therefore, PU risk scales scores must indicate PU risk when there is really a PU risk (sensitivity), and they must indicate that there is no risk when there is no risk (specificity). Based on obtained sensitivity and specificity, other useful estimates like predictive values and likelihood ratios can be calculated. Both sensitivity and specificity of a test must be high (nearly 100%) to be useful in clinical practice.25

Studies on diagnostic accuracy are typically applied to investigate this issue. In these investigations, the outcomes from one or more tests under evaluation are compared with outcomes from the reference standard, both independently measured in subjects who are thought to have or not have the condition of interest.26 The reference standard is considered to provide the best information whether the condition of interest is present or absent. Unfortunately, to our best knowledge, an unequivocal and approved reference standard for PU risk does not exist. Therefore, in a strict sense, diagnostic accuracy cannot be investigated because the ‘truth’ cannot be observed.27 Nevertheless alternative approaches may be taken. In PU risk scale research it is common to use the actual development of PU’s as reference standard to which obtained risk scores are compared.

In recent systematic reviews,28–31 diagnostic accuracy estimates of more than 30 studies have been identified and synthesized. In a meta-analytic approach, Pancorbo-Hidalgo et al29 presented pooled sensitivity and specificity estimates for the most investigated instruments and concluded that the Braden scale shows optimal validation and has the best balance between sensitivity (57.1%) and specificity (67.5%) as compared to the Norton and Waterlow scales. These rather low values suggest that Braden scale scores poorly predict who will develop a PU and who will not.

On the other hand, due to the influence of PU preventive measures, nearly all obtained sensitivity and specificity estimates in PU risk scale research are biased. Applied preventive interventions decrease the probability of PU development, and sensitivity and specificity as well.32,33 One can also put it the other way round: “High sensitivities and specificities can only be reached if a study has been conducted in a health care institution that does not use effective preventive interventions” (p. 40).32 This may be feasible but is clearly undesirable.

In their systematic review Pancorbo-Hidalgo et al29 conclude further that the Braden scale score is a good “pressure ulcer risk predictor” (OR 4.1, CI 95% 2.6–6.5). This conclusion points to another conceptual problem in PU risk scale research. First, PU risk can be actually there but the risk cannot be predicted. Second, when using PU development as reference standard to investigate predictive validity of PU risk scales, then obtained risk scores are treated like predictors of who will develop a PU and who will not. This is far different from being at PU risk.34–36 One can argue that increased PU risk can be regarded as an increased probability for PU development, but due to the complex nature of PU development and hopefully preventive interventions, there is still large uncertainty.

Construct validation

PU ulcer risk scales aim at measuring the highly complex construct “PU risk”. The term “construct validation” is somewhat of a tautology because validity almost always refers to constructs14 but as compared to criterion or content validation, the evaluation of construct validity refers to testing the underlying theoretical assumptions and it includes a broader set of methodological approaches. Three approaches will be discussed: known groups, convergent validation, and discriminant validation.

Known groups

Construct validation by known groups is simple: risk scale scores are assessed in groups of individuals who are supposed to differ in their PU risk level. Then, it is empirically tested whether they differ indeed. For example it can be assumed that due to higher PU prevalence and incidence rates on intensive care units as compared to other specialties37 ICU patients are also at much greater PU risk. This assumption was confirmed in numerous studies38,39 supporting the construct validity of PU risk scales. However, the known group design is a necessary step in instrument validation but it is not sufficient.15

Convergent validation

Investigation of convergent validity may include testing how closely PU risk scale scores are related to other measures of the same construct, eg, other PU risk scales or other ways of risk estimation. This approach was taken, for example, by Gould et al.40,41 Based on simulated case examples, the authors investigated whether scores of the three common Norton, Waterlow, and Braden risk scales generated by clinical nurses were congruent with the nurses’ own clinical judgment assessed on a visual analogue scale (VAS) and with the degree of PU risk independently agreed by the consensus view of an expert panel. One result was that the nurses’ own clinical judgment showed a greater level of agreement with expert panel view than assessment with any of the three risk scales. The authors concluded that the nurses’ own clinical judgment was more likely to give a valid PU risk estimate than any of the tested scales.

A comparable study in a clinical practice setting was conducted by Kottner and Dassen.42 Two samples of ICU nurses rated PU risk according to two scales and according to their own clinical judgment on a VAS. Coefficients of determination between PU risk scale and VAS scores varied between 0.3 and 0.6 indicating that 40% up to 70% of variances could not be explained by score differences. In other words, the same construct of PU risk was only partly covered by the three measures.

Discriminant validation

Since all PU risk scales are expected to measure the same construct, obtained scores can be expected to correlate with each other. On the other hand, they should not be related to scores of dissimilar or unrelated constructs. Since PU risk and PU development are complex phenomena it seems nearly impossible to decide what health problem or condition is not related to PU risk. In fact, empirical evidence suggests that PU risk is indistinguishable from general health status. For example, two studies showed that a scale measuring overall care dependency performs equally in identifying patients at PU risk as compared to PU risk assessment scales.43,44 Similar results were found when the diagnostic performance of the Minimum Data Set of the Resident Assessment Instrument was compared to that of the Braden scale in long-term care.45 There are strong associations between Braden scale and Glasgow Coma Scale scores,46 and Palliative Performance Scale scores.47 Capon et al identified statistically significant relationships between factors such as previous stroke, previous trauma, and cognitive decline and PU risk, according to the Braden scale.48

Reliability and agreement

The terms reliability and agreement are often used interchangeably but they are conceptually distinct. Reliability can be defined as the ability of scores of a test or scale to differentiate among subjects or objects, and agreement is the degree to which scores or ratings are identical.49 Both concepts are important since they provide information about the amount of error inherent in scores and measurements and therefore they determine the upper limit of the validity of any measurement.50,51

In PU risk scale research, various ways of determining reliability and agreement were used and therefore the results are hardly comparable. Two reviews found that there is little empirical evidence regarding inter-rater reliability and agreement of the Braden and Waterlow scales when used in clinical practice.52,53 Systematic synthesis of reliability and agreement estimates of other PU risk scales is lacking.

Recent study results suggest that there are considerable differences regarding the amount of measurement error of individual items, indicating that some items are much harder to rate than others.42,54 This may be explained by ambiguous wording of some items, the qualification and training of raters, or by properties of the rated subjects.23,55 Additionally it could be shown that despite high inter-rater reliabilities for total scores, the probability of exact agreement among users of PU risk scales in clinical practice is low. For instance, when applying the Braden scale total score in nursing home and home care settings, differences of up to 3 or more points can be expected.55,56 Finally, scale scores that proved to be somewhat reproducible in one care setting may perform poor in another setting.42 This indicates that reliability and agreement coefficients like sensitivity and specificity measures are population specific.

Clinical impact

Investigations of validity and reliability are important in evaluating the quality of PU risk scales but they are insufficient to judge their clinical value. Results of diagnostic tools should be used to guide interventions and to improve patient outcomes important to patients.57 Randomized controlled trials in which investigators randomize patients to groups using different diagnostic approaches are the best way to assess the clinical impact of diagnostic strategies58 but quasi experimental or observational studies may also provide some information.

Empirical evidence

In PU research only few attempts have been made to compare different PU risk assessment strategies. Looking at the literature until 2003, Pancorbo-Hidalgo et al in their systematic review29 identified three studies investigating the impact of using the Norton scale compared to clinical judgment on PU incidence. They concluded that there was no evidence that the use of risk assessment scales decreased pressure ulcer incidence. The objective of a recent Cochrane review was to determine whether using PU risk assessment, in any health care setting, reduces the incidence of pressure ulcers. Since no studies met the inclusion criteria the authors were unable to answer their review question.59

In a controlled trial, Vanderwee et al60 compared two diagnostic strategies for PU prevention: one group of patients received preventive measures (support surface, turning) when Braden scale scores were less than 17, and in the other group prevention was started when nonblanchable erythema (Category 1 PU) appeared. In the end there was no statistically significant difference regarding PU incidence in the two groups. Furthermore, since more preventive interventions were conducted in the “risk scale group”, the authors concluded that using nonblanchable erythema instead of the Braden scale score would lead to a considerable reduction of preventive measures without resulting in an increase in pressure ulcers.

In a smaller quasi-experimental study, Saleh et al61 compared three patient groups of one hospital: the Braden scale was used in one group, including an intensive education and training program. Another group received the same program without implementing the Braden scale; and a third group performed as usual including standard ongoing education. PU frequency reduced in all three groups with no significant differences. The authors concluded that clinical judgment may be as effective as using a PU risk scale in determining appropriate care.

Finally, based on a comparison between 16 nursing homes with very high and very low PU prevalence rates, Bates-Jensen et al62 were unable to detect any differences in the use of PU risk scales. In other words, PU care was not better when PU risk scales were used.

Opinions

Irrespective of study results, the main goal of PU risk scales is to predict who will develop PUs for the purpose of planning effective prevention strategies.34,63 There is the idea that the main benefit of PU risk scales lies in its acting as a reminder to nurses about possible PU development. PU risk scales aim at increasing the awareness of factors contributing to PU risk for practitioners.9,33 Recently, it was argued that scores of single PU risk items provide information for planning preventive measures, and are more able to guide clinical practice than broad risk categories or sum scores.30,63,64 Scales should be used as a kind of checklist leading to specific preventive interventions. For example, when there are restrictions regarding mobility or activity, pressure reducing surfaces or repositioning schedules should be applied. Likewise, special skin care and cleansing are required when the risk assessment reveals a problem with skin moisture.65,66

Although each of these arguments for use of PU risk scales sounds reasonable it has to be noted that they are based on opinion rather than scientific evidence. Well-conducted studies which investigate the impact of the use of any PU risk assessment instrument on nurses’ clinical decision making are still lacking. In a small mixed-methods study67 conducted in a long-term care setting, both the assessment of patients’ PU risk, and the allocation of preventive interventions, appeared to be marginally guided by the information provided by use of the Waterlow scale. Instead, it was observed that nurses changed the total scores so that they matched the patients’ PU risk as perceived by them. Similarly, alterations of the Waterlow sum scores were often not followed by an adaption of the care plan. Thus, based on these results the author questioned “... whether assessment tools such as the Waterlow scale add anything more to the knowledge that a nurse has of an older person ...”(p. 38).67

Other considerations

So far two important properties of PU risk scales were addressed. However, there are some other areas in health care where PU risk scales may play an important role.

Legal issues

Litigation against healthcare providers is increasing and PU development is an increasing reason for lawsuits.68–70 From the legal perspective comprehensive documentation is of utmost importance because in court it is argued “what was not documented was not done”.71 This is also true for PU prevention. Continuing documentation of PU risk scale scores is used as evidence that a PU risk assessment, as the first necessary step in the prevention process, was performed.72

Research

PU risk scale scores are used as case-mix adjuster when making comparisons of PU incidence or prevalence measures between units, institutions, or even countries. Stratification according to a specified cut-off point enables adequate comparisons of different samples.39,73–75 Total or item scores are used in multivariate models to adjust for PU risk,76,77 and in clinical trials comparing different preventive interventions item and total scores are used as eligibility criteria.78,79

Quality assurance

Institutions and professionals are required to provide high quality care. Among others, the outcome of having developed PU is regarded as one indicator of the quality of pressure ulcer prevention,80 but recently it was argued that the measurement of PU occurrence per se has hardly any beneficial effect.81 Instead, one should focus on the whole process of PU prevention including the risk assessment. PU risk assessment on admission could be used as an auditable quality indicator.81,82 For example Duncan proposed to use the proportions of completed risk assessments using validated PU risk scales as a quality indicator. Proportions of patients with completed PU risk assessment on admission or the proportions of patients reassessed daily could be used.83 The number of completed PU risk assessments using a PU risk scale was also used as an audited quality indicator by a team of Canadian nurses in a pediatric setting.84

Discussion

Evaluation of PU risk scales can be performed from different perspectives: (1) from the measurement point of view, (2) from the clinical impact point of view, and (3) based on other practical considerations.

Quality of PU risk scale scores

When focusing on classical measurement theory then it is apparent that the evidence supporting validity and reliability of PU risk scale scores is inconclusive. Without doubt, PU risk scale scores are related to the probability of PU occurrence and therefore they are also somehow related to PU risk. Perhaps this association makes one imagine that PU risk scales work,85 but when going more into detail major drawbacks become apparent.

First of all, while there is a large number of PU risk factors described in the literature, the underlying evidence is inconsistent and varies depending on the population and factors taken into consideration. Thus, the content validity of PU risk scales is debatable. Although there are some risk factors that are included in nearly every scale (eg, mobility) there are many more factors that are not included, included in some scales only, or are not adequately weighted. Second, using PU development as a reference standard is inappropriate when computing diagnostic accuracy as long as the effect of prevention is not adequately taken into account. Therefore results of most diagnostic accuracy studies are limited. Third, results of construct validity studies are sparse but they reveal that the concept of PU risk is hardly separable from general health. One can also put it the other way round: PU risk scales do not measure PU risk but the degree of functional impairments and general care dependency. This finding is not surprising because most factors included in PU risk scales are similar to items of other measures of basic human functioning like the Activities of Daily Living (ADL) index86 or the Functional Independent Measure (FIM).87

Results from reliability and agreement studies are difficult to interpret. Although total scores proved to be reliable in some care settings, it is very unlikely that exact agreement will be achieved in clinical practice. Therefore, scale users must decide whether disagreements are clinically acceptable. It depends on the purpose and consequences of obtained risk scale scores how much error will be allowed to be introduced into clinical decision making. This challenges especially the use of rigid cut-off points and risk categories. There are various examples in the literature and clinical practice where measurement errors are not adequately addressed. For example, a Braden scale score of 18 or less is often used to trigger special PU preventive interventions or applications of pressure relieving devices.88,89 Given that the inter-rater reliability was high in the settings using this cut-off point, say >0.9, then score differences of up to 3 can be expected. Consequently, there is no reason to exclude patients with Braden scale scores of 20 from this regime.

Today, the validity and reliability limitations of PU risk scales are widely acknowledged. To deal with these problems the solution that was often recommended was to combine scores of PU risk scales with clinical judgment.1 Unfortunately, this recommendation, albeit often seen in the literature, is logically inconsistent because as Papanikolaou et al23 put it: “If PU risk assessment scales have such limitations, what contribution can they make to our confidence in clinical judgment, other than prompting us about the items, which should be considered when making such judgments?” (p. 294).

Clinical benefit

As of today, there is no evidence that PU prevention is more effective if PU risk scales were used. If PU risk scales are regarded primarily as ‘aide memoires’ then why should one not rather use simpler but perhaps more comprehensive forms of reminders instead of completing time consuming scales? There may be other, and perhaps more effective, ways of increasing the awareness of the PU problem in health care, eg, continuous monitoring of PU prevalence or incidence, staff involvement in PU studies, and education.33,61,90

Other considerations

Without doubt documentation of diagnostic and therapeutic interventions is not only important for multidisciplinary communication but also for reimbursement, quality assurance, and protection against lawsuits. However, based on the described limitations we doubt if using and documenting PU risk scores can satisfy these additional requirements. If PU risk assessment based on standardized scales is hardly valid and precise, and if scale use does not lead to a reduction in PU frequency, how can these tools improve clinical care?

Conclusion

Empirical evidence supporting the validity of PU risk assessment scale scores is weak and obtained scores contain varying amounts of measurement error. A clinical impact due do the application of these scales could not be demonstrated. These issues should be seriously taken into account whenever such instruments are applied in clinical practice. Since PU risk is strongly related to the general health status and severity of illness, it is questionable whether an additional completion of standardized PU risk scales is really needed. There is no rationale to use PU risk scales as a quality indicator for care processes.