Outcome Instruments in Spinal Trauma Surgery: A Bibliometric Analysis.

Study Design Literature review. Objective To identify outcomes instruments used in spinal trauma surgery over the past decade, their frequency of use, and usage trends. Methods Five top orthopedic journals were reviewed from 2004 to 2013 for clinical studies of surgical intervention in spinal trauma that reported patient-reported outcome instruments use or neurologic function scale use. Publication year, level of evidence (LOE), and outcome instruments were collected for each article and analyzed. Results A total of 58 studies were identified. Among them, 26 named outcome instruments and 7 improvised questionnaires were utilized. The visual analog scale (VAS) for pain was used most frequently (43.1%), followed by the Short Form 36 (34.5%), Frankel grade scale (25.9%), Oswestry Disability Index (20.7%) and American Spinal Injury Association Impairment Scale (15.5%). LOE 4 was most common (37.9%), and eight LOE 1 studies were identified (10.3%). Conclusions The VAS pain scale is the most common outcome instrument used in spinal trauma. The scope of this outcome instrument is limited, and it may not be sufficient for discriminating between more and less effective treatments. A wide variety of functional measures are used, reflecting the need for a disease-specific instrument that accurately measures functional limitation in spinal trauma.

Introduction

Patient-reported outcome instruments (PROIs) have become the gold standard for measuring treatment success in evidence-based medical research. The advantage of PROIs over other outcome measurements, such as radiographs, complication rates, or process of care measures, is that only PROIs directly address the patient's perception of their quality-of-life improvement. Studies of patient-reported outcomes are therefore integral to proving the effectiveness of treatments as our health care system changes, yet at this juncture no PROI has been designed for the specific needs of spinal trauma surgery research.1 2

High costs and broad impacts on patient quality of life make spine surgery particularly well suited for the application of PROIs. An estimated 160,000 spinal injuries occur per year in North America. The majority of these injuries are in the young or working population,3 4 so return to work and sustained quality-of-life outcomes, measured by many PROIs, are of great importance to evaluating the effectiveness of an intervention.

Another important clinical outcome in the treatment of spinal injuries is neurologic deficit. A large retrospective study at a level 1 trauma center in the United States found neurologic deficit present in approximately 20% of spinal trauma cases.4 Neurologic deficit is measured using neurologic function scales, such as the Frankel grade scale or American Spinal Injury Association Impairment Scale (AIS), which are administered by medical professionals rather than self-reported by patients.

As more studies incorporate patient outcomes as end points, it is important for researchers to use the available tools in a consistent fashion so that their results can be interpreted and compared across studies. To standardize the use of existing instruments in spinal trauma surgery research or to develop a spinal trauma–specific PROI, it is first necessary to consider how PROIs are currently used in the field. Therefore, a review of spinal trauma literature over the past 10 years was performed to identify the outcome instruments currently in use, their frequency, and usage trends. We hypothesize that a wide variety of PROIs are currently used in spinal trauma surgery and that no instrument captures all the domains of interest in spinal trauma research.

Methods

A review was conducted of articles published in five top orthopedics journals—Spine, European Spine Journal (ESJ), The Spine Journal, The Journal of Bone and Joint Surgery American (JBJS [Am]), and The Journal of Bone and Joint Surgery (British Volume) * (JBJS [Br])—from 2004 to 2013. Journals were selected based on impact factor and a track record of publishing relevant, high-quality spine research. The titles of all clinical articles published in these journals were screened on PubMed. The titles referring to clinical studies of surgical interventions in which outcomes were measured by PROIs or neurologic function scales were included for the analysis (Fig. 1). If the inclusion criteria could not be assessed from the title alone, the abstract was reviewed. The full-text article was reviewed if ambiguity remained. Review articles were excluded as were articles that had level 5 evidence. Level of evidence (LOE) was determined according to the definitions provided by the Oxford Center for Evidence Based Medicine.5 Only articles having LOEs 1 to 4 were included in our study. For each study that met the inclusion criteria, the following variables were recorded: title, author, year, LOE, sample size, general diagnosis, and outcome instruments used. The literature search and assignment of the LOE was performed by a spine fellow and three medical students he trained for these tasks. The analysis of outcome measures and their domains was performed by an attending spine surgeon, two fellows, and three medical students.

Fig. 1

Flow diagram depicting literature search strategy used to screen and identify clinical studies of surgical interventions in spinal trauma that report use of outcomes instruments (patient-reported outcome instruments or neurologic function scales). Abbreviation: LOE, level of evidence.

All studies in this review are of patients with the diagnoses of spinal trauma. Specific diagnoses within spinal trauma included cervical, thoracic, and lumbar burst fracture, odontoid fracture, cervical transverse process fracture, sacral fracture, cervical dislocation, and spinopelvic dissociation. The trend in outcome instrument usage over time was reported along with the number of studies published by journal and frequency of each LOE.

PROIs encountered were categorized into subgroups of pain scales (e.g., visual analog scale [VAS]), functional/disability scales (e.g., Oswestry Disability Index [ODI]), and general health-related quality of life (HRQoL) questionnaires (e.g., Short Form 36 [SF-36]). The top five most frequently used outcome instruments were recorded and analyzed. Additionally, errors and inconsistencies of PROI score reporting that were identified as problematic for interstudy comparisons were analyzed and are reported in Table 1.

Table 1

Errors and inconsistencies of patient outcome score reporting

Articles excluded for:

1. Not reporting numerical data (e.g., graphical or statistical reporting of PROI scores)

2. Reporting only change in scores without preoperative or postoperative scores

3. Failing to report preoperative or postoperative scores

4. Significantly modifying a standard PROI

5. Reporting only domain or component scores rather than total score for a PROI

6. Baseline scores measuring health state prior to the spinal injury as recalled by patient

Abbreviation: PROI, patient-reported outcome instrument.

Results

From 2004 to 2013, Spine, ESJ, The Spine Journal, JBJS (Am), and JBJS (Br) published 19,736 articles. We identified 1,079 clinical studies of surgical interventions that made use of one or more outcome measures. Of these, 58 articles focused on spinal trauma. Outcome instrument use increased over the past 10 years from 3 studies in 2004 to 11 studies in 2013, although not in a consistent progression (Fig. 2). ESJ published the most studies using outcome instruments with 30 (51.7%) of the 58 total articles in this study (Fig. 3).6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

Fig. 2

Number of studies reporting use of outcome instruments by year.

Fig. 3

Number of studies using outcome instruments by journal. Abbreviations: ESJ, European Spine Journal; JBJS, Journal of Bone and Joint Surgery.

Overall, 33 unique outcome instruments were used, including 26 named instruments and 7 improvised questionnaires. Table 2 lists the instruments that were used more than once. PROI use statistics are broken down by PROI category and type in Table 3. The top five outcome instruments were the VAS pain scale (25 uses, 43.1% of articles), SF-36 (20 uses, 34.5%), Frankel grade scale (15 uses, 25.9%), ODI (12 uses, 20.7%) and the American Spinal Injury AIS (9 uses, 15.5%; Fig. 4). The total usage of these top five instruments accounted for 66.9% (81 uses) of all instrument uses. An average of 2.1 outcome instruments were used per study (range 1 to 8). In descending order, the top five most common combinations of two PROIs were SF-36 with VAS (9 studies), ODI with VAS (7 studies), SF-36 with ODI (6), VAS with AIS (5), and tied for fifth were VAS with Frankel, SF-36 with Frankel, and SF-36 with AIS (4 studies each). These results are expected given the frequency of use of these instruments individually.

Table 2

Outcome instruments used more than once (2004–2013)

Instrument	Uses	%
Visual analog scale	25	43.1
36-Item Short Form	20	34.5
Frankel grade scale	15	25.9
Oswestry Disability Index	12	20.7
American Spinal Injury Association Impairment Scale	9	15.5
Denis Pain Scale	5	8.6
Neck Disability Index	4	6.9
Roland Morris Disability Questionnaire	4	6.9
Hannover Functional Ability Questionnaire	2	3.4

Reporting percentages of total spinal trauma studies (n = 58).

Table 3

Spinal trauma outcome instrument usage summary

Statistic	Value
No. of uses of PROIs by category
Pain scalesa	31 (3 unique scales)
Functional and disability outcome instrumentsb	30 (12 unique scales)
General HRQOL questionnairesc	22 (3 unique scales)
Other	13 (13 unique scales)
No. of uses of neurologic function scalesd	24 (2 unique scales)
Total instrument uses	120
Studies using outcome instruments	58
Average no. of outcome instruments per study	2.1 (range 1–8)
No. of unique named instruments	26
No. of improvised questionnaires	7

Abbreviation: HRQOL, Health Related Quality of Life; PROI, patient-reported outcome instrument.

Includes visual analog scale, Denis Pain Scale, and McGill Pain Questionnaire.

Includes Cervical Spine Outcomes Questionnaire, Functional Independence Measure, Hannover Functional Ability Questionnaire, Gibbons Nerve Functional Outcome Criteria, Majeed Functional Outcome Criteria, North American Spine Society Cervical Spine Outcome Instrument, Neck Disability Index, Neck Pain and Disability Scale, Nurick Score, Oswestry Disability Index, Roland Morris Disability Questionnaire, and Smiley-Webster Scale.

Includes Short Form 36, EQ-5D, Every Day Life Questionnaire.

Includes Frankel grade scale and American Spinal Injury Association Impairment Scale.

Fig. 4

Outcome instrument frequency as a percentage of total articles (n = 58). Abbreviations: AIS, American Spinal Injury Association Impairment Scale; ODI, Oswestry Disability Index; SF-36, Short Form 36; VAS, visual analog scale.

A subanalysis comparing studies that focused on injuries of the thoracolumbar spine (45 studies) and cervical spine (13 studies) showed far less use of VAS among cervical studies (Fig. 5). ODI, which measures low back pain, was appropriately seen only in thoracolumbar studies.

Fig. 5

Comparison of frequency of use of common patient-reported outcome instruments by region of injury (cervical versus thoracolumbar). Abbreviations: AIS, American Spinal Injury Association Impairment Scale; ODI, Oswestry Disability Index; SF-36, Short Form 36; VAS, visual analog scale.

Level 4 evidence was most common among the articles (22 articles, 37.9%; Fig. 6). LOE 1 was least common, with 8 (10.3%) LOE 1 articles published in the past 10 years. ESJ published the most studies using outcome instruments and had the highest-quality evidence as determined by the greatest number of LOE 1 and 2 studies (Fig. 7).

Fig. 6

Level of evidence (LOE) of studies using outcome instruments as a percentage of total articles (n = 58).

Fig. 7

Level of evidence (LOE) of studies using outcome instruments categorized by journal. Abbreviations: ESJ, European Spine Journal; JBJS, Journal of Bone and Joint Surgery.

Discussion

The growing demands for evidence-based medical care have driven increased use of PROIs in spinal trauma surgery research. For the results of these studies to be comparable and useful, standards of PROI use are important. However, at the current juncture, no such standards exist, nor has a PROI been developed that addresses the specific needs of spinal trauma surgery research. Our review represents the first study, to our knowledge, that identifies the top outcome instruments in spinal trauma surgery, their relative frequency of use, and the trends in their use. These statistics will be useful for investigators selecting among PROIs for a study, for guideline and consensus builders in need of a starting point for drafting standards of PROI use, and for the spinal trauma research community at large to take a closer look at the validity and reliability of PROIs currently relied upon for proving the effectiveness of new or existing treatments.

Our review of five top orthopedic journals found 58 studies of spinal trauma surgery that used PROIs. Thirty-three unique PROI were used and an average of 2.1 PROIs were used per study, which reveals a lack of standardization in PROI selection. In order of frequency of use, the top five instruments were VAS, SF-36, Frankel grade scale, ODI, and AIS. The VAS was the preferred pain scale and PROI overall. There was no consensus regarding a preferred functional/disability outcome instrument as 12 different PROIs were used to measure functional/disability outcomes in the 58 studies.

A review of the methods and results sections of the studies that used the top instruments revealed a variety of errors and inconsistencies of score reporting. The failure to report numerical data (i.e., reporting scores only graphically or statistically), reporting only change in scores, reporting only postoperative scores, and significantly modifying standard outcome instruments prevent valid interstudy comparisons from being possible. These errors and inconsistencies of score reporting need to be recognized by authors and reviewers and corrected to ensure that studies being considered for publication will add to, rather than obfuscate, the evidence for a particular treatment.

This review is the first of its kind in the spinal trauma literature; however, Hunt and Hurwit conducted a similar study of PROIs in foot and ankle surgery research.64 Stadhouder et al identified 21 outcome measures commonly used in spinal trauma research and reviewed the literature for validation studies of these instruments in the spinal trauma population.2 The researchers found an overall lack of research into the applicability and validity of existing outcome measures in spinal trauma research, but, as a road map for future work, the authors highlight key HRQoL domains from the World Health Organization's comprehensive International Classification of Functioning, Disability and Health that should be measured by a new instrument. Among other studies of spinal trauma surgery, the outcomes considered most important in patients with or without spinal cord injury (SCI) are neurologic recovery, stability, return to work time, and pain.3 17 65 66 The top instruments from the present study are reviewed next in regard to their validity and relevance to these core outcomes of interest.

Pain Scales

VAS pain scale is a simple tool used to measure pain intensity on a scale of 0 to 10. This PROI can be completed in less than a minute, minimizing the burden to patients. However, McCormick et al questioned the value of the VAS pain scale in a recent review of spine surgery research, stating that “research has failed to consistently find meaningful and reliable use for these tools.”67 Nonetheless, ease of use bolsters the popularity of the VAS.

General Health-Related Quality of Life Questionnaires

General HRQoL questionnaires capture several widely relevant health status domains, which enable comparisons to be made across different diseases. The SF-36 has become prevalent since the 2002 consensus statement by Neugebauer et al that encouraged its use.68 The SF-36 can be used to study cost-effectiveness by converting its scores to health utility scores on a scale of 0 to 1.

The SF-36 is validated for many spinal disease populations including patients with back pain, spinal injury, and disk herniation.69 70 The most recent version is the SF-36v2.71 Three important limitations of this instrument are the patient burden of a 36-question survey, its poor sensitivity to functional limitations in spine patients, and the fact that normative values for the SF-36 do not represent the spinal trauma population well.1 The SF-36 in combination with a disease-specific PROI may be the ideal set of outcome assessment instruments for spinal trauma.

Neurologic Function Scales

Although the Frankel grade scale and AIS are not PROIs, they provide crucial prognostic information and outcome assessments, particularly in patients with SCI who represent a considerable fraction of all spinal trauma patients. In one study, patients with SCI comprised 21.7% of all spinal injury patients.4 The Frankel grade scale assesses neurologic impairment on a scale of A to E, where A indicates complete paralysis and E indicates normal motor and sensory function.72 The AIS is a newer scale that builds upon the strengths of the Frankel grade scale.73 The Frankel grade scale and the AIS were used consistently in spinal trauma studies (24 uses in 58 studies), with the Frankel grade scale being used more than the top functional/disability instrument (the ODI). Despite being newer and generally more supported than the Frankel grade scale, the AIS was less frequently used over the past 10 years (9 versus 15 uses).

Publishers of the AIS emphasize the importance of supplementing impairment scales with PROIs to fully assess function and activities of daily living.73 Two PROIs recommended by SCI experts are the Functional Independence Measure and the Spinal Cord Injury Measure.2 73 Weaknesses of these PROIs include their narrow focus on the SCI population, and the suitability of the Spinal Cord Injury Measure only for patients with substantial neurologic damage.2 Despite their recommended use, the Functional Independence Measure and Spinal Cord Injury Measure were used in very few of the 58 studies in our review.

Functional/Disability Questionnaires

The ODI is a 10-question survey that measures the impact of chronic low back pain on activities of daily living. The survey is concise and straightforward, allowing patients to complete it quickly. The ODI was developed by Fairbank et al in 1980 for chronic low back pain and has undergone several revisions, the latest of which is the ODIv2.1a.74 75 76 77 It is used widely in degenerative lumbar spine research and is therefore familiar to many investigators. However, its suitability for spinal trauma has been questioned.2

The results of our study are consistent with the top instruments analyzed by Schoenfeld and Bono and by Stadhouder et al; however, only our study analyzed the full range of instruments and their frequency of use.1 2 The widest range in patient outcome instrument use was seen among functional/disability questionnaires. The ODI was the most common of these, appearing in 20.7% (12) of articles included in our study. A total of 12 different functional/disability questionnaires were used in all. One reason for the wide range of functional/disability instruments (e.g., ODI, Neck Disability Index, Roland Morris Disability Questionnaire, Hannover Functional Ability Questionnaire) is that no outcome instrument has been developed specifically for the study of spinal trauma, as Schoenfeld and Bono and Stadhouder et al noted,1 2 which is a significant obstacle to proving the effectiveness of treatments because radiographic measures, complication rates, and process of care measures are no longer considered sufficient evidence.78

Although the ODI, Neck Disability Index, and Roland Morris Disability Questionnaire are the most frequently used functional/disability instruments, these instruments were developed for studies of chronic back pain, and none of them have been validated in the spinal trauma population. As stated by Stadhouder et al, outcome measures designed for chronic back pain “may theoretically have limited applicability to spinal trauma patients in the sense that the domains they measure and the relative weighting of each in the scoring do not correspond to the domains perceived to be important in spinal trauma patients, a fundamentally different population.”2 For example, each of the 10 questions of the ODI asks about the degree to which lower back pain limits various activities of daily living. The pain domain may not be sufficient to discriminate between treatment success and failure in spinal trauma, so functional/disability questionnaires like the ODI that focus entirely on pain may not be suitable measures of outcome in spinal trauma surgery.

We acknowledge that the current study has several limitations. First, we did not search neurosurgical journals. However, Spine and the Spine Journal reach both orthopedic and neurosurgical audiences. Our study is intended as a select sampling and not a comprehensive review of the literature. Furthermore, we believe that inclusion of more journals and articles may not have substantially altered the main conclusions of the study. Second, it is possible that an article was overlooked during our extensive review of the literature if neither title nor abstract alluded to measurement of patient outcomes.

Conclusion

An earlier review of PROI in spine surgery by McCormick et al proposed that well-designed studies should utilize a general HRQoL instrument, pain scale, and disease-specific PROI.67 Neurologic status is the most important outcome to assess in spine trauma, and in those patients who are neurologically intact long-term, back pain becomes most important. The current review of the spinal trauma literature shows a preference for the SF-36 as a general HRQoL instrument and the VAS as a pain scale, while highlighting the need for a disease-specific PROI to be developed. Until that goal is realized, the best approach to studying outcomes in a standardized fashion is a commonly used battery of PROIs that minimizes respondent and administrative burden where possible. To that end, we suggest VAS for pain, SF-36 for general HRQoL, and Frankel for neurologic status.