Development and validation of a computerized algorithm for International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI).

STUDY
DESIGN: Validation study.
OBJECTIVES: To describe the development and validation of a computerized application of the international standards for neurological classification of spinal cord injury (ISNCSCI).
SETTING: Data from acute and rehabilitation care.
METHODS: The Rick Hansen Institute-ISNCSCI Algorithm (RHI-ISNCSCI Algorithm) was developed based on the 2011 version of the ISNCSCI and the 2013 version of the worksheet. International experts developed the design and logic with a focus on usability and features to standardize the correct classification of challenging cases. A five-phased process was used to develop and validate the algorithm. Discrepancies between the clinician-derived and algorithm-calculated results were reconciled.
RESULTS: Phase one of the validation used 48 cases to develop the logic. Phase three used these and 15 additional cases for further logic development to classify cases with 'Not testable' values. For logic testing in phases two and four, 351 and 1998 cases from the Rick Hansen SCI Registry (RHSCIR), respectively, were used. Of 23 and 286 discrepant cases identified in phases two and four, 2 and 6 cases resulted in changes to the algorithm. Cross-validation of the algorithm in phase five using 108 new RHSCIR cases did not identify the need for any further changes, as all discrepancies were due to clinician errors. The web-based application and the algorithm code are freely available at www.isncscialgorithm.com.
CONCLUSION: The RHI-ISNCSCI Algorithm provides a standardized method to accurately derive the level and severity of SCI from the raw data of the ISNCSCI examination. The web interface assists in maximizing usability while minimizing the impact of human error in classifying SCI. SPONSORSHIP: This study is sponsored by the Rick Hansen Institute and supported by funding from Health Canada and Western Economic Diversification Canada.

Introduction

Spinal cord injury (SCI) results in impairment to motor and sensory function.[1] A reliable and a valid assessment of the extent and severity of these physical impairments is critical to supporting clinical care, prognosis and research.[2] The International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI;[3]) were developed for this purpose. They were originally published in 1982 by the American Spinal Injuries Association (ASIA;[4]) and are now overseen by the International Standards Committee within ASIA. The most recent update was made in 2011, which includes updates to the worksheet and introduction of the non-key muscle examination.[3]

The ISNCSCI examination and classification provide a common language to describe the extent of motor and sensory dysfunction due to SCI. Using the classification rules as outlined in the ISNCSCI, there are a number of derived elements that include the following: motor and sensory levels — right and left, a single neurological level of injury (NLI), the completeness of SCI, zone of partial preservation, and the severity of SCI using the ASIA Impairment Scale (AIS).

Monitoring the level (motor, sensory and NLI) and severity (AIS) of SCI and the total motor and sensory scores from the time of injury throughout the individual's lifetime is routinely done as a part of clinical care to detect an improvement or a deterioration in neurological impairment[5, 6] or to evaluate the influence of clinical interventions on neurological recovery.[7, 8] The ISNCSCI examination is also often utilized as the primary outcome of clinical studies, and its classification components are used to determine eligibility and stratification of participants in clinical trials.[9, 10, 11, 12, 13] As such, it is imperative that the ISNCSCI be applied both accurately and reliably.

The reliability of the raw individual motor and sensory scores and the rectal examination has been previously studied and optimized through education and training of examiners.[14, 15, 16, 17, 18] With the implementation of the Rick Hansen Spinal Cord Injury Registry (RHSCIR;[19]), a pan-Canadian prospective, observational registry for traumatic SCI, the majority of RHSCIR examiners received training in the ISNCSCI examination and classification. Training has been demonstrated to improve accuracy in scoring and classification.[20, 21] We noted that, although the ISNCSCI was widely used, when the RHSCIR central coordinating site staff examined the raw motor, sensory and rectal examination data and compared the AIS and NLI derived from the local RHSCIR site, there was a significant problem with classification errors. Other studies have also reported high misclassification rates in the AIS (11.9–13%) and motor level (18–26%) determination.[16, 21, 22] Reasons for some of these classification errors include multiple revisions to the ISNCSCI rules over the decades, as well as inconsistent interpretation and implementation of scoring and classification rules among examiners.

Concerns regarding errors in the interpretation and application of the ISNCSCI classification led to the development of the Rick Hansen Institute-ISNCSCI Algorithm (RHI-ISNCSCI Algorithm), a computerized algorithm to provide the correct interpretation of the ISNCSCI neurological exam and improve the accuracy and validity of the derived AIS, NLI, motor score and other classification elements. The advantages of a computer algorithm include the following: improving the accuracy of the classification (particularly in deriving the AIS, total motor score and NLI), reducing the time to classify large numbers of cases and providing education on the ISNCSCI by producing immediate feedback to confirm clinician classification.

The purpose of this paper is to describe the development and validation of the RHI-ISNCSCI Algorithm.

Materials and methods

ISNCSCI algorithm working group

The RHI-ISNCSCI Algorithm (henceforth referred to as the Algorithm) was built based on the 2011 version of the ISNCSCI and 2013 version of the worksheet.[23] To ensure that the ISNCSCI rules were interpreted correctly and to promote collaboration, an international group of experts from ASIA and International Spinal Cord Society (ISCoS), including members of the ASIA International Standards and Education Committees, were engaged in an advisory capacity. These international experts and the RHI team of clinical experts and software developers formed the ISNCSCI Algorithm Working Group (IAWG). The IAWG collaborated on design, development and validation of the Algorithm.

Architecture

The Algorithm is made up of two components: a library containing the logic required for performing the ISNCSCI calculations and classifications and a web interface. Both components were developed using an agile process where the capabilities of the application were increased incrementally. Every increment was reviewed by the IAWG and tested in an internal test website before proceeding onto the next increment.

The algorithm library

The algorithm library component, written in C#.Net, uses raw motor and sensory examination grades as an input to calculate the following derived elements:

Motor and sensory subscores

Neurological Levels — sensory right and left, motor right and left

NLI

Completeness of Injury — Complete or Incomplete

AIS

Zone of partial preservation — sensory right and left, motor right and left

The web interface

A web interface was built to make the Algorithm publicly accessible and to provide a method of supporting beta testing by members of the international SCI community. The dermatome map is in Standard Vector Graphics format compatible with modern web browsers. The interface is freely accessible and is located at http://www.isncscialgorithm.com/ (Figure 1).

Figure 1

RHI-ISNCSCI Algorithm Web Interface.

The web application approach was chosen as it provides many benefits. The Algorithm is not tied to a specific operating system or a software version that allows it to be shared and used easily, thus minimising associated developmental costs and distribution efforts. Given that the application is centralized at a specific web address, there is only one application that requires updating when a new version becomes available. Users can be sure that they are using the latest version of the application. In addition, providing frequent updates ensures that there are no delays in releasing new features, which is a frequent problem with formal product re-releases. An internet connection and a modern web browser are the only requirements for accessing the application, which makes the Algorithm easily accessible internationally.

Development and validation

Once the development of the application was complete, it underwent validation testing of both the Algorithm and the web interface components. Each of the five phases is briefly described below (Figure 2).

Figure 2

RHI-ISNCSCI Algorithm Development and Validation. InSTeP, International Standards Training e-learning Program; RHSCIR, Rick Hansen Spinal Cord Injury Registry.

Phase one

During phase one (the initial logic development), a logic model was developed to classify cases where all raw scores were known (that is, cases did not include ‘not testable' (NT) values) using the 2011 version of the ISNCSCI.[3] The initial logic model was developed using International Standards Training e-learning Program[24] cases originally developed by ASIA for ISNCSCI training and hypothetical cases designed to assess specific areas of the logic model (including unlikely scenarios, for example, where every dermatome and myotome are left blank or have the value ‘0', so on.). These included expert cases that were accumulated from the literature[25] and provided by Dr Ralph Marino to test what were considered to be uniquely challenging classification rules.

Phase two

Because of the importance of real world testing, the developed logic was tested in phase two using the real-life cases obtained and scored by trained clinicians at acute and rehabilitation hospitals in one of the 31 Canadian RHSCIR sites. Any discrepancies between the Algorithm and the clinician-generated results were independently reviewed by at least two of the IAWG experts, and the Algorithm was updated accordingly. On completion of internal testing, the Algorithm was presented at both the ISCoS and Academy of Spinal Cord Injury Professionals 2012 meetings at which time the public beta testing website was launched. The beta testing website was used to further validate the Algorithm and to gather feedback.

Phase three

In phase three, additional features of the Algorithm were developed that included incorporating logic for NT data and other user-friendly features. This logic was developed using cases from phase one and additional hypothetical cases with NT values.

Phase four

Phase four included re-running cases used in phase two and additional randomly selected RHSCIR cases including those with NT values. This tested the new logic developed in phase three and ensured the ongoing accuracy of logic previously developed in phase one. Any discrepancies between the Algorithm and the clinician-generated results were independently reviewed by at least two of the IAWG experts, and the Algorithm was updated accordingly.

Phase five

In phase five, cross-validation testing of the logic with new RHSCIR cases was conducted to confirm accuracy of the Algorithm. Version 1.0 of the Algorithm was published on the public website in May 2014. The Algorithm and interface were then made available in an open source format in http://www.isncscialgorithm.com/SourceCode.

Results

Algorithm features

A list of key features of the algorithm is available in Table 1. Features designed to classify challenging cases include the following: addition of the ability to record non-key muscle functions and the ability to classify cases with NT dermatomes/myotomes. This NT logic provides AIS classification and calculation of all derived variables even when one or more dermatomes/myotomes have a value of NT, provided that only a single option is possible based on the other values entered. If more than one option is possible, unable to determine (UTD)[23] along with all possibilities will be displayed. Also, unique to this Algorithm is the use of a new symbol (‘!') to denote motor or sensory deficit not related to SCI. This allows standardized documentation and tracking of non-SCI-related changes on top of providing accurate AIS classification.

Table 1

Key features included in RHI-ISNCSCI Algorithm

Classification features
5* option for motor score grading	Aligns to 5* definition of ISNCSCI (i.e. pain, disuse atrophy or other)
! Symbol	Allows tracking of motor or sensory impairment due to reasons other than SCI that are inconsistent with the 5* definition (i.e. peripheral neuropathy, plexopathy, pre-existing myoneural disease, other) and maintains accurate classification of the SCI
Data entry for non-key muscles	Two dropdown boxes for right and left non-key muscle function below the level of injury. Once entered, the algorithm considers non-key muscle function on each side using the, ‘motor function more than three levels below the motor level for that side along with S4/5 sensory sparing or ‘deep anal pressure'[23] rule to determine motor completeness. Aligns with ISNCSCI's newly defined non-key muscle function standardized innervations list
NT logic	Allows selection of NT for both key myotomes and dermatomes. This NT logic provides an AIS classification even with an NT score at one or more dermatomes/myotomes if only a single AIS option is possible based on the values entered. If more than one AIS option is possible, ‘UTD' is displayed. However, even in those cases, the algorithm will calculate and can show all possibilities.
User friendly features
• Colour-coded dermatome man provides visual feedback on sensory scores as they are entered to facilitate clinical review and identification of clinical abnormalities/potential raw score errors (e.g. sensation in C4 that extends rostrally and is entered as T3)
• Ability to save or print exam information in the format of the international standards 2013 assessment worksheet that could be filed as a part of the health/research record (application does not store any exam information for privacy purposes)
• Web interface mirrors the 2013 international standards assessment worksheet to maximize learnability, efficiency and memorability for users already familiar with the assessment worksheet — minimizes error due to web interface and the assessment worksheet having differing locations of data fields
• Data entry supported via keyboard or a pre-defined pick list of only valid values for each field (e.g. when C5 light touch right is selected, the pick list will only display the options 0, 1, 2, 0!, 1!, NT and NT!). If keyboard entry is used, the values that are accepted for a field are restricted to the pre-defined values only. The pick list of pre-defined values is displayed, regardless of the method the user chooses to use to enter data (i.e. pick list or number pad) in order to help educate users about what information to put into each box
• Downward value propagation that automatically fills the cells below with the first entered value to save users from entering same values multiple times
• Application controls (e.g. ‘calculate' or ‘save as pdf' buttons) fixed at the top of the screen to remove the need to scroll up or down to do frequently performed actions
• Compatible with the tablet by ensuring that all clickable elements (i.e. buttons and cells) are optimised to support touch input using a finger
• Ability to collect feedback from the international SCI community including desired features and/or disagreement with algorithm-obtained determination of classification to allow constant improvement

Abbreviations: AIS, ASIA Impairment Scale; NT, not testable; RHI-ISNCSCI, Rick Hansen Institute-International Standards for Neurological Classification of Spinal Cord Injury; SCI, spinal cord injury; UTD, unable to determine.

Features increasing usability include the following: a colour-coded dermatome man who provides visual feedback on sensory scores entered and the presence of non-SCI-related deficits, value entry restriction, downward value propagation (to speed entry of identical sensory and motor exam values), saving and printing the entered examination in 2013 ISNCSCI worksheet format, tablet compatibility, user feedback function, so on. These features were developed to minimize errors in data entry while maximizing usability, user education and the ability to capture comprehensive information about a patient's neurological impairments. On the basis of user feedback and input from the IAWG, the Algorithm is being continually evaluated. As of 01 July 2015, the Algorithm website has had 29 266 visits from 139 countries.

Validation results

Eleven International Standards Training e-learning Program cases and 37 hypothetical cases were used for logic development in phase one. These cases and 15 additional cases were used for NT logic development in phase three. Logic testing was performed in phases two and four using 351 and 1998 real-life RHSCIR cases, respectively. There was a 6.6% (23/351 cases) discrepancy in phase two logic testing with 8.7% (2/23) of discrepant cases leading to changes in the Algorithm. In phase four, 14.3% (286/1998) of the cases were discrepant with only 2.1% (6/286) of discrepant cases leading to changes in the Algorithm. The remaining 97.9% (280/286) contained clinician errors. Phase five, the cross-validation testing, had a 8.3% (9/108) discrepancy between the clinician and the Algorithm determined classification, but no changes to the Algorithm were required, as all were determined to be clinician classification errors. See Table 2 for more details.

Table 2

Summary of validation results for phases I–V

Of the 295 (286 in phase four, 9 in phase five) RHSCIR cases with discrepancies between the Algorithm and the clinician classifications in phases Four and Five, 289 cases contained either single or multiple clinician errors. The errors involved NLI in 151 cases, AIS in 80 cases, motor level in 86 cases and sensory level in 39 cases.

Discussion

The Algorithm was developed and validated to assist clinicians and researchers with correctly performing the neurological classification as per the ISNCSCI standards. The Algorithm builds on the current ISNCSCI standards by including logic to classify cases with NT dermatomes/myotomes and logic to support previous grey areas of classification, specifically the non-key muscle function and dermatome/myotome changes due to pathology other than SCI.

The Algorithm development was initiated in November 2011, and three other computerized algorithms for ISNCSCI are known to have been developed and validated before or since that time.[22, 26, 27] Two of these three ISNCSCI computerized algorithms published data on the validation; one used a small group of patients[22] and the other used patients from only one phase of the care continuum (sub-acute SCI patients undergoing rehabilitation).[27] The Algorithm in this paper was created and validated on data from 930 patients (2106 unique exams) in both acute and rehabilitation phase of care, which may better reflect the heterogeneity and challenges associated with classifying real-life patient cases. In addition to accurately reflecting the latest version of ISNCSCI classification rules and worksheet, the Algorithm includes user-friendly features (for example, dropdown propagation of initially entered exam values, provision of tools to support integration into electronic medical record systems) and the ability to record details of a patient's neurological impairment (for example, non-key muscles, impairments not related to SCI), which are currently not included in other algorithms.

When comparing the clinician classifications with the Algorithm, the Algorithm identified clinician errors in 14.0% (280/1998) and 8.3% (9/108) of the cases in phases four and five of the validation testing, respectively, which are comparable to other studies that reported 10.2–13% of clinician errors in ISNCSCI classification.[22, 28] The most common type of clinician error was miscalculating NLI (151 cases) followed by motor level (86 cases), AIS (80 cases) and sensory level (39 cases). This is similar to the results of other studies examining ISNCSCI classification errors with[29] or without[16, 21] algorithms, both reporting greater difficulty in classifying AIS over the sensory level. A greater percentage of the discrepancies was due to clinician classification error in phase four, which contained NT values when compared with phase two, which did not contain any NT values (91.3% in phase two vs 97.9% in phase four). The increased clinician error seen in this phase may indicate that clinicians have a harder time classifying cases with NT values. Further clarification and education of classification in cases with NT values should be provided and can be supported by the use of the Algorithm, which provides all possible options for calculated fields.

Clinician errors persist even after receiving training on performing the ISNCSCI classification likely due to staff turnover, the complexity of the classification rules and the fact that there are some cases where the classification rules are not very clear. This suggests that there is a need to incorporate computer algorithms into both research and clinical settings to ensure the use of the most current classification rules in a standardized manner and support ongoing clinician education. Use of the Algorithm will improve the accuracy of the neurological data used in clinical trials (for example, stratifying participants, determine inclusion/exclusion criteria, ensure quality of clinical data), assist clinicians in targeting appropriate interventions based on the AIS[2] and educate clinicians on how to apply the ISNCSCI classification rules.[22, 27] The importance of accurate assessment of NLI, AIS and the total motor score in monitoring patients for neurological deterioration and the influence of therapeutic interventions, be they pharmacological, surgical or medical, is paramount.

Despite its many uses, the Algorithm, like other computerized algorithms, remains vulnerable to data entry errors and cannot compensate for inaccurate scores obtained during the clinical examination.[22] There will always be scenarios that preclude the use of a standardized computer algorithm (for example, an individual sustains a SCI at two different levels). As such, no algorithm can replace the clinical reasoning required to accurately classify these exceptional cases. For this reason, the Algorithm has included a clinician sign off section on its printable PDF, and its feedback feature encourages users to submit discrepant cases to allow for ongoing improvements.

ASIA has provided ongoing publications[3, 23, 25] and educational tools[24] to help clarify both the examination and classification components of the ISNCSCI as they have changed over the years. Despite this, many challenges remain in obtaining a reliable level and severity of neurological classification of SCI. Reliable classification requires correctly performing the clinical examination to determine motor, sensory and rectal examination scores and then accurately applying the classification rules according to the most updated version of the ISNCSCI while ensuring that non-SCI-related changes are appropriately identified as such during the assessment and classification. Additional challenges emerge if any of the dermatomes/myotomes are NT, as no formally recognised method exists to account for this during the classification even with the 2011 update of the ISNCSCI.[23, 25] Other areas that could use further definition by the Standards Committee include: capturing non-SCI-related weakness so that it is clearly not included in the classification but utilised to track patient function, developing a standardized method of incorporating NT scores into the classification, clarification of how to determine motor complete or incomplete status (that is, AIS B vs C) in an individual where the motor level is below S1, etc. Collaborating with members of the ASIA Standards Committee on the IAWG allowed for discussion and communication of these issues so that they could be considered by the Standards Committee in the future, while ensuring that the resulting Algorithm aligns with expert opinion. Ongoing clinician training on the current methods of examination and classification is required, and the use of a computerized algorithm to support classification accuracy and learning (that is, reconcile clinical classification with computerized algorithm) will help ensure more accurate classification of individuals with SCI.

The Algorithm is currently being used for data validation (for both clinical trials and observational studies) and as a part of clinical training on how to complete and classify cases using ISNCSCI. Also, version 1.0 of the Algorithm provides tools to support the implementation of the Algorithm components into existing electronic medical records and research databases. As the Algorithm is a valuable tool for those who are learning how to perform the examination, additional features to support accurate, real-time clinical assessments are being planned. These include tablet and/or smart phone compatibility, ability to link to ASIA's learning resources for motor and sensory testing of a specific myotome/dermatome, etc. Ongoing support for integration of the ISNCSCI examination into electronic medical records and research databases and efforts to maintain alignment to ISNCSCI as they are updated by ASIA's Standards Committee are also needed. Additional ideas for features submitted by users will also be considered to ensure that the application continues to meet the needs of the international SCI clinical and research communities.

Conclusion

The Algorithm provides a current, validated and a standardized method to determine the level and severity of a SCI in alignment with version 2011 of the ISNCSCI and version 2013 of the worksheet. The web interface and Algorithm library were designed to maximize usability while minimising the impact of human error in performing the derivations required to complete the classification. Although there are areas of the ISNCSCI that require clarification moving forward, the integration of international experts from both ASIA and ISCoS in this project provided a unique collaboration opportunity that will continue as the ISNCSCI evolves.

Data Archiving

There were no data to deposit.