Orchard Sports Injury Classification System 10.1 Plus: An End-User Study.

The purpose of this study was to establish the level of ease of use and effectiveness of the Orchard Sport Injury Classification System (OSICS) 10.1 Plus for recording injuries and interventions. Three hundred and forty-two (males=148, females=192, no response=2; age=30.9±9.5y; experience=9.1±10.5y) athletic trainers (ATs) in the United States completed the survey. Participants were primarily employed in the secondary school (n=135) or collegiate setting (n=171). Participants entered system includes the OSICS 10.1 to catalog injuries and Current Procedural Terminology (CPT) codes to document interventions. Participants completed an 18-item end-user evaluation to assess the ease of use and effectiveness of the OSICS 10.1 Plus (5-point Likert scale). Participants indicated that the OSICS 10.1 Plus is overall easy to use (4.1±0.7pts), easy to enter an injury (4.1±0.8pts), and easy to enter the associated interventions (3.9±0.8pts). Respondents were neutral about whether the OSICS 10.1 Plus matched their current injury (3.5±1.0pts) or intervention (3.5±0.9pts) records. A majority of participants indicated that they could find the injury (281/342, 82.2%) and interventions (225/342, 65.8%) of interest. A majority of respondents (205/342, 60.0%) indicated they would consider using OSICS 10.1 Plus for injury surveillance in clinical practice. The OSICS 10.1 Plus could serve as an effective and useful mechanism for injury surveillance with minor modifications; however, we, as professionals in sports healthcare, need to improve regular medical documentation first so that we are better able to conduct injury surveillance among our patients.

INTRODUCTION

Participation in sports has dramatically increased over the last three decades and sport related injury has increased proportionally (9). Sports injury surveillance classification systems have become increasingly popular during this time to gather information on these injury types, rates, mechanisms, and risk factors (11). Although many systems exist, little is known about the time and training that is necessary to use them. However, it is commonly assumed by practitioners that injury surveillance is an added burden to their already heavy workload (4). The data from these surveillance systems allow for an objective assessment of the risks of sports participation and guide changes to rules, equipment, and injury prevention efforts (9,11). Sport injuries have been examined thoroughly in previous studies; however, there is a need to create a universal or comprehensive system.

The Orchard Sports Injury Classification System (OSICS) is a widely used (20), free program, specifically designed for sports medicine application (21). Currently, OSICS is being used for a broad spectrum of sports, including soccer, cricket, tennis, and rugby that can be easily adapted to include other sports. OSICS has gone through many modifications since its inception, resulting in different versions, which address the criticisms of previous editions (22). Drastic changes were made in the tenth edition to improve efficiency, accuracy, and inter-user agreement over previous editions (22). These upgrades include the development of a fourth character in the coding system for more detailed coding (22). Additional sections of codes include pediatric, and post-surgical sections, and expanded to include inadequately covered codes (22). OSICS encompasses the full range of sports medicine diagnoses, adaptable to all sports and levels of competition, making it a preferable injury surveillance system (10).

The International Classification of Disease (ICD) has been used to classify disease for over a hundred years (15). Although previous research identified gaps in the sports and leisure injury codes (6), the expansive list of procedure codes allow for accurate documentation of treatment initiated by healthcare providers (19). In 1970, the Current Procedural Terminology (CPT) code system, broadened to include diagnostic and therapeutic procedures in surgery, medicine, and the specialties (22). The combined use of ICD and CPT codes expand the ability to code for medical, surgical, diagnostic, and therapeutic services (3,19). However, CPT codes are owned and maintained by the American Medical Association, therefore; they are exclusively used in the United States. ICD and CPT codes have potential to be used a universal language for medical documentation, injury surveillance, and communication among healthcare professionals internationally.

Currently, there are many different sports injury surveillance classification systems. However, each of these systems only focuses on their particular subset of patients and collects injury data based on the needs and concerns of the sports’ governing body. For example, the National Collegiate Athletic Association (NCAA) collects information on collegiate athletes in the United States, and the Fédération Internationale de Football Association (FIFA) collects information on international professional soccer players. Although the information from these surveillance systems is important for each individual organization, the information can only be used for that particular demographic, and is not generalizable. There is a lack of universality and currently none of the sports injury surveillance classification systems evaluate treatment method per pathology over time. Therefore, we created the OSICS 10.1 Plus, which combines the universality of OSICS 10.1 and CPT codes. The purpose of this study was to establish the level of usefulness and level of effectiveness of the OSICS 10.1 Plus for recording injuries and interventions as a comprehensive system that can be utilized regardless of setting. Level of effectiveness was assessed by the ability of OSICS 10.1 Plus to match the participant’s previous records and level of usefulness was assessed by the ease of use statements.

METHODS

Participants

Three hundred and forty-two (males=148, females=192, no response=2; age=30.9±9.5y; experience=9.1±10.5y) athletic trainers (ATs) in the United States completed the survey. Participants were primarily employed in the secondary school (n=135) or collegiate setting (n=171). A summary of the demographic information for the participants that completed the OSICS 10.1 Plus assessment is depicted in Table 1.

Table 1

Demographic Information of Participating Athletic Trainers.

Characteristic	Mean (S.D.)
Age	30.95±9.52
Sex	Male (n=148)Female (n=192)No Response (n=2)
Years Practicing	9.09y±10.46
Setting	Collegiate (n=171)Secondary School (n=135)Other (n=36)

Protocol

We completed an end-user study based on retrospective data. Participants collected the location of injury, severity of injury, and treatment pertaining to any injury that occurred within the last six months. The participants recorded the collected information into a web-based survey we created using Qualtrics Software (Qualtrics LLC, Provo, UT). The survey includes the Orchard Sports Injury Classification System 10.1, to collect information on the injury, and CPT codes to collect information on the treatment initiated, both of which are validated tools. At the end of the survey, the participants completed a questionnaire to analyze the usefulness, and level of agreement for OSICS 10.1 Plus in recording injuries and intervention methods. Level of effectiveness was assessed by the ability of OSICS 10.1 Plus to match the participant’s previous medical records and level of usefulness was assessed by the ease of use statements.

Statistical Analysis

We performed analyses of central tendency to evaluate the effectiveness, and usefulness of the OSICS 10.1 Plus. Grounded theory was utilized to develop the codes that were extracted from the qualitative feedback. Grounded theory is a systematic evaluation of data that uses codes to group common themes to serve as the basis for a new theoretical framework.

RESULTS

Participants indicated that the OSICS Plus was useful with scores indicating overall agreement to the statements: easy to use, easy to enter an injury, and easy to enter the associated interventions. Respondents indicated that OSICS 10.1 Plus was neither effective nor ineffective with scores indicating neutral agreement to the following statements: that the OSICS matched their current injury or intervention records. A majority of participants indicated that they could find the injury and interventions of interest. A majority of respondents indicated they would consider using OSICS 10.1 Plus for injury surveillance at their institution. The summary of the OSICS 10.1 Plus assessment by participants is depicted in Table 2.

Table 2

Level of Agreement Statements OSICS 10.1 Plus Assessment

Statements	Likert Scale
Easy to Use	4.1±0.7pts
Easy to enter an injury	4.1±0.8pts
Easy to enter interventions	3.9±0.8pts
OSICS 10.1 Plus matched their current injury records	3.5±1.0pts
OSICS matched their current intervention records	3.5±0.9pts
	Yes/Total (%)
Could find the injury	281/342 (82.2%)
Could find the intervention	225/342 (65.8%)
Consider using OSICS	205/342 (60.0%)

5 point Likert scale (1=strongly disagree, 5=strongly agree)

The feedback from the open-ended questions provided insight regarding the OSICS 10.1 Plus. Three major themes surfaced from the analysis of the feedback questions. These themes were 1) feedback that promoted the use of OSICS 10.1 Plus as an injury surveillance system, 2) feedback that did not support the use of OSICS 10.1 Plus, and 3) a general misunderstanding of documentation and injury surveillance by the respondent Athletic Trainers. Each of the three major themes can be broken down into two subthemes. The major themes and subthemes are depicted in Figure 1.

Figure 1

Emergent themes and subthemes from qualitative feedback.

The feedback from the open-ended questions provided supporting comments to promote the use of OSICS 10.1 Plus. Participating athletic trainers indicated that the feasibility of the system was the most useful aspect. Feedback revealed that the system was easy to use, convenient, organized, easy to navigate, and user friendly. When asked “compared to your current system, what did you like about OSICS?” respondents denoted that the system had a nice appearance, was streamlined in a logical order, and that the selecting options were preferable to the typing that was involved in their current system. The qualitative feedback was supported by the quantitative data as well. Respondents indicated that the OSICS 10.1 Plus is overall easy to use, easy to enter an injury, and easy to enter the associated interventions.

The second subtheme that developed was the potential OSICS 10.1 Plus had as an injury surveillance system and for use in the profession. For example, respondents said that it would be great if OSICS 10.1 Plus could be imbedded into their current electronic medical record (EMR). In addition, respondents indicated that with fine-tuning, the OSICS system and individual customization, the system would more widely appeal to them.

Conversely, several participants provided feedback on their experience with OSICS 10.1 Plus, which does not promote the use of the system in its current state. The first subtheme is the need for modification of the current format. Feedback indicated that participants had concerns about compound injuries because OSICS10.1 Plus, in its current format, only allows for the recording of singular injuries, rather than compound injuries such as a lateral ankle sprain and a fibular fracture.

Additional feedback revealed that some participants felt modifications needed to be made to the interface, including too many screens to load; the system was not navigational on mobile devices, no go back button or arrow, and a technical glitch that caused some sections to loop twice. However, most participants indicated if these issues were resolved than the system would be much more effective.

The second sub theme that emerged was lack of information, and that many additions need to be made to make the system more effective. Participants felt that the system was not for non-traditional settings and patients. Missing sports included Olympic sports, recreational activities, martial arts, performing arts, ice hockey and injuries that were out of season or not sport related. Additionally participants indicated missing interventions as well, such as referral, non-traditional medicine, varying concussion treatments, and emergency care. OSICS 10.1 Plus has preliminary support for the use as an injury surveillance system. With modification, respondents indicated that OSICS 10.1 Plus could be effective system. Additionally, the quantitative data seems to support the need for modifications and additions. Only 65.8% (225/342) of participants felt as though they could find the interventions they were looking for and 60.0% (205/342) would consider using OSICS 10.1 Plus.

The most prevalent, yet unexpected theme that emerged from the participants’ feedback was the apparent misunderstanding of 1) the difference between medical documentation and injury surveillance and 2) the use of CPT for billing for reimbursement. Feedback from participants indicated that they wanted the specificity of medical documentation such as progress notes, daily notes, specific parameters, etc.

DISCUSSION

Sports injury surveillance classification systems are designed to gather information on injury types, rates, mechanisms, and risk factors (11). The data from these surveillance systems allows for an objective assessment of the risks of sports participation and guide changes to rules, equipment, and injury prevention efforts (9,11). The effectiveness of a well-designed surveillance system is based on the following nine characteristics according to the World Health Organization: simplicity (12,7), flexibility (12,7), acceptability (12,7), reliability (12,7), utility (12), sustainability (12), timeliness (7), security (12), and confidentiality (12). The quantitative data and qualitative feedback indicate support for and potential use of the OSICS 10.1 Plus for simplicity, flexibility, acceptability, reliability, utility, and timeliness. Additional research in 2014, indicated that surveillance systems needed to be user-friendly and delivered in a range of accessible formats to increase reach, adoption, implementation, and maintenance (4).

Increasing the use of surveillance systems and injury surveillance data will increase our understanding of the extent of the injury problem (4). Unfortunately, the majority of sports injury surveillance systems exist in professional and elite sport settings, resulting in a knowledge gap about injuries in amateur and community sport settings (5). With minor modifications, OSICS 10.1 Plus could serve as an effective, useful, and comprehensive injury surveillance system to address this gap. In addition, recently published research revealed that future sports injury surveillance research should address potential discrepancies in reporting that are associated with participating ATs’ employment status (13). Full-time secondary school ATs reported higher overall injury rates and average numbers of athletic training services in high school football than their part-time counterparts (13). Thus there is the need for improved injury surveillance and an increase in full-time ATs in the high school setting (13).

Respondents reflected on the potential that the OSICS system could have on the Athletic Training profession, “This would be a great tool for tracking injures in order to show the need for Athletic Trainers based on the number and types of injuries occurring” and “It can be very beneficial to our profession and our institutions to utilize injury surveillance systems and analyze the data to incorporate injury prevention.”

The goal of the tool was to create a comprehensive injury and intervention recording system regardless of injury or setting. The creation of such system is essential to the advancement of injury surveillance. OSICS 10.1 Plus is designed to gather information on these injury types, rates, mechanisms, and risk factors in large populations of athletes. This information is essential to guide rules and equipment changes, and injury prevention efforts such as altered training regimens and interventional assessments. Uniquely, OSICS 10.1 Plus also allows for the objective assessment of interventions per pathology over time, which can potentially improve patient care.

Recent developments and publications in injury surveillance systems indicate that injury surveillance has improved, but still needs development. The Datalys Center which is responsible for the NCAA ISP, partnered with the National Athletic Trainers’ Association Research and Education Foundation to launch the National Athletic Treatment, Injury and Outcomes Network (NATION) Surveillance Program (SP) in high schools (2). Datalys Center has improved both NATION and its sister project, the NCAA ISP, by identifying previous issues with the ISP. The NATION-SP goal is to provide a comprehensive examination of both time loss and non-time loss injuries, sustained by student athletes that receive services from ATs working in the high school setting (2). However, two fundamental problems still exist 1) only 70% of US public high schools and 55% of public high school student-athletes had access to an AT (2), and 2) participation in data entry. The NCAA ISP has a long standing history for injury surveillance, however during the five academic years from 2009–10 through 2013–14 the participation among teams for the study period ranged from a low of 0.7% in men’s tennis to a high of 13.2% in men’s ice hockey (14). This is a fundamental problem because the majority of the injuries are not being recorded on injury surveillance systems, thus injury types, rates, mechanisms, and risk factors are not accurate.

OSICS 10.1 Plus provides a mechanism to promote the athletic training profession by tracking the interventions used, per pathology overtime to aid in the efforts to achieve reimbursement. However, this research indicates that there is an apparent misunderstanding of the use of CPT codes for reimbursement and with difference between medical documentation and injury surveillance. The NATA recognizes the difficult nature of reimbursement and using CPT codes. The 2011 NATA reimbursement manual states, “Coding is a complex subject that requires educating yourself or hiring a Certified Coder. Unless you hire a coder or billing agency you should purchase new coding books every year when they are updated” (17).

Prior to the start of the survey, the following statement was provided, to participants:

“The OSICS 10.1 Plus is NOT a medical documentation system. Sport injury surveillance systems are designed and developed to gather information on injury types, rates, mechanisms, and risk factors in large populations of athletes to allow for an objective assessment of the risks of sports participation. The information gathered from these injury surveillance studies are key to guiding rule and equipment changes, and injury prevention efforts such as altered training regimens and interventional assessments. Medical documentation on the other hand is much more detailed and serves an entirely different purpose. Medical documentation includes the observable, measurable, and reproducible findings from examination and diagnostic tests, and assessment. Injury surveillance and medical documentation are both important to medical professionals and the medical field, however this study is specifically looking into the use of OSICS 10.1 Plus as an injury surveillance system.”

However, participants still struggled to differentiate between medical documentation practices and the role of injury surveillance. This was particularly obvious in the lack of understanding about using CPT codes to bill for reimbursement. CPT codes are used for reimbursement of services and tend to be broad as compared to the specificity a medical professional is expected to have in their documentation. For example, CPT code 97032 is for electrical stimulation and CPT code 97110 is for therapeutic exercise. Each intervention would be billed in fifteen minute units, however, no additional details are included in the process to bill. A practitioner would be expected to document, in their medical records, the parameters of the electrical stimulation intervention as well as for the therapeutic exercise. This difference or misunderstanding was demonstrated in the feedback provided by the participants. Many participants wanted to include the specific parameters of their intervention (as they would in a medical note), however, those details are beyond the scope of OSICS 10.1 Plus and injury surveillance in general. OSICS 10.1 Plus was designed to objectively assess the use of interventions per pathology over time, not to detail each treatment and its parameters.

Additionally, future research on the implementation of EMR, and injury surveillance needs to be conducted with careful consideration (18). Research has indicated that lack of EMR adoption by clinicians was a result of systems development, implementation factor concerns, and organizational pressures (18). The lack of understanding demonstrated in this study may also be a factor for the lack of adoption and full utilization of EMR by ATs. There is a need for a conceptual framework to guide the implementation and evaluation of EMR from the perspectives of different stakeholders (18).

Electronic Medical Record (EMR) keeping, the most effective mechanism to engage in medical documentation, has been present in the medical field for years. Drastic uptake began in 2009, with the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009, which provided financial incentives to implement EMRs (8). Currently, the uptake of EMRs in athletic training facilities at colleges and universities, secondary schools, and professional sports is not described in the literature and their implementation rate is unknown (1). The advancement of technology and the sports medicine profession has led to development and advancement of computer and web-based medical record systems. These advantages of EMR include portability, ease of access, control of access, and adaptability (16). Although there are several different systems available, there needs to be a standardization of the data and a uniform documentation system (16). The standardization of data would allow for the exchange of information between systems, analyses across teams at varying levels of play, improved information flow, and analysis of injury patterns with the goal of preventing or reducing injuries (16).

In conclusion, a comprehensive surveillance system is needed to record injuries and intervention methods regardless of setting. The information gathered by a comprehensive system allows us to assess the risks of sports participation and guide changes to rules, equipment, and injury prevention efforts. The use of OSICS 10.1 Plus as an injury and intervention tracking system has potential to promote the employment of ATs by demonstrating a need for patient care and assist in reimbursement efforts by accurately monitoring interventions. The results of this end-user study indicate that participants agree that OSICS 10.1 Plus is useful and could serve as an effective mechanism for injury surveillance with minor modifications. However, this study also indicates that we, as ATs, need to work toward better understanding regular medical documentation and reimbursement first, as we continue to develop easy to use and effective injury surveillance tools.