Collecting data on organizational structures of trauma centers: the CAFE web service.

BACKGROUND: During the past several decades, the American College of Surgeons has led efforts to standardize trauma care through their trauma center verification process and Trauma Quality Improvement Program. Despite these endeavors, great variability remains among trauma centers functioning at the same level. Little research has been conducted on the correlation between trauma center organizational structure and patient outcomes. We are attempting to close this knowledge gap with the Comparative Assessment Framework for Environments of Trauma Care (CAFE) project.
METHODS: Our first action was to establish a shared terminology that we then used to build the Ontology of Organizational Structures of Trauma centers and Trauma systems (OOSTT). OOSTT underpins the web-based CAFE questionnaire that collects detailed information on the particular organizational attributes of trauma centers and trauma systems. This tool allows users to compare their organizations to an aggregate of other organizations of the same type, while collecting their data.
RESULTS: In collaboration with the American College of Surgeons Committee on Trauma, we tested the system by entering data from three trauma centers and four trauma systems. We also tested retrieval of answers to competency questions. DISCUSSION: The data we gather will be made available to public health and implementation science researchers using visualizations. In the next phase of our project, we plan to link the gathered data about trauma center attributes to clinical outcomes. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Beginning in the 1970s, the American College of Surgeons (ACS) worked to establish and refine criteria that set standards for trauma center organization, professional care provider qualifications, and performance.1 Until the last several years, these standards focused largely on measures of structure and process because only recently has it become possible to obtain reliable outcome data. The most recent ACS standards for trauma center verification were published in Resources for the Optimal Care of the Injured Patient in 2014.1 In 2008, the ACS established the Trauma Quality Improvement Program (TQIP) to build a process for trauma centers to measure and compare their risk-adjusted patient outcomes to similar organizations.2 Though TQIP has evolved and matured during the last decade, at this time, the correlation between organizational structure and patient outcomes is not yet defined.3–6 Further, beyond the standards themselves, the ways in which the particular attributes of trauma centers, and, on a larger scale, of regional trauma systems, contribute to optimal patient outcomes have not yet been identified or measured.6 7

The TQIP efforts and the ACS trauma center verification standards paved the way for more rigorous research on the implementation of trauma centers and quality control. However, a gap remains between linking specific organizational structures to specific patient outcomes. Combining TQIP data with participation in a quality collaborative leads to significant improvement in patient outcomes compared with benchmarking alone.2 This suggests that only when institutional practice and resources are compared do centers obtain the maximum benefit from risk-adjusted data. As a result, institutional benchmarking alone does not identify the procedural or structural changes that are most likely to positively affect patient outcomes.

A major gap is created by the lack of a controlled vocabulary and well-developed methodologies to collect information about trauma center attributes that contribute to better patient outcomes. This gap contributes to the fact that great variability in risk-adjusted mortality rates exists even among trauma centers functioning at a similar level.8–10 In designated level 1 and 2 trauma centers in a single state, the variability in the risk-adjusted OR of survival for each trauma center when compared with the best trauma center ranged from 0.2 (significantly different) to 0.9 (not significantly different).11 Even while comparing only level 1 trauma programs, the variability ranged from 0.4 (significantly worse) to 0.9 (no difference), suggesting that a significant difference in mortality is present. In Ohio, mortality varied from 3.8% to 24.2% across level 1 trauma centers despite similar patient characteristics and injury severity.12 Nationally, the median incidence of survival after trauma ranges from 52.6% to 87.3% depending on the region of the country where injured.13 These data support a wide variation of survival after trauma.

Consider a hospital wanting to achieve better outcomes for injured patients in a developing nation with limited resources. It might be possible for the hospital’s administrative team to identify several trauma centers that have reported improving their patient outcomes as part of the TQIP effort. Achieving high-performance measures is an important goal, but the developing hospital cannot determine which trauma center attributes support the attainment of the high-performance standard.2 If the evolving institution has to choose between a major equipment purchase, operating suite upgrades, or changes in staffing patterns for physicians and nurses, to name a few, which choice might lead to the greatest return on investment? The same questions apply to remote and rural areas of North America and to more urban environments where some proliferation and duplication of trauma care resources have been questioned.6 14–16

Comparative Assessment Framework for Environments of Trauma Care (CAFE) is a National Institutes of Health-funded project (R01GM111324) that aims to build a web-based system that collects detailed information on the particular organizational attributes of trauma centers and trauma systems to conduct an anonymous self-assessment of the organizational structures of such institutions. The future plan is to link the gathered data about trauma center attributes to clinical outcomes. Once this is complete, it will be possible to identify attributes of trauma centers and trauma systems that strongly correlate to patient outcomes.17

One CAFE use case is to enable the comparison of the organizational structures in one evolving institution aiming for trauma center verification with the organizational structure of verified trauma centers. To facilitate that, CAFE provides a common terminology covering all relevant aspects of trauma center and trauma system management.18

The purpose of this article is to introduce the trauma surgery community and trauma system community to our work conducted in collaboration with the ACS. We give an overview of the CAFE project, describe the methodologies underlying our data collection tools and the CAFE questionnaires, and describe the real-time comparison functionalities. The CAFE web services are open and freely available to the entire trauma care community both within the USA and internationally. In addition, this article reports on the outcome of the initial data collection and beta testing which led to significant improvement of the CAFE questionnaires.

Methods

The CAFE web-based infrastructure serves two main functionalities (figure 1):

Figure 1

The CAFE web service. Representatives of trauma centers or trauma systems use the questionnaire to enter data that are stored in the CAFE RDF triple store. During the data entry process, respondents receive feedback on how their institution compares to other institutions of the same type. These data are retrieved from the CAFE RDF triple store. All CAFE RDF data will be made available to implementation science and public health researchers. CAFE, Comparative Assessment Framework for Environments of Trauma Care; OOSTT, Ontology of Organizational Structures of Trauma centers and Trauma systems; RDF, Resource Description Framework.

Allow representatives of trauma centers and trauma systems to enter information about their institution and receive a real-time comparison with the organizational structure of other institutions of the same type.

Provide public health and implementation science researchers with data on organizational structures of trauma centers and trauma systems.

We addressed the second functionality by creating prototype visualizations which were evaluated during a focus group of trauma surgeons and trauma researchers at the 2018 ACS Clinical Congress in Boston. The visualizations have been refined based on the input from the focus group. The focus group also provided us with a list of research questions that potential users may wish to answer employing the data.

In conceiving the CAFE project, one requirement was the ability to create flexible categories based on multiple organizational attributes and to program the CAFE system to automatically sort individual institutions or components into those categories. Utecht et al have demonstrated that Semantic Web Technologies (SWT) are a successful strategy for automatic data classification in sorting evidence types.19 Based on these considerations, we decided to build the CAFE data management system using SWT.

SWT are methodologies that facilitate linked data and support running automated inferences on data. A core of SWT is the Resource Description Framework (RDF), a knowledge representation language. RDF consists of subject-predicate-object three-part statements (triples). RDF allows assigning a unique identifier to each entity in the domain of discourse. This permits one entity to be either the subject or the object in multiple triples. In effect, RDF enables the building of complex graphs out of the simple subject, predicate, and object structure. These graphs composed of triples are retained in RDF triple stores. The CAFE triple store is the central knowledge base component for this project.20

To enable the full potential of automatic inference, the RDF triple store must contain background information about the organizational components and attributes relevant to the domain of trauma care and management. In an SWT environment this type of information is encoded in ontologies that are loaded into the RDF triple store along with the particular data to be analyzed. An ontology is a representation that is based on the real-world entities, processes, and relations of a given domain, rather than data. It represents the types of things in a given domain and how they are related to each other. For the domain of trauma centers, trauma systems, their organizational structures and processes, we created the Ontology of Organizational Structures of Trauma centers and Trauma systems (OOSTT) specifically for the CAFE project.21

For the stage of the CAFE project discussed in this article, we used the SWT components in the following ways, which will be described in more detail below:

Retrieving definitions from OOSTT for display in the questionnaire.

Using semantic schemata to transform all answers into RDF triples.

Loading the RDF triples into the CAFE RDF triple store.

Extracting data from the CAFE RDF triple store to create figures for real-time comparative assessment.

The key data collection tool of the CAFE project is the CAFE questionnaire (http://cafe-trauma.com/). Its purpose is to gather data about organizational structures of trauma centers and trauma systems while providing real-time comparison to those who enter the data. Data entry occurs through a response to specific questions about the trauma center’s structures and processes. Rolling the cursor over a bold-faced term brings up the definition used to guide respondents in consistent interpretation. The definitions are automatically retrieved from OOSTT’s underlying controlled vocabulary that was specifically created for CAFE.22 The terms for the controlled vocabulary were largely derived from the ACS Pre-Review Questionnaires (PRQ) used for trauma center verification and for trauma system consultations. Definitions were developed by a group of trauma system experts and then sent out for review to a group of trauma surgeons and subject matter experts. These carefully developed definitions enable a comparison of the user’s trauma center or system to all other trauma centers or systems that have entered the data using the same definitions to identify structures and processes.

The trauma center questionnaire includes the following sections: basic trauma program information, trauma center personnel and their responsibilities, and specialty services (figure 2). Types of question responses include yes/no, multiple selection, and a drop-down box with response choices. As data are entered, the database builds the description of the trauma center in an RDF computer-interpretable language that will enable subtle similarities and differences between trauma centers to be distinguished. RDF statements created by the questionnaire tool are stored real time in the CAFE RDF triple store.23 To facilitate the progress being saved and general logistics of the questionnaire process, the answers are also recorded in a PostgreSQL database.24 Answering a question in the CAFE questionnaire creates one or more triples in the CAFE RDF triple store. In figure 3, ‘_:x’ is the object of one triple and the subject of five potential triples, only one of which is actually created based on the respondent’s answer.

Figure 2

The CAFE questionnaire. The CAFE questionnaire is composed of Boolean (yes/no), multiple-choice (drop-down menu), or multiple selection (click boxes) questions. To facilitate orientation, the questionnaire is divided into three chapters with a total of 13 subsections. The right side of the screen shows the comparison of the answers given to other institutions of the same type that are already in the RDF triple store. CAFE, Comparative Assessment Framework for Environments of Trauma Care; RDF, Resource Description Framework; N/A, not applicable; TMD, trauma medical director.

Figure 3

RDF statements created by answering the question ‘Who does the trauma medical director (TMD) report to?’ Answer choices are: ‘Hospital Board’, ‘Chief Executive Officer’, ‘Chief Medical Officer’, and ‘Head of Clinical Services’. This is a multiselection item, so the user may select multiple options. RDF, Resource Description Framework.

The resulting information is pulled from the CAFE RDF triple store and used to build the comparison with other trauma centers in real time. The trauma center of interest will be compared with response averages from all trauma center respondents (figure 2). Data are deidentified when entered, so it is not possible to compare a trauma center with another specific trauma center.

The CAFE RDF triple store was built using the RDF4J triple store.25 Utecht and Brochhausen have measured the usability of multiple RDF triple stores based on the CAFE requirements and used the results to make their selection.26

Based on our goal of motivating potential users to share information about their institution, it is clear that the system needed to begin with an initial data set that could not be acquired from users. The American College of Surgeons Committee on Trauma (ACS-COT) facilitated this process. The ACS-COT received consent from three trauma centers and four trauma systems to use data from their trauma center verification reports or trauma system consultation reports to complete the questionnaire and to store and share the data in a deidentified manner in the CAFE system. Seven ACS staff members used the live questionnaire to enter data from the reports, providing us with feedback on the questionnaire and its functionalities.

Results

The data entry effort conducted by the ACS was successful and generated the CAFE RDF triple store holding a total of 2194 triples, which describe the organizational aspects of three trauma centers and four trauma systems.

To confirm that the data management system is operational, we ran a set of test queries to ensure that the queries retrieved the correct answers. Queries were run to retrieve the answers to the following questions:

To whom does the trauma medical director (TMD) report?

At which level are the trauma centers verified?

How many certified registered nurse anesthetists (CRNA) are employed by each of the hospitals?

How many CRNAs are employed by each hospital that has an affiliated medical school?

The results affirmed that we are able to correctly retrieve the information entered into the system by the ACS staff members.

ACS staff provided feedback on the organization of the questions in the questionnaire and the definitions, which are pulled from OOSTT and provided as pop-ups during the questionnaire process. We revised the organization of the questionnaire and added more categories of questions to make the individual categories smaller and more consistent. To improve the definitions guiding the correct interpretation of trauma center structures and process, we initiated a thorough review of the definitions in OOSTT, relying on domain expert feedback. In total, our 142 terms and definitions received 1197 reviews. That response rate allowed us to have, on average, eight different people review each term and definition. Of the 1197 individual reviews, 986 considered the term and definition suitable, meaning that 82.37% of our definitions were deemed suitable. Definitions that were not deemed suitable have been edited.

Next steps and discussion

Table 1 gives an overview of the next steps for the CAFE project and projected tasks for a renewal that was recently submitted.

Table 1

Next steps

Task	Timeline
Conduct a usability study on the CAFE web service to test how users regard the performance of the system as a whole.	Summer/Fall 2020
Finalize a web service that allows researchers interested in health outcomes or implementation science in trauma care to access the organizational data collected as part of the CAFE framework.	Fall/Winter 2020
Link CAFE data elements to trauma patient outcomes.	2021, after end of current CAFE project period(funding proposal submitted March 2020)

CAFE, Comparative Assessment Framework for Environments of Trauma Care.

One concern is who within a trauma center or trauma system would be responsible for entering data into the CAFE questionnaire. At the most basic level, answering the questions requires access to a large amount of data that must be obtained from disparate sources. At a more subtle level, the experience, perception, and level of engagement of the person answering the questions may affect the answers, as many require some nuance of interpretation. We assume that most of the data are readily available at trauma centers that have undergone ACS verification or trauma systems that have received ACS consultations. This is because the CAFE questionnaire and its terms were derived largely from information requested in the ACS trauma center verification process PRQ. In practice, the completion of the PRQ is often the responsibility of the trauma program manager (TPM) and the TMD, but is frequently a collaborative process involving many stakeholders from various areas in the hospital. In analogous fashion, the TPM and TMD may take responsibility for entering data about their trauma center’s structure and process into the CAFE questionnaire, but the responses may be more truly reflective if the input of a broader group is sought. The same process may be used by the TPM of a regional or state trauma system in collaboration with the system TMD.

Since the CAFE questionnaire for trauma centers is based on the PRQ it would make sense to incorporate the CAFE questionnaire as part of the PRQ process rather than creating a redundancy. At over 300 questions, the PRQ is more extensive than the current version of the CAFE trauma center questionnaire (102 questions). Currently, the CAFE questionnaire includes only questions that are relevant to organizational structure, that do not identify the organization completing the questionnaire (eg, employee names, zip codes), and that do not have extensive text fields as answers. The major advantage of the CAFE questionnaire is that it collects data in a computer-parsable way by using a knowledge representation language. This allows the computer to draw inferences without human intervention and to manipulate the data immediately. The decision to implement an SWT-driven component of the PRQ process lies with the organizations and agency responsible for trauma center assessment and verification.

At a future time, the CAFE project hopes to continue its collaboration with the ACS to link this database to outcome data, such as the TQIP database or another risk-adjusted benchmarking database. Once such a linkage occurs, the CAFE project will help fulfill the promise of investigating which trauma center or trauma system organizational structures and processes are most essential for certain patient outcomes.7 27

Conclusion

Based on the reported effort, we conclude that the CAFE questionnaire is ready to be used as a data collection tool to acquire organizational information about trauma centers and trauma systems. Visualization of comparative data supports a real-time assessment and comparison of the organizational structures of the participating trauma center or trauma system with organizations of the same kind that are already in the database. The collected data are well curated and maintained, and data can be obtained from the CAFE RDF triple store by running queries.