A Multistep Maturity Model for the Implementation of Electronic and Computable Diagnostic Clinical Prediction Rules (eCPRs).

INTRODUCTION: The use of Clinical Prediction Rules (CPRs) has been advocated as one way of implementing actionable evidence-based rules in clinical practice. The current highly manual nature of deriving CPRs makes them difficult to use and maintain. Addressing the known limitations of CPRs requires implementing more flexible and dynamic models of CPR development. We describe the application of Information and Communication Technology (ICT) to provide a platform for the derivation and dissemination of CPRs derived through analysis and continual learning from electronic patient data. MODEL COMPONENTS: We propose a multistep maturity model for constructing electronic and computable CPRs (eCPRs). The model has six levels - from the lowest level of CPR maturity (literaturebased CPRs) to a fully electronic and computable service-oriented model of CPRs that are sensitive to specific demographic patient populations. We describe examples of implementations of the core model components - focusing on CPR representation, interoperability, electronic dissemination, CPR learning, and user interface requirements.
CONCLUSION: The traditional focus on derivation and narrow validation of CPRs has severely limited their wider acceptance. The evolution and maturity model described here outlines a progression toward eCPRs consistent with the vision of a learning health system (LHS) - using central repositories of CPR knowledge, accessible open standards, and generalizable models to avoid repetition of previous work. This is useful for developing more ambitious strategies to address limitations of the traditional CPR development life cycle. The model described here is a starting point for promoting discussion about what a more dynamic CPR development process should look like.

Introduction

Clinical Prediction Rules as a Form of Evidence-Based Medicine (EBM)

Evidence-based medicine (EBM) has long been advocated as one way of supporting diagnostic reasoning that is based on a more rigorous and systematic approach.1–2

One form of EBM is the Clinical Prediction Rule (CPR).3–5 CPRs are typically derived through conducting manually intensive observational studies that elicit quantified epidemiological associations using statistical or probabilistic techniques.5 The current highly manual nature of deriving CPRs also makes them difficult to use and maintain.6 With some exceptions the format for CPRs dissemination has traditionally been literature based, putting an onus on clinicians to search literature for suitable CPRs.7

Addressing the known limitations of CPRs requires implementing more flexible and dynamic models of CPR development. We describe the application of Information and Communication Technology (ICT) to provide a platform for derivation and dissemination of CPRs derived through analysis and continual learning from electronic patient data. We present an incremental model of CPR development that is of interest to those in the clinical research community advocating wider use of evidence-based CPRs through translational research, and to those designing or implementing service-oriented, rule-based decision support systems (DSS).

Limitations of Traditional CPR Development

Despite the existence of an accepted development methodology for producing CPRs (Table 1), the development of many CPRs focuses on the derivation phase (Level 1) of the CPR life cycle,8 lacking subsequent validation (Levels 2 and 3) and impact analysis (Level 4).9

Table 1.

Accepted CPR Development Methodology

CPR CATEGORY	LEVEL OF EVIDENCE REQUIRED
Derivation (Level 1)	Factors with predictive power are identified in order to base the rule on a derivation patient population.
Narrow validation (Level 2)	The rule is applied to a different patient set with characteristics similar to the original derivation population.
Broad validation (Level 3)	The rule is applied to another population with different characteristics from the original derivation population.
Impact analysis (Level 4)	The impact of the rule may be tested and assessed in terms of its effect on clinical outcomes, physician behavior, or costs.

This lack of validation for many CPRs limits their applicability to the same patient populations used for the original derivation. Scores may vary when the CPR is applied to populations with gender, age, or clinical settings that are different from the original derivation population. The Alvarado score, for example, has been found to perform best in adult male populations.10 This has implications for the applicability of any published CPR as changes take place over time in the demographics of the original rule-derivation study population.

Existing Implementations of CPRs in Decision Support Systems (DSSs)

There are a number of additional barriers to consider when implementing CPRs electronically as part of DSSs. Previous attempts to deploy CPRs highlight additional issues to consider in implementing clinical DSSs, including the following:

Validation and impact analysis of CPRs is restricted due to lack of connectivity to wider patient populations beyond the original electronic tools to which they are initially deployed and tied.11–13

With some exceptions where evidence is disseminated using open standards, and separate from the application itself,14–19 decision support tools are tied to specific proprietary clinical systems, which lack support for wider Electronic Health Record (EHR) workflow integration across other systems.

Implementations of decision support tools focus on individual CPR models and are not easily portable to implement other CPRs, resulting in redevelopment efforts for each implemented rule.11–13,20–26

Successful implementations and clinical acceptance of the deployed tools necessitate a collaborative multidisciplinary approach to define the nature of the intervention required and the actual workflow of the CPR in practice.11,21–23

These limitations can be considered part of a wider problem of successfully translating clinical research knowledge into clinical practice using ICT tools.

CPRs and the Learning Health System (LHS)

Rapidly translating clinical knowledge into practice is a core objective of the learning health system (LHS). The current traditional model of CPR development is considered to be at a low level of technological development with respect to what has been termed “the pyramid of evidence.”27–28 Research initiatives have defined what should constitute the core components of the LHS.29–30 Within a virtuous cycle of health improvement a number of important requirements have been identified to support this knowledge translation capability, including the following:

Generating valid clinical knowledge;

Packaging and curating knowledge so it is widely accessible and actionable, and putting knowledge to use to effect change;

Developing meaningful use of the EHR to support diagnostic and therapeutic support based on evidence;

Developing a computable representation of research evidence and making that available to EHR systems as a Web service; and

Developing a means of providing diagnostic or therapeutic prompts within an EHR that works across a variety of EHR systems.30

We propose the implementation of LHS knowledge-translation capabilities in the constructing of electronic and computable CPRs (eCPRs). The eCPR can be considered an evolution of the traditional CPR development methodology that moves CPR development toward the top of the traditional “pyramid of evidence.” It provides for the electronic derivation and dissemination of CPRs that are computable, updateable, and versionable based on continuous learning obtained from analysis of underlying derivation data. This platform implements model-based and service-oriented architectures, using open interoperability standards to exploit the potential of data mining of aggregated sources of EHRs for CPR development.

Components of the eCPR Maturity Model

The multistep maturity model for eCPR implementation consists of six incremental levels, as shown in Figure 1 and described below. The model can be used to assess the current level of development for an organization using CPRs, and how the organization might develop it further.

Figure 1.

A Multistep Maturity Model for eCPRs

Each model level also describes interoperability characteristics that it supports. The definition of interoperability we are using is as provided by the Office of the National Coordinator for Health IT.31 This definition describes four interoperability layers:

Syntax: content and structure;

Semantics: vocabulary and code – sets and terminology;

Transport: method by which information is moved from system to system; and

Services: the infrastructure components deployed and used to accomplish specific information exchange objectives.

Level 1: Literature-Based CPRs

Interoperability Layers: Not interoperable – stand-alone tool.

A literature search of CPRs may identify CPRs that are potentially useful in the particular clinical environment in which they are to be employed. This may involve developing an electronic query-based search strategy to identify candidate CPRs for further consideration. The identified literature-based CPRs provide the starting point for developing subsequent electronic CPRs deployed as decision support tools. One such systematic review of published literature identified CPRs specifically relevant to the family practice setting. Almost 800 published papers were identified, indicating the increasing level of research and interest in this area.32

Level 2: Electronic Document-Based CPRs

Interoperability Layers: Not interoperable – stand-alone tool.

An improvement on the traditional literature-based CPR is to provide electronic document-based equivalents. These CPRs are not interoperable with other clinical systems in themselves but are documented as part of a collected, searchable register of rules in an electronic format with appropriate Web links to the original document-based sources. This is with a view to overcoming one of the initial difficulties in using CPRs by allowing more user friendly searching and identification of appropriate CPRs for any presenting patient complaint, as shown in Figure 2. These search capabilities include searching based on the life cycle stage of the CPR, the clinical domain or condition targeted by the rule, and the clinical settings in which it is suitable for deployment.33

Figure 2.

A Search Result from a Web-Based Register of CPRs (Example of Level 2 Electronic Document-Based CPR)

Level 3: Electronic Computable Individual CPR Tool

Interoperability Layers: syntax – Stand-alone tool, potential to integrate within a single organization EHR.

The majority of decision support tools implementing CPRs are at Level 3 of the model.11–13,20–26 Level 3 implements specific literature-based CPRs in decision support tools used at point of care in clinical practice. The representation of the rule is specific to the CPRs used and limited to use within the information systems in which they are deployed and tested. There may be some integration of the tools with another single organizational EHR and associated patient populations (a narrow CPR validation in practice). An important improvement is the wider dissemination of CPRs into clinical practice. The rule can be deployed electronically in a controlled clinical environment and made available to support subsequent validation and impact analysis efforts. This may take the form of randomized control trials testing the effectiveness of the electronic tool versus the performance of a control group without access to the tool.11

Level 4: Service-Oriented Generalized CPR

Interoperability Layers: syntax, transport – Interoperable and reusable with many different clinical applications within a single organization using open interoperability standards; lack terminology integration allowing access from other external systems that use different clinical coding schemes.

The wider scale reuse of computable CPRs beyond their initial development environments may be achieved through a service-oriented architecture of CPR resources. This service-based approach has been increasingly deployed in DSSs as a means of promoting reuse of evidence and reducing development effort.14–19

Broad validation of CPRs becomes possible when CPRs that were originally developed for use by an individual hospital department or family practice are reused as evidence to support wider dissemination, validation, and impact analysis in tools developed for other patient populations. This decouples the provision and querying of CPRs from the original deployment applications that use them. Evidence is accessed through widely used open standards from any development environment, thus supporting easier workflow integration.

The flexibility of such a service also depends on implementing a CPR model that captures computable structures common to all CPRs. This provides a model that can be used to deploy CPRs in a computable format that is accessible using open standards by third party tools via a Web service.34–36 An example of a general model of CPRs that has been implemented is shown in Figure 3. The general model components captured in the model include the following:

Figure 3.

A General Model of CPR Structure (Example of Level 4 Generalized CPR)

A presenting patient problem or reason for encounter (RFE);

One or several differential diagnoses to consider associated with the RFE;

A clinical prediction rule associated with a particular diagnosis;

The CPR rule elements comprising of diagnostic cues and associated criteria to be checked, with a score that quantifies the significance of the rule element to the clinical outcome;

A threshold-based score scheme that interprets the CPR possible score as risk bands with optional clinical action or recommendation to be carried out in response to the interpreted risk-based score bands;

The demographic context of the derivation population from which the cue scores were derived for use by a CPR; and

A standard code binding or clinical vocabulary term associated with each RFE, diagnosis, or diagnostic cue.

The general model of CPRs is then made available as a REST-based Web service that can be queried from any third-party development environment using open standards including XML, REST, and JSON.37–39 An example of a REST-based query from a decision support tool for a computable representation of the Alvarado Score is accessible using: http://localhost:8080/ClinicalEvidenceRESTService/interfaces/query/cprs/AlvaradoScore1_0.

The XML output using the model is generated and returned to the call decision support tool application as shown in Figure 4.

Figure 4.

A Web Service-Based Call for Details of the Alvarado Score (Example of Level 4 Service-Oriented Generalized CPR)

Level 5: CPRs with Terminology Services Integration

Interoperability Layers: semantics – Semantically interoperable with many different ICT applications across multiple organizations through addition of standard clinical code bindings.

The importance of integrating DSS tools into the wider clinical workflow has been highlighted as a key factor for their broader acceptance and implementation success.40 The capability for binding individual CPR-model terms with several clinical terminologies and vocabularies to support wider semantic interoperability and broader uptake of CPRs is crucial.41 This may be supported through providing the service-based CPR models in conjunction with clinical terminology or vocabulary services that enable terminology lookup, binding, and mapping of models to different vocabularies.42

Integrating DSS tools with EHR systems based on coded patient data helps in identifying workflow related patient events that can be used as a contextual trigger for initiating diagnostic CPRs as a form of decision support. In addition the patient record data itself can then be utilized to provide patient demographics or patient historical data that may be used to contextualize CPR execution and selection of suitable scoring schemes based on the context of the particular patient.

As an example we have added code bindings supporting National Health Service (NHS) read codes widely used by EHR systems in the United Kingdom.43 Multiple code binding types can be added for each CPR cue to support other coding schemes used in other countries. An example of the output of one CPR cue element with code bindings for the “nausea” element of the Alvarado Score is shown in Figure 5. Where multiple patient codes may be suitable for triggering a cue, the “isPrimary” tag denotes the primary code and text to use for display in applications.

Figure 5.

A Web Service-Based Call to Alvarado Score with Code Bindings (Example of Level 5 CPRs with Terminology Services Integration)

Level 6: Learning, Versionable CPR

Interoperability Layers: services – Interoperable with many different ICT applications across multiple organizations; capable of deriving CPRs electronically.

The development and continuous analysis of aggregated sources of electronic patient data to facilitate evidence generation and learning is a crucial part of the broader LHS vision. A number of existing aggregated sources of patient data are found at local and national levels.44–49 These contain large amounts of longitudinal population-health data suitable for data mining or statistical analysis with a view to deriving actionable knowledge.

The potential for using such data sources for data mining has been demonstrated in the TRANSFoRm project that utilized aggregated sources of European primary care data provided by the Transition project.49–51 An open source data-mining tool called KNIME was used to produce quantified association rule combinations describing the relationships identified between ICPC2 coded diagnostic cues, demographic variables, and diagnostic outcomes from the aggregated data sources.52–53 This process provided empirically quantified diagnostic associations using calculated likelihood ratios.54–55

An example of a CPR construction tool creating a data-mined CPR for diagnosis of Urinary Tract Infection is shown in Figure 6. The tool presents data-mined evidence (left-hand side) through the Web service to construct versioned CPRs using the recognized formal CPR structure described in Level 3 (right-hand side). This allows for definition of normalized scoring schemes based on threshold approaches to decision-making. The score schemes risk levels and associated actions are defined through manual clinical review and interpretation of the general evidence and associated quality measures.

Figure 6.

A CPR Construction Tool Based on Data-Mined Evidence (Example of Level 6 Learning, Versionable CPR)

Saving the constructed CPR makes it available through the Web service, and it can be accessed using a standard Web-based call from other applications: http://localhost:8080/ClinicalEvidenceRESTService/interfaces/query/cprs/DataMinedUTIRule.

The XML output of the call is shown in Figure 7.

Figure 7.

A Web Service Call to a Data-Mined CPR (Example of Level 6 Learning, Versionable CPR)

Putting all the levels of the model together we can illustrate an architecture for electronic derivation of CPR evidence as shown in Figure 8.

Figure 8.

Summary of Electronic Derivation and Deployment of CPRs (Example of Level 6 Learning, Versionable CPR)

Tracking of CPR versioning, change control, and usage will mean that in practice there should be at least two deployment environments such as a “development” and a “live” production CPR service. This can facilitate deployment of CPRs through the service for restricted narrow validation, and then promotion to wider usage for wider scale broad validation and impact analysis.

Clinical Workflow and User Interface Integration Considerations

The integration of eCPRs into existing electronic clinical care systems is crucial to their wider usage, but there are more factors to consider than simply the technical ones. User interface design considerations are also important to promoting the development and use of eCPRs more broadly in care settings. Studies have demonstrated the importance of consulting end-users regarding integration of eCPRs with existing systems.26 Another study that examined the deployment of EHR systems across the United Kingdom stressed the importance of “soft skills” such as training and multidisciplinary teams as being key to the uptake and usage of electronic clinical systems.56

The type of CPR being developed should also be considered. CPRs may be related to diagnostic or prognostic outcomes. A diagnostic CPR estimates the probability or risk of the presence or absence of a disease at a fixed point in time, for a specific individual.10 A prognostic CPR is more complex, having an additional temporal aspect after the prognostic prediction has been made. It requires follow-up to see if a particular clinical event relating to the prognosis transpires at some defined subsequent point in time.57 The workflow implications are less complex for embedding diagnostic CPRs within EHRs or decision support applications, since the diagnostic CPR can be event driven and can trigger a recommendation made and recorded at a fixed point in time without the need for future follow-up.

The models described here focus on implementing diagnostic CPRs, and they are our primary examples. On that basis, interoperability considerations have been a core focus for development of this model before considering more complex time-dependent workflow integration.

Conclusion

The traditional focus on derivation and narrow validation of CPRs has severely limited their wider acceptance. The evolution and maturity model described here outlines a progression toward eCPRs achieving the vision of an LHS. The model provides an incremental framework consistent with the wider goals of the LHS and which demonstrates how we can consolidate work done by others in order to achieve this – using central repositories of CPR knowledge, accessible open standards, and generalizable models to avoid repetition of work. This is useful for developing more ambitious strategies to address limitations of the traditional CPR development life cycle, with a view to enabling wider implementation and acceptance of the benefits that intelligent use of CPRs can provide.

The model as presented here has limitations and can address only some of the issues with CPR development. For example, it does not address the design and type of clinical interventions to which any particular diagnostic CPR is best applied. It does not address what data elements may actually be available in a target EHR system to trigger event-driven use of our models. The wider availability of large volumes of aggregated data sources to support data mining approaches may be currently feasible only in limited cases.

It is time to look again at how we develop, disseminate, and test CPRs in clinical practice. The model is a starting point in promoting discussion about what a more dynamic CPR development process should look like.