Simulation modeling validity and utility in colorectal cancer screening delivery: A systematic review.

OBJECTIVE: This study sought to assess the impact and validity of simulation modeling in informing decision making in a complex area of healthcare delivery: colorectal cancer (CRC) screening.
MATERIALS AND METHODS: We searched 10 electronic databases for English-language articles published between January 1, 2008, and March 1, 2019, that described the development of a simulation model with a focus on average-risk CRC screening delivery. Included articles were reviewed for evidence that the model was validated, and provided real or potential contribution to informed decision making using the GRADE EtD (Grading of Recommendations Assessment, Development, and Evaluation Evidence to Decision) framework.
RESULTS: A total of 43 studies met criteria. The majority used Markov modeling (n = 31 [72%]) and sought to determine cost-effectiveness, compare screening modalities, or assess effectiveness of screening. No study reported full model validation and only (58%) reported conducting any validation. Majority of models were developed to address a specific health systems or policy question; few articles report the model's impact on this decision (n = 39 [91%] vs. n = 5 [12%]). Overall, models provided evidence relevant to every element important to decision makers as outlined in the GRADE EtD framework. DISCUSSION AND
CONCLUSION: Simulation modeling contributes evidence that is considered valuable to decision making in CRC screening delivery, particularly in assessing cost-effectiveness and comparing screening modalities. However, the actual impact on decisions and validity of models is lacking in the literature. Greater validity testing, impact assessment, and standardized reporting of both is needed to understand and demonstrate the reliability and utility of simulation modeling.

INTRODUCTION

Background and significance

A simulation model is a computer-generated representation of a real-world system or process used to analyze the evolving behavior of a system over time or to predict results of various “what if” scenarios. Using trial data, population statistics, and probability estimates, simulation models have been developed in a broad range of areas in health care. They have been used to predict outcomes, resource requirements, feasibility, and costs of proposed interventions. Simulation models offer an invaluable decision aid for policymakers and healthcare leaders in a variety of healthcare applications, including colorectal cancer (CRC) screening. CRC screening uses fecal tests, diagnostic imaging, or endoscopic examination to assess for possible CRC among asymptomatic individuals at increased risk of developing CRC. Screening has been shown to be effective at detecting CRC earlier, as well as at reducing CRC incidence, morbidity, and mortality. There are many variations for CRC screening protocols with different modalities (ie, fecal testing vs colonoscopy) and delivery methods (ie, testing kit mailed to individual vs in person at a clinic visit). Determining the optimal modality, delivery method, and interval follow-up between tests in a population of a particular geographic region or jurisdiction can be complex and difficult to assess. Simulation modeling offers an opportunity to address this gap by providing a method to test interventions in a simulated environment to inform decision making by health leaders, policymakers, and other end users.,

From our review of the literature, we identified 2 gaps in the literature that we aim to address with this systematic review: (1) no systematic review has looked at the application of simulation in CRC screening within the last 10 years, as the most recent review included articles until 2007 inclusively; (2) multiple systematic reviews have looked at the utility, strengths, and opportunities for simulation modeling in health care but have not evaluated the impact of models on health system and policy decision making.,

Therefore, we aim to address these knowledge gaps by assessing the validity and impact of simulation modeling in health system and policy decision making for CRC screening delivery as reported in the literature. Recognizing the risk of underreported impact, we also aim to assess the potential impact of a model on a healthcare or policy decision by assessing the utility of the evidence generated by the simulation model using the GRADE EtD (Grading of Recommendations Assessment, Development and Evaluation Evidence to Decision) framework: an internationally recognized set of criteria for evidence-informed decisions in health systems and policy.

Objective

This study aims to assess the validity and impact of simulation modeling in the health system and policy decision making, where it has been frequently applied in health care: CRC screening delivery.

MATERIALS AND METHODS

Protocol and registration

This systematic review was conducted in accordance with the Cochrane Handbook for Systematic Reviews and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) statement, and was registered in PROSPERO (no. 130823)., A detailed description of the study protocol including the development process and rationale has been published elsewhere. Here, we provide a summary of the methods and minor adjustments from the published protocol.

Search strategy and eligibility criteria

We searched for articles published between January 1, 2008, and March 1, 2019, from 10 electronic databases (Medline, Embase, Cochrane Central, Scopus, National Health Service Economic Evaluation Database, ACM Digital Library, Econolit, Health Technology Assessment Database, IEEEXplore, and Cost-Effective Analysis Registry) using controlled vocabulary (MeSH [Medical Subject Headings]) supplemented by citation searches of all included articles. The final inclusion and exclusion criteria used to select articles are outlined in Table 1.

Table 1.

Inclusion and exclusion criteria

Inclusion criteria

Full-text articles published in English that meet all 3 criteria:1. Developed a simulation model derived from clinical data2. Focused on the delivery of colorectal cancer screening to average-risk individuals3. Simulated screening using only the following modalities of screening recommended by Canadian guidelines within the last 10 years:• Fecal occult blood test• Fecal immunohistochemical testing• Flexible sigmoidoscopy• Colonoscopy

Exclusion criteria

Excluded if any of the following 4 criteria are met: Used other screening modalities not recommended in Canadian screening guidelines as identified previously Commentary or review articles Simulation model including screening of other cancers No mention of colorectal cancer screening delivery

Selection of studies for review

All article titles and abstracts were screened by 3 independent reviewers (H.S., P.V., and C.K.) using the Abstrackr abstract screening program followed by a screening of selected full-text studies. All were reviewed by H.S., and 50% were each reviewed by C.K. and P.V. to assess for compliance with the eligibility criteria as mentioned in Table 1 using DistillerSR, version 2019 (Evidence Partners, Ottawa, Ontario, Canada). Conflicts were resolved by discussion among all 4 authors until a consensus was reached.

Data extraction

All included studies were reviewed by H.S., C.K., and P.V. who each reviewed 50% of the included studies. Using DistillerSR, data were extracted regarding the model description, validation, and impact as outlined in Tables 1 and 2 of the published research protocol (Tables 1 and 2). Discrepancies were identified and conflicts resolved through discussion to obtain consensus among the authors. Missing or incomplete data were requested from the study authors.

To assess model accuracy, all models were assessed in accordance with the guidelines of the International Society for Pharmacoeconomics and Outcomes Research-Society for Medical Decision Making (ISPOR-SMDM) good practice guidelines for modeling in health care, including recommendations on model validation. We also assessed for the involvement of “end users”: decision makers, clinicians, and administrators who would use the model for decision making.

Studies were then assessed for the extent to which the study has or could potentially have contributed evidence toward informed decision making. For contribution, each article was searched in its entirety for statements referring to the simulation model results informing decision making. Recognizing that the impact on decision making is often not communicated at the time of publication, we also assessed the potential contribution a simulation model could have made to evidence-informed decision making based on whether the results align with important factors for making informed decisions as outlined in the GRADE EtD (Grading of Recommendations Assessment, Development and Evaluation Evidence to Decision) framework.,

Corresponding authors of each study were also contacted to provide further information on model validation, end-user involvement, and impact of the simulation model. Results were analyzed using Microsoft Excel (Microsoft Corporation, Redmond, WA) and RStudio version1.1463 (RStudio, Boston, MA). In contrast to the protocol report, no subgroup analysis was conducted due to the heterogeneity of included articles.

RESULTS

A total of 492 relevant articles were identified through database and reference searching. Of these, 43 met inclusion criteria for data extraction (Figure 1, Table 2). Cohen’s kappa interrater reliability was 0.82 for title and abstract screening and 0.93 for full-text screening. Modeling methods were identified in all articles. Several used multiple modeling methods (n = 9 of 43 [20%]), and the most frequently encountered combination of techniques was Monte Carlo simulation with Markov modeling (n = 6 of 9 [67%]) (Table 2). The most frequently used methods, including when they were used in combination with other techniques, were Markov modeling (in 31 of 43 studies [72%]), Monte Carlo simulation (in 7 of 43 studies [16%]), and microsimulation (in 6 of 43 studies [14%]) (Figure 2). A total of 33% were based on preexisting models, with the MISCAN-Colon (MIcrosimulation SCreening ANalysis Colorectal Cancer) model the most frequently cited as the base model. A majority of models had multiple applications (39 of 43 studies [90%]). The most common applications were to assess cost-effectiveness of screening (in 32 of 43 studies [74%]), compare screening modalities (in 18 of 43 studies [42%]), or assess resource utilization of screening (in 15 of 43 studies [35%]) (Table 2). Models often included multiple screening modalities, with colonoscopy being the most frequently included (26 [63%]). Inputs were mostly generated from published trials, registries, and population statistics databases within the 5 years of publication (Figure 3). Occasionally, authors conducted primary data collection to populate the model (6 [14% of models]). Model outcomes most often included cost-effectiveness (30 [70% of included articles]), mortality (19 [44%]), cancer detection (13 [30%]), resource utilization (6 [14%]), or screening participation (7 [16%]).

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) flow diagram. CRC: colorectal cancer.

Table 2.

Included article summary

GRADE EtD: Grading of Recommendations Assessment, Development, and Evaluation Evidence to Decision . “X” denotes validation efforts were described but not explicitly performed.

Figure 2.

Popularity of simulation model methods.

Figure 3.

Model inputs.

Details of the simulation models were not consistently described. The model entities, attributes, and variables were often identified; in contrast, 29 (67%) articles included a figure of the conceptual model, and 23 (54%) identified the software program used to generate the model. Of those that mentioned a software program, TreeAge Pro (TreeAge Software, Williamstown, MA) and Microsoft Excel were most frequently used (48% and 26%, respectively). Few articles described the role of end users in developing or evaluating the simulation model (4 [9%]). We contacted all authors for further information, and among the 12 who responded, 11 (92%) reported end-user involvement in both model conceptualization and development and 8 (67%) reported involvement in model validation.

Model validation was not consistently reported (Figure 4). A majority of authors mentioned steps in their methods that suggest model validation (42 [98%]), in contrast to 25 (58%) that reported performing any form of model validation, and no article addressed all 4 aspects of model validation recommended by ISPOR-SMDM. Internal and predictive validation were most often reported, whereas face validation was rarely reported (34% and 37% vs 7%). All 12 authors who responded to our correspondence reported conducting face validation, and 4 (33%) reported conducting all types of validation recommended by ISPOR-SMDM.

Figure 4.

Model validation.

A majority of simulation models were developed to address a particular health system or policy decision in CRC screening delivery (91%), and all articles contributed some evidence important to informed decision making as outlined in the GRADE EtD framework (Figure 5). A total of 12% of articles described the study as having an impact on decision making, whereas 7 of the 12 (58%) authors who provided feedback reported that the model did have an impact on health system or policy decision making. All 7 of these “impactful articles” also described having end users involved in the research question formulation and model conceptualization. Articles with a reported impact on decision making showed no significant difference in the relative contribution to the GRADE EtD framework (published research protocol, Table 2).

Figure 5.

Impact on decision making.

DISCUSSION

Simulation modeling informing decision making

This review demonstrates that simulation modeling is a valuable tool for informed decision making in CRC screening delivery by providing evidence that meets all 9 criteria for informed health system and policy decisions outlined by the GRADE EtD framework. In particular, simulation models were found to be useful for estimating the long-term outcomes and cost-effectiveness of different screening strategies in a specific population or geographic region, which is consistent with its application in other areas of health care., However, the translation of these models to evidence-informed decisions appears to be lacking or underreported. Studies rarely describe an impact on decision making, in contrast to the majority of the 12 authors who responded to our correspondence (16% vs 58%). The generalizability of these 12 responses is limited, given that they only represent 28% of all contacted authors and may have been more inclined to respond given their impact on decision making. Nonetheless, this gap in reporting of impact has been identified in multiple reviews and appears to be an issue unique to simulation modeling in health care, unlike in other sectors.,,, Reasons for this may include the journal focus, word count limitations, or timing between publication and policy decision. While we cannot confirm the actual impact of models in the context of these limitations, we find that the potential for impact is very strong based on the GRADE EtD criteria. As clinicians and experts in health systems research, we find this represents a critical gap. The fact that a majority of models were developed to address a specific decision in CRC screening delivery further supports the argument that these models may be impactful, but their impact remains underreported. Other studies not included in this review suggested that simulation modeling is very useful in decision making, as demonstrated by the use of multiple iterations of the MISCAN-Colon model in the Netherlands., If the purpose of simulation modeling is to inform decision makers, then this needs to be reported. Only then can we understand how to enhance methods of developing impactful simulation models in CRC screening.

End-user engagement is critical to the success of impactful simulation modeling. A majority of authors with whom we corresponded indicated that end users were involved in the study and that the model influenced their decision making, but they did not report this in publication. Only 1 included article provided a thorough description of end-user involvement in model development and evaluation. Various articles, outside the inclusion criteria of this review, have advocated for greater end-user participation, termed participatory simulation modeling, whereby end users collaborate with researchers in all phases from conceptualization to validation of the model. The extent to which similar collaborations actually occur in the development, analysis, and validation of “nonparticipatory” simulation models of CRC screening delivery remains unknown. If our correspondence with authors is representative of all articles included in this review, then end users are frequently involved but their participation is not reported in the published article. The goal of simulation modeling is to serve as a decision support technique for end users. Therefore, it is important to report on the methods used to engage with end users and the extent to which the model supports their decisions, recognizing their importance and that they may vary between projects.

Model validation and reporting

For simulation models to be useful for decision makers, models must be accurate and valid for application. Few articles in this systematic review describe model validation. This concern has been raised by many other reviews of simulation modeling in health care. In further discussion with the authors, we found that validation was often conducted but not reported. The validation process is not an isolated set of procedures, but rather is an integral part of model development. It provides the model with higher levels of credibility and ensures that the model’s behavior is close enough to that of the actual system so that the model can be a good substitute. Reporting the validation process is highly important in model development. There are methods available for testing the validation of simulation models using both subjective comparisons and statistical procedures. Multiple guidelines for validation have been developed but are not consistently adopted. A potential next step to address this issue is to require model registration, as is done with clinical trials to reinforce standardization of model validation, promote collaboration, and minimize redundancy of one-off model building.

Model components and model conceptualization were also not consistently reported. Only 28 (68%) articles included a figure of the conceptual model. To provide a verifiable simulation model, the model entities, attributes, states, activities involved, and the events a system will encounter should be identified to be able to develop a verified simulation model. Providing a conceptual model allows readers to understand the simulation model functions, and see that it mimics the actual system in a way that is accessible to both modelers and end users. Reporting the model components and the conceptual model are also important for assessing the credibility and reproducibility of a model. Templates such as those presented by Banks et al can be used to categorize the model components and to construct an efficient conceptual model.

An initial aim of this study was to assess the validity and utility of simulation modeling in CRC screening delivery. However, our final observations are limited due to potentially incomplete reporting by authors at the time of publication. Few studies reported the validity and utility of the simulation model; however, the majority of the 12 authors who responded to our correspondence indicated that their simulation models had been validated and were impactful for end-user decisions. Further research with a larger cohort of author responses is needed to conclude if these authors are representative of all included studies.

CONCLUSION

In this study, we found that simulation models can be a powerful adjunct to empirical research in understanding optimal CRC screening delivery and have been used to generate evidence critical to informing decision making for a broad range of health system and policy decisions in average-risk CRC screening delivery. However, reports of impact on decision making and the validity of simulation models are limited in the literature. Greater validity testing, impact assessment, and standardized reporting of both is needed to understand and demonstrate the validity and utility of simulation modeling.

FUNDING

This work was supported by a research grant from the University of Ottawa Telfer School of Management and a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada.

AUTHOR CONTRIBUTIONS

HS is the guarantor of this review. HS, CK, and PV performed the screening, data extraction, and analysis outlined in this manuscript. All authors participated in the conceptualization and design of this review, and have read and approve of this manuscript.