Screening For Colorectal Cancer in the Age of Simulation Models: A Historical Lens.

In late March, The Washington Post, citing unnamed officials, quoted Dr Anthony S. Fauci’s response to the demands of modeling the novel coronavirus epidemic: “I’ve looked at all the models. I’ve spent a lot of time on the models. They don’t tell you anything. You can’t really rely upon models.” The provocative claim from the Director of the National Institute of Allergy and Infectious Diseases was quickly seized upon by critics of the Coronavirus Task Force’s recommendations. Simulation models, however, are certainly not new and have played a prominent role in epidemiology for many decades. Less visibly, they have also come to dominate other areas of medicine, especially preventive medicine. When the American Cancer Society (ACS) issued a modification of the United States Preventive Services Task Force (USPSTF) recommendations on colorectal cancer (CRC) screening guidelines in 2018, the society justified its actions by rerunning a simulation model. The new recommendation by the ACS encountered opposition and was not universally embraced.3, 4, 5, 6 Tellingly, no new empirical studies of screening were involved in the revision: the updated recommendations emanated from a recalibration of the incidence of CRC within a simulation model.

In this commentary, we report on the historical evolution of simulation modeling as applied to screening for CRC. We highlight how simulation modeling has come to be regarded by some as equivalent to evidence-based data, but we caution against overreach by relying too heavily on modeling relative to empirical testing.

Computer simulation models sit uncomfortably alongside evidence provided by controlled observational studies and formal randomized trials. Simulations are at once experimental—researchers can adjust parameters and observe different outcomes—and yet also fundamentally nonempirical, in that the parameters, inputs, and assumptions need not have strong empirical bases of support. According to Douglas Owens of Stanford University and the USPSTF, the introduction of simulation modeling into CRC screening recommendations helped address questions that clinical trials had left unanswered. But as Eric Winsberg’s survey of the rise of computer simulation has noted, “It is a mistake to think of simulations simply as tools for unlocking hidden empirical content.” Simulations do not produce empirical evidence; they enable the creation of “present futures” or alternative visions of the future to help decision making in the present.

As recently as 1996, however, simulation models were not used at all in screening recommendations. The rapid rise in their importance over the following 2 decades, and their central role in shaping current screening recommendations, should occasion reflection about how and why simulation models have become an influential tool for medical decision making.

The Rise of Screening Models

Although the introduction of electronic computers and algorithms into medicine broadly dates back to the 1950s, the idea of simulating the effects of screening with computers arose only in the early 1970s. In 1973, E.G. Knox, a professor of social medicine at the University of Birmingham in England proposed introducing computer simulation to provide a more rigorous economic justification for preventive measures such as screening. Knox’s first simulation program was for cervical cancer screening and constituted 650 statements of Fortran code. It used the insight that each individual in a population could be modeled as moving between a series of states, from one of “normal,” to others of “occult invasive disease,” “clinical invasive disease,” “dead of other causes,” “treated early-invasive,” “dead of the clinical cancer,” and so on. The simulation model assigned probabilities of transferring between these states. Knox began with a cohort of 10,000 people with a known life-table taken from the Registrar-General’s reports for England and Wales in 1970 and used a series of punched cards fed into an electronic computer to model transitions between 26 disease states.

The basic elements of Knox’s simulation have remained in many subsequent models: vital statistics data are combined with empirical evidence about transitions between disease states to predict the number of deaths or life-years lost. This simulation can then be rerun after introducing a preventive screening intervention. Empirical evidence about screening, such as the effect of screening on incidence and stage of tumors, can be incorporated and the change in deaths or life-years lost computed, and ultimately weighed against the financial and medical costs of the screening intervention. There are of course different varieties and subtleties among models—state transitions can be treated as continuous or discrete, for example—but we still use Knox’s essential idea that a simulation should be run twice, once with and once without an intervention to estimate the intervention’s effects on progression of disease.

Knox’s work obviously required the existence of widely available electronic computers, but it was more directly a product of reforms in social medicine meant to address the increasing costs of chronic disease. Knox’s original model was published as part of a symposium on the future of preventive medicine within the National Health Service, and the importance of modeling studies would only increase with the creation of the Canadian Task Force on the Periodic Health Examination in the late 1970s and the USPSTF in 1984. , During the 1980s, computer simulations of screening regimens proliferated as researchers sought to understand their potential benefits and costs.14, 15, 16

Until the twenty-first century, however, computer simulations were not seen as providing evidence definitive enough to shape screening recommendations. Randomized controlled trials continued to provide the highest level of empirical evidence for the USPSTF and the Canadian Task Force on Preventive Health Care. The 1996 Guide to Clinical Preventive Services of the USPSTF did not rely on even established simulation models and 2 years later USPSTF representatives advocating for the use of models still acknowledged “important obstacles” because models were “invariably complex and involve numerous assumptions and subjective judgments.” ,

A major turning point was the development of the Cancer Intervention and Surveillance Network (CISNET) in 2000. CISNET is a consortium of modelers supported by the National Cancer Institute who, while independently modeling the effects of different cancer control interventions for breast, prostate, colorectal, esophagus, and lung cancers, collaborate whenever possible by addressing a similar question with common inputs and outputs so models can be compared. Thus, models and modelers might act in effect as a panel of experts, each weighing and assessing the empirical evidence, fleshing out disagreements in the service of producing a consensus recommendation.

In the case of CRC, 3 models are part of the CISNET group: MIcrosimulation SCreening Analysis Colorectal Cancer Model (MISCAN-Colon), Simulation Model of Colorectal Cancer (SimCRC), and Colorectal Cancer Simulated Population model for Incidence and Natural history (CRC-SPIN). Models for CRC screening were often based on those designed for breast and cervical cancers, but emerged later because empirical studies on the effects of CRC screening arrived later. The first CRC models were created in the 1990s, , incorporating the first randomized controlled studies of CRC screening.22, 23, 24, 25 The resulting simulation models within the CISNET group are similar in many regards: all are stochastic microsimulation models (meaning that instead of cohorts they model an individual’s probabilistic transition between disease states) and all take into account the natural history of CRC as well as Surveillance, Epidemiology, and End Results (SEER) incidence and mortality data.

Simulation Models in Contemporary Preventive Medicine

In preparation for its 2008 recommendations on CRC screening, the USPSTF augmented their systematic evidence review for the first time with models provided by CISNET researchers, prompting praise for the use of state-of-the-art technology, but also concern that modelers had made some “surprising choices,” including not quality adjusting the estimates of benefits or accounting for patient and institutional costs of adherence. By 2016, the USPSTF’s CRC recommendations relied heavily on CISNET researchers to provide information on the starting and stopping ages, as well as recommended intervals for screening methods. In particular, CISNET researchers modeled the experience of 40-year-old individuals with no previous cancer diagnosis, and aggregated those experiences over an entire hypothetical cohort to measure both the benefit (in life-years gained [LYG]) and the cost (in number of lifetime colonoscopies required) of screening. Effectiveness was defined by an “efficient frontier,” highlighting the strategies that provided the “largest incremental increase in LYG per additional colonoscopy performed.” They concluded “the strategies of colonoscopy every 10 years, annual FIT (fecal immunochemical testing), SIG (flexible sigmoidoscopy) every 10 years with annual FIT, and CTC (computerized tomographic colonography) every 5 years performed from ages 50 to 75 years provided similar LYG and a comparable balance of benefit and screening burden.”

All 3 models predicted that starting screening at age 45 would result in an increase in LYG, but the USPSTF rejected recommending earlier initiation dates. They reasoned that the increase in LYG relative to the burden of increased colonoscopy examinations was small. Combining an earlier starting age with an extension of the subsequent screening interval from every 10 to every 15 years would have maintained a similar estimate of required lifetime colonoscopies but 1 of the 3 models (MISCAN-Colon) predicted this solution would cause a loss in LYG. In the end, the USPSTF considered these findings and concluded that “the evidence best supports a starting age of 50 years for the general population, noting the modest increase in LYG by starting screening earlier, the discordant findings across models for extending the screening interval when the age at which to begin screening is lowered, and the lack of empirical evidence in younger populations.”

The ACS’s review 2 years later addressed the question of initiation age by homing in on the discordant findings across models. In particular, they noted the apparent outlier, the MISCAN-Colon simulation model, would have agreed with the other 2 about the benefit of starting screening at age 45 if it had incorporated new information on the increased incidence of CRC in younger people. All 3 CISNET models had used SEER incidence rates taken from 1975 to 1979 (providing an incidence rate before the institution of appreciable screening activity, hence more reflective of the underlying natural history). The ACS research team used MISCAN-Colon to simulate “a cohort of adults aged 40 years in 2015, and assumed that this cohort had a 1.591-fold increased CRC incidence across all ages compared with the original model.” The incidence multiplier was derived from comparing SEER data from 1975 with more recent incidence rates. Rerunning the MISCAN-Colon model with updated assumptions resulted in the ACS Guideline Development Group providing a qualified recommendation that screening begin at 45, while maintaining initiation at 50 as a strong recommendation. ,

Discussion

In this instance, the use of a simulation model for CRC screening recommendations engendered controversy rather than resolved questions. Although there was indeed new epidemiologic evidence on CRC incidence published between 2016 and 2018, there were no new empirical studies about the effects of earlier screening. In fact, as the ACS group acknowledged, there has never been any substantial research on the effects of starting screening before age 50 for CRC. Although the model predicts benefits in earlier detection of CRC, a number of unintended consequences could potentially follow, primary of which is the diversion of resources away from higher risk populations for whom the relative benefit of screening is substantially greater.

Moreover, the agreement or disagreement of models is not a straightforward measure of the strength or weakness of evidence. It is difficult to know whether differences in predictions are due to differences in model design or model inputs. Two models that have been accurately validated using existing studies may have quite different predictions about future outcomes. A 2016 study by CISNET-CRC modelers, for example, noted that the essential but unobservable process of transition from benign adenoma to malignant tumor, or dwell time, is simulated by all the models but such a value is an output rather than input or assumption. As a result, even if models are carefully calibrated to match existing data, information about how sensitive predictions are to different dwell-time distributions, for example, remains unknown.

Given the expense of trials and the difficulty of combining multiple parameters simultaneously, such as age of initiation, screening interval, and preferred modality—modeling may indeed be an essential component for decision making. The ease of modeling compared with large-scale clinical trials, as well as the ability of models to combine evidence from many different sources also favors their use. Models have significant limitations, however, from their imposition of assumptions about the natural history of disease, the screening process, and the behavior of individuals, to the possibility of producing results with misleading levels of precision.

Most important, no matter how elaborate the design of the virtual laboratory created by a model or the quality of the experiments within, empirical studies remain essential. As has become abundantly clear in the case of statins and hormone replacement therapy, in the absence of large and unexpected effects, only randomized controlled trials of adequate size performed in applicable settings can provide reliable information about an intervention’s effectiveness. As Dr Fauci said recently, “All models are just models. When you get new data, you change them.” New epidemiologic data on incidence rates can certainly justify updating a model’s assumptions, which then might lead the model to make different predictions about the effects of screening, but without formal randomized trials of the new intervention, we have no new empirical information about effectiveness.

Computer simulations have subtly expanded what counts as evidence-based medicine. Balancing reliance on models with careful assessment of the systematic and empirical evidence is the core responsibility of organizations developing and delivering prescriptions for public policy. Careful deliberation is required to avoid over reliance on modeling. Models are no substitute for real-world data, and their conclusions remain bound by the availability of rigorous empirical studies.