Methods to construct a step-by-step beginner's guide to decision analytic cost-effectiveness modeling.

BACKGROUND: Although guidance on good research practice in health economic modeling is widely available, there is still a need for a simpler instructive resource which could guide a beginner modeler alongside modeling for the first time. AIM: To develop a beginner's guide to be used as a handheld guide contemporaneous to the model development process.
METHODS: A systematic review of best practice guidelines was used to construct a framework of steps undertaken during the model development process. Focused methods review supplemented this framework. Consensus was obtained among a group of model developers to review and finalize the content of the preliminary beginner's guide. The final beginner's guide was used to develop cost-effectiveness models.
RESULTS: Thirty-two best practice guidelines were data extracted, synthesized, and critically evaluated to identify steps for model development, which formed a framework for the beginner's guide. Within five phases of model development, eight broad submethods were identified and 19 methodological reviews were conducted to develop the content of the draft beginner's guide. Two rounds of consensus agreement were undertaken to reach agreement on the final beginner's guide. To assess fitness for purpose (ease of use and completeness), models were developed independently and by the researcher using the beginner's guide.
CONCLUSION: A combination of systematic review, methods reviews, consensus agreement, and validation was used to construct a step-by-step beginner's guide for developing decision analytical cost-effectiveness models. The final beginner's guide is a step-by-step resource to accompany the model development process from understanding the problem to be modeled, model conceptualization, model implementation, and model checking through to reporting of the model results.

Introduction

In countries where health technology assessment mechanisms are well established, decision analytical cost-effectiveness models (subsequently referred to as models) play a pivotal role in addressing difficult health-care decisions. Developing models is a complex process – it requires numerous steps and different skills are required to complete each step in a way that adheres to best practice in modeling.

In 2012, the International Society for Pharmacoeconomic and Outcomes Research (ISPOR) and the Society for Medical Decision Making published an update of the 2003 recommendations for best practices in modeling.1–7 These publications set out the “gold standard” of modeling practice; however, by their own admission they are “not intended as primers on their subjects”1 and may not be well understood by those embarking on modeling for the first time.1,8 Few resources are available to accompany the process of model development. Chilcott et al suggested that “although checklists and good modelling practice have been developed, these perhaps indicate a general destination of travel without specifying how to get there.”8 This research directly addresses this gap by constructing a beginner’s guide (BG) to modeling to be used by model developers contemporaneous to model building. The objective is to construct a resource that is more basic than the ISPOR guidelines to improve the ease and accuracy by which a new modeler learns to model. For example, let us presume that a new researcher embarking on practical modeling for the first time consults the ISPOR guidelines and begins conceptualizing the model. The details of model conceptualization set out by Roberts et al and their supplementary material are very comprehensive in their guidance as to what to do and they include how to go about doing it.2 However, after conceptualization,2 readers are guided according to the type of model they will develop (State Transition Modelling,4 Discrete Event Simulation,3 or Dynamic Transmission Modeling5) and it is difficult for novice modelers to fill this gap of selecting an appropriate model structure. What would be useful is a resource that bridges this gap and potentially guides the modeler through something more aligned to the algorithm set out by Barton et al9 or the taxonomy of Brennan Chick and Davies.10 Therefore, a valuable resource would be one which guides the novice modeler through each step in the model development process and is a more elementary and supportive resource. What is required is a hybrid of best practice recommendations and a primer-style instructive text to enable early modelers to quickly achieve standards of modeling aspired to by ISPOR.

Aim

The aim of this research is to develop a BG to support model developers alongside the development of decision analytical cost-effectiveness models.

Methods

Four phases of research were undertaken to develop the BG, which are shown in Table 1 and described below.

Table 1

Methods used to develop the beginner’s guide (BG) to decision analytic cost-effectiveness modeling

Research	Phase I	Phase II	Phase III	Phase IV
Method	Systematic review	Submethods review	Nominal group technique (consensus agreement)	Validation
Objective	Review of best practice modeling guidelines	Review of submethods literature	To reach agreement on the draft BG	To test the usability and completeness of the BG
Outcome	An exhaustive list of steps undertaken to develop a model	The guidance, recommendation, or instructive statement to support each step identified in Phase I	A finalized BG	Revised guide to include reviewer feedback
Output (result)	Framework for developing the BG	Draft BG	BG ready for testing	Tested and improved BG

It was necessary to develop the BG incorporating two components: the first, an exhaustive list of steps involved in the model development process, and the second, an “instructive statement” attached to each step setting out any guidance, considerations, or recommendations related to carrying out that step. For example, one step could be to define the perspective of the model analysis and the instructive statement could potentially read something like: “consider a payer perspective when […]” and so on. To achieve this level of detail, it was necessary to first do a systematic review of guidelines (Phase I) to derive the list of all possible steps, and then a methods review (Phase II) to develop the instructive component of the BG for all steps, as described below.

In Phase I, a systematic review of best practice guidelines was performed to identify all steps potentially undertaken during model development. Best practice guidelines were defined as publications intended for the purpose of improving the methods and quality of models published by health economics experts or organizations, such as ISPOR. Through a process of data extraction, deductive reasoning, and guideline synthesis, an exhaustive list of steps used in developing a model was constructed. Full details of the systematic review are provided in the Supplementary materials.

For Phase II, the steps identified in Phase I were grouped into submethods and focused literature searches were performed to identify methods literature for each submethod. For example, when developing a model, it is necessary to identify resources and value those resources. A focused review was performed to identify best practice in identifying and valuing resources in order to include in the BG “how” to go about doing these two steps. Review of methods papers informed the “how to” part of the BG. Full details of the focused searches are provided in the Supplementary materials.

In Phase III, the preliminary BG was subjected to a consensus approach to evaluate whether it was current (included all information aligned with current methodological thinking in modeling), complete (covered all relevant aspects of model development), and clear (logical and unambiguous). A modified nominal group technique consensus method11 was used combining private (electronic) feedback in round 1 and interactive (face-to-face) feedback in a structured format with a facilitated meeting in round 2.

In round 1 of the consensus phase, the BG was distributed electronically via email to participants for familiarization, review, and feedback. For each model development step, participants were asked whether the step should be included, excluded, or reworded. Where ≥75% of respondents supported inclusion or exclusion of the step, this was considered final and not taken forward to the second consensus phase. Where no consensus was reached or ≥50% of the experts suggested rewording, the issue was instead addressed in round 2. In this way, the face-to-face time with experts was prioritized toward dealing with issues in which broad consensus did not already exist.

In Phase IV, two independent novice modelers at the University of Leeds developed a model using the guide. They evaluated the guide with respect to whether they understood the steps, found them useful, and performed them at that time point during the model-building process. In addition, the researcher redeveloped a case study model consisting of a five-state Markov model and recorded each step in the model-building process to evaluate the completeness of the guide.

Results

The BG is designed to be used by a beginner modeler alongside the model-building process and it is therefore necessary to structure the content in alignment with the model development process. Current literature divides the modeling process into a number of phases. The first phase addresses the question of what is to be modeled and looks at understanding the real-world decision problem and its context.8,12–14 Models, by their nature, simplify a complex system into one that can be captured using less complex mathematical methods. The second phase requires a model developer to consider how the real-life decision problem translates into a model. During this phase, there is a tension between what Sonnenberg describes as the “theoretical model,” which represents an understanding of the natural history/biological fact, and the “practical model,” which is the “most detailed model which can be constructed given the limitations of the available data and the need for the model to be understood.”14 A third phase involves the actual programming of the model,8,12,14 including obtaining the data to be used in the model.15 In the fourth phase, the different strategies are evaluated13 by considering likely outcomes, validation of these outcomes, and sensitivity analysis.8,15 The final phase involves engaging with the decision-makers8,13 and disseminating the model results along with uncertainty of the model. These model phases are iterative, and it is common for developers to go back and forth between different phases of model development.8

The phases of model development are summarized in Table 2.

Table 2

Consolidated summary of descriptions of the model development process

Briggs 2000(alongside RCT)15	Sonnenberg (1994)14	Stahl (2008)13	Sargent (2010)12	Chilcott (2010)8 Tappenden (2012)66
What will be modeled?
–	1. Biological truth to be modeled	2. Describe the system under study	1. Problem entity	1. Understanding the decision problem
How will it be modeled? (conceptual)
1. Setting a reference case of methods	2. Theoretical model (represents an understanding of the biological truth)3. Practical model (“most detailed model which can be constructed given the limitations of the available data and the need for the model to be understood”)	1. Describe the development of a simulation model	2. Conceptual model	2. Conceptual modeling (conceiving the model, cognitive processes of thinking about the model and the potential methods to be used, information available, etc). “Defining the boundary and depth of a model”8 and identifying critical factors for inclusion in the model are important during this phase.
Building the model
2. Specifying clinical/demographic patient characteristics	4. Implementation model (actual model in a software package)	–	3. Computerized model	3. Implementation of the model (the actual programming of the model software)
Running the model
3. Applying Bayesian methods to estimate data and distributions for the running of PSA	–	3. Evaluate the consequences of a given strategy4. Explore the model	–	4. Model checking phase is undertaken to verify that the model is working as it should be and “includes all activities used to check the model”8
Disseminating the model results
–	–	5. Predict or forecast the behavior of the system and persuade decision-makers through consensus and evidence	–	5. Engaging with the decision (reporting of the model results and answering the original decision problem)

Note: The numbering in this table (read vertically within each column) corresponds to the order of the task described by the original (referenced) author.

Abbreviation: RCT, randomized controlled trial; PSA, probabilistic sensitivity analysis.

Within each model development phase, multiple steps are undertaken. The number and order of these steps may differ across cases and are influenced by factors, including the type of model being developed, modeler’s preference, and data available to populate the model (eg, if using primary data from a randomized controlled trial, then the literature search strategy will differ versus if secondary data are being used to populate the model). The BG is structured according to the five phases of model development set out by Chilcott et al.8

Phase I – systematic review

Thirty-two best practice guidelines were data extracted, synthesized, and critically evaluated to identify steps taken and submethods used in model development. For a list of guidelines, please see Supplementary materials. A total of 148 steps involved in model development were identified and arranged into eight broad submethods, namely, evidence, model structure, resource valuation, effectiveness, uncertainty, validity, reporting, and general.

Phase II – methodological reviews

Developing a model involves many so-called submethods. For example, the steps relating to the evidence used in a model may be grouped together in an “evidence” submethod, which covers literature searching and review, evidence selection, and evidence grading. Similarly, separate submethods can be described for aspects, such as measuring and valuing health-related quality of life, characterizing uncertainty, and testing the validity of models. Nineteen submethod reviews were undertaken to identify and review literature relating to the eight submethods identified in Phase I. A summary of the literature reviewed for each submethod is shown in Table 3 and described below.

Table 3

Scope of review of eight submethods to inform the content of the beginner’s guide

Scope of submethod review	Literature reviewed
1. Evidence
Literature search retrieval and selection, selecting evidence for input parameters, evidence grading if relevant	Booth (2010)16; Paisley (2010)17; Braithwaite et al (2007)18; Nuijten (1998)19; Cooper (2007)20
2. Model structure
General methods for selecting model structure	Barton et al (2004)9; Cooper et al (2007)21; Brennan et al (2006)10
3. Resource valuation
Identifying and valuing resources, discounting	Miners (2008)22, Hay et al (2010)23; Shi et al (2010)24; Brouwer et al (2005)25; Claxton et al (2006)26; Claxton et al (2011)27; Gravelle et al (2007)28; Nord (2011)29
4. Effectiveness (health outcomes)
Clinical health (efficacy/effectiveness), methods to describe health-related quality of life (disease-specific; disease- and symptom-specific; generic measures (Euro-QOL EQ-5D, SF-6D, Health Utilities Index). Methods to value health-related quality of life (standard gamble; time trade off; rating scales), eliciting preferences (patient; medical experts; general population), discounting outcomes	Gray et al (2010)30; Craig et al (2009)33; Brazier (2008)31; Neumann et al (2000)32; Gold et al (2002)35; McDonough and Tosteson (2007)34
5. Uncertainty
Types of uncertainty (parameter; methodological, model). Methods to characterize uncertainty: deterministic (univariate and multivariate), probabilistic; model averaging; model selection; parameterizing subfunctions	Briggs (2000)15; Briggs and Gray (1999)36; Briggs and Gray (1999)37; Claxton (2008)38; Groot Koerkamp et al (2007)57; Sculpher (2008)40; Andronis et al (2009)43; Limwattananon (2008)44; Bojke et al (2006)45; Strong (2012)67; Brisson and Edmunds (2006)41; Groot Koerkamp et al (2010)42
6. Validity
Types of validity (face; internal; external; convergent; predictive) and validation methods	Schlesinger (1979)47; Sargent (2010)12; McCabe and Dixon (2000)48; Kim and Thompson (2010)49; Sendi et al (1999)50; Halpern et al (1998)51; Weinstein (2001)68; Weinstein et al (2003)52; Chilcot et al (2010)8; Stahl (2008)13
7. Reporting
Incremental cost-effectiveness ratio and confidence intervals; cost-effectiveness plane; cost-effectiveness acceptability curve; cost-effectiveness acceptability frontier; net benefit approach; value of information analysis	Black (1990)69; Briggs and Gray (1999)36; Briggs et al (1997)58; Fan and Zhou (2007)59; Dinh and Zhou (2006)60; Polsky et al (1997)61; Barton et al (2008)53; Briggs (2000)15; Fenwick and Briggs (2007)54; Fenwick et al (2001)55; Fenwick et al (2004)56; Groot Koerkamp et al (2007)57; Claxton (2008)38; Groot Koerkamp et al (2010)42
8. General
Methods to achieve transparency, selecting time horizons, subgroup analysis	Sculpher (2008)40; Eddy (2006)62; Cooper (2007)20

The evidence submethod included the steps involved in literature searching and review, literature selection, evidence grading, and selecting input parameters for a model. Five publications were reviewed to explore search methods and the use of evidence in models,16,17 evaluate methods for selecting evidence based on quality and other criteria, respectively,18,19 and discuss methodological challenges when using evidence for modeling.20

The model structure submethod was defined as processes (eg, algorithms) and methods to select the most appropriate model structure. Three papers were reviewed which provide a taxonomy of model structures or guidance on choosing between them.9,10,21

The resource valuation submethod was defined as the methods used for identifying and quantifying relevant resources, assigning costs to resources, and the discounting of costs. Eight papers were relevant for the submethod dealing with resource valuation, including those dealing with quantifying resources,22 valuing resources,23,24 and approaches for discounting.25–29

The effectiveness submethod was defined as having two components. The first being capturing the measurement of clinical benefits both with respect to efficacy data from randomized controlled trials, and effectiveness data from other studies, such as observational studies and registries. The second component focuses on health-related quality of life with emphasis on methods to describe (measure) health-related quality of life, such as, disease-specific, disease- and symptom-specific, and generic measures (eg, Euro-QOL EQ-5D, SF-6D, Health Utilities Index). This component also includes methods to value health-related quality of life (standard gamble, time trade off, rating scales); means of eliciting preferences (patient, medical experts, general population); and discounting outcomes. Relevant guidelines here include an overview of general issues regarding effectiveness,30–32 adverse events,33 quality of life measurement,34 and methods used to incorporate quality of life into models.35

The uncertainty submethod was defined as types of uncertainty and methods to characterize uncertainty. A total of 12 papers were reviewed for uncertainty, including several that contribute to an understanding of terminology,15,36–40 methods to address uncertainty in general,41,42 and more focused guidelines concentrating on parameter uncertainty43,44 and structural uncertainty.45,46

Ten key publications formed the basis of the review of validity, which covered types of validity and validation methods.8,12,47–52 Schlesinger sets out recommended terms to describe model credibility,47 while Sargent defines and outlines methods of validation and verification.12 Other papers included a framework for assessing validity in models48 or applied broad issues in validation using specific models.49,50 In several other papers, validity is considered even though the focus of the papers appears to lie elsewhere.8,51,52

Model reporting was defined as any numeric or graphical results of a model, which included incremental cost-effectiveness ratio and confidence intervals; cost-effectiveness plane; cost-effectiveness acceptability curve; cost-effectiveness acceptability frontier; net benefit approach; and value of information analysis. For reporting of models, in six papers the graphical presentation of uncertainty from probabilistic sensitivity analyses is considered.15,53–57 Several older papers compare methods of presenting uncertainty around incremental cost-effectiveness ratios,36,58–61 with more recent papers instead considering the use of the net benefit approach and value of information analysis.38,42 Several other papers did not fit neatly into the defined submethods and were included in a general section which covered topics, such as methods to achieve transparency,62 selecting time horizons,63 and subgroup analysis.40

As a result of the submethods literature above, each of the 148 steps involved in model development were expanded upon to take the form of an instructive or directive statement/step along with explanatory notes, examples, and relevant references for each of the steps. This submethod review thereby informed the content of the BG, which was subjected to consensus agreement.

Phase III – consensus agreement

For the consensus agreement phase, purposive sampling was used to identify experts from across the UK. A total of 22 experts were originally contacted, of whom 18 responded. Of these, six experts contributed in round 1 (with the others either not replying at all or not replying in time to allow their responses to be incorporated) and 12 contributed in round 2. A total of six experts participated in both rounds.

For each of the 148 model development steps, participants were asked whether the step should be included, excluded, or reworded. Included as such were 133 steps (90%) following round 1 and four steps (3%) were excluded at round 1. This left ten steps, which were discussed at round 2, in addition to four new steps proposed for discussion after feedback in round 1.

The final panel invited for round 2 comprised nine academics, one industry participant, and two from contract research organizations. Five were experts in health economic modeling, two in guideline development, and one each in health technology assessment, literature searching, utilities, uncertainty, and statistics, respectively.

A total of ten experts attended round 2, with their responses digitally recorded and transcribed with the attendees’ permission. The number of steps increased from 148 to 156 after the consensus phase. Please see Supplementary materials for the final list of 156 steps.

Phase IV – validation

Two researchers provided feedback on three main aspects of the guide: whether they understood each of the 156 steps, the usefulness of the steps (1= not useful; 2= useful; 3= very useful), and whether they undertook the steps during that particular phase of model development. The results are summarized in Table 4.

Table 4

Validators’ feedback on understanding, usefulness, and timing of the steps in the beginner’s guide

Question	V1		V2
	n	(%)	n	(%)
Total steps	156	100	156	100
Did you understand this step?
Yes	125	80	105	67
No	9	6	1	1
Missing	22	14	50	32
Missing values – input mean values for repeated steps*	10	6	38	24
Rate the usefulness of this step on a scale of 1 to 3 (1 = not useful; 2= useful; 3= very useful)
3= very useful	92	59	102	65
2= useful	19	12	3	2
1= not useful	4	3	0	0
Missing values treated as missing	41	26	51	33
Missing values – input mean values for repeated steps*	29	19	39	25
Did you perform the step here (or at another stage in the modeling process)?
Yes	103	66	91	58
No	27	17	17	11
Missing values treated as missing	26	17	48	31
Missing values – input mean values for repeated steps*	14	9	30	19

Note:

Mean entries have been inserted for literature search, source, selection, and evidence grading steps, which were rated as done at that time point by both validators and useful by V1 and very useful by V2.

Abbreviations: V1, validator 1; V2, validator 2.

In summary, there seemed to be a good understanding of the content of the BG; it was useful for the model development process and most of the tasks were timed to coincide with the flow of the guide. In response to the feedback from Phase IV, a more detailed set of user instructions was compiled and links to the relevant literature resources were included in the BG.

In a second validation step, one of the authors (TR) redeveloped a model developed at the Centre for Health Economics, Technology Assessment Group at the University of York.64 Independent of the BG, each step of the model was undertaken and recorded to complete the development of the model according to the finished product. This recorded list was then cross-checked with the BG to determine whether this step was included in the BG and where it was not included, to explain why. The original and redeveloped model was a decision tree in Microsoft Excel® (Microsoft Corporation, Redmond, WA, USA), which evaluated responders and nonresponders over lifetime duration, and included mortality. The model adopted the perspective of the National Health Service and Personal Social Services and the model output was cost per quality adjusted life year. Health effects were measured as quality adjusted life years and both costs and outcomes were discounted at 3.5%. Probabilistic and deterministic analyses were undertaken.

Limitations

The BG needs to be evaluated with respect to the limitations of the research. Firstly, the foundation for the BG is two literature reviews, one focusing on guidelines and one on submethods. The guideline review informed the draft of the BG consisting of the steps undertaken during model development and was updated in 2011. The steps were agreed during consensus development and verified during Phase VI. On this basis, an updated review of guidelines is not anticipated to change the steps in the BG substantially. However, the submethod reviews which informed the instructive component of each step will need to be updated annually as the empirical modeling methods are evolving in the discipline of health economics. The number of experts included in the consensus phase was constrained by geographical limitations and research funding; however, all experts participated voluntarily and none had any other participation in the research. There is a potential to evaluate the BG in a wider audience and for a range of model types. Moving forward, it would be valuable for more experienced modelers to also provide feedback on the BG. As the paper-based version of the BG was developed, it has evolved into an interactive web tool, which may be used to collect user feedback. Areas of paucity in the BG reflect the variation in the current literature, for example, limited guidance on the selection of clinical efficacy and effectiveness data as model inputs.20,33,65

No BG of the type reported here can stipulate the “correct” methods to be used, as this is likely to differ over time (with methodological and computational advances) and because the choice of appropriate methods is affected by the context of the decision to be made. However, in using the BG model developers should be confident that they have satisfactorily considered, suitably chosen, and can justify the submethods used for model development. Chilcott et al recognized a need for an aid to achieve best practice and considered this “a priority for future development.”8 The BG is a sound resource to fill that gap. Irrespective of these limitations, the current format of the BG will be of value to beginner modelers.

Discussion

The BG is intended to bridge the gap between theory and practical model development. The BG is intended as a complement to, rather than a replacement of, the ISPOR guidelines and this topic is the subject of another paper. In summary, there are three main distinguishing features of the BG. Firstly, the ISPOR guidelines set out a set of recommendations while the BG sets out a comprehensive list of steps to be considered and, if applicable, undertaken contemporaneous to model development. Secondly, the ISPOR guidelines are arranged according to model conceptualization, three specific modeling techniques, uncertainty, and validation, whereas the BG is arranged according to the five phases of model development. Thirdly, the BG takes a novel approach to integrating the concepts of uncertainty and validation into each step of model development rather than discrete concepts to be addressed separate to, or upon completion of, model development. Arguably, if model developers are made aware that a certain step in the model-building process may contribute to greater or lesser uncertainty in the model structure/results, then they are better informed to make adequate judgments as to how to minimize the uncertainty introduced during this step and also to consider the impact of potential uncertainty within and around the model results. Similarly when using the BG, a modeler is made aware of the impact of each step on model validity and is therefore better able to maximize the validity of the model. For example, during model conceptualization the modeler is made aware of the importance of consulting clinical experts to verify the face validity of the model – in this way, the modeler is aware of how this step potentially influences the face validity of the model. Each step is also, where relevant, linked to bias and heterogeneity throughout. For example, when selecting the comparator, selecting the relevant comparator influences the face validity of the model and if incorrectly chosen may introduce bias into the model (eg, if a costly comparator is selected). The BG condenses relevant aspects of model development into a single, accessible resource to inform modelers about the methodology of model development while they are going about developing a model. Where it is able to, it provides direct guidance; otherwise, it lists relevant references which describe and discuss potential methods. It also includes ancillary resources, for example, a quick reference section to evidence selection, detailed information on types of uncertainty and validity, and the methods used to address both.

The BG is potentially valuable for users of models, both by increasing the quality of what it produced and highlighting any deficits in documentation. In this way, it increases the transparency of the model development process and alerts users to potential sources of bias. Disaggregating the process into its smallest steps makes the process explicit and clear and the weaknesses of the process can be better perceived.

Conclusion

A BG has been developed based on four research methods. It has demonstrated usability in the model development process. Research is ongoing; however, the BG has the potential to be used in the operationalization of best practice recommendations in modeling.