Planning and priority setting for vaccine development and immunization.

We review a sequence of strategic planning efforts over time in the United States, all involving processes to prioritize new vaccine candidates. The Institute of Medicine of the National Academies of Sciences, Engineering, and Medicine has been involved in three priority setting processes, each using different metrics and methodologies: infant mortality equivalents (1985-1986), cost-effectiveness (2000), and more recently, the implementation of a software system based on a broader multi-criteria systems approach that can include either of the earlier metrics among other various considerations (2015). The systems approach offers users the flexibility to select, combine, rank, weigh and evaluate different attributes representing their perspectives, assumptions, and particular needs. This approach also overcomes concerns relating to the previous single-metric ranking approaches that yielded lists that, once published, were static, and could not readily accommodate new information about emerging pathogens, new scientific advances, or changes in the costs and performance features of interventions. We discuss the rationale and reasoning behind the design of this multi-criteria decision support approach, stakeholder feedback about the tool, and highlight the potential advantages from using this expanded approach to better inform and support vaccine policies.

Plans and priorities

“Plans are worthless,’ Dwight Eisenhower once said, “but planning is everything. There is a very great distinction because when you are planning for an emergency you must start with this one thing: the very definition of ‘emergency’ is that it is unexpected, therefore it is not going to happen the way you are planning” [1]. So it is with infectious diseases, especially when a virulent strain emerges or re-emerges to affect public health and policy.

The bubonic plague killed well over half of the populations in European nations between 1346 and 1353 [2]. In London and Newcastle, over 10,000 people died in 1853, the year before John Snow’s famous pump handle action quelled an 1854 cholera outbreak with only 600 dead [3]. In 1918–1919, a deadly influenza outbreak killed upwards of 50 million people [4]. Even in recent years, the world has seen panic over potential pandemics. In 2002–2003, an outbreak of the SARS virus paralyzed international travel. The FIFA Women’s World Cup events were shifted from China to the United States. Beijing closed schools for weeks. Canada quarantined over 5000 people. Universities in the U.S. banned foreign student enrollment from affected areas [5]. Most recently, following a major Ebola outbreak in West Africa with a case fatality rate of 40%, the desire to develop and deploy an effective Ebola vaccine became stronger, only after previous efforts were shelved a decade earlier for various reasons, including lack of manufacturers’ interest. The recent Zika outbreak provides anew a demonstration of the suddenness with which new threats can emerge and the need for a way to analyze their importance against other infectious diseases.

Clearly—demonstrating Eisenhower’s perceptiveness—none of these outbreaks could have been predicted in time, location, or severity. What then can strategic planning and priority setting do for the world of infectious diseases and for vaccine development and deployment? Our review of these issues begins with the 2010 National Vaccine Plan issued by the Department of Health and Human Services (HHS; Table 1 )—the most recent planning effort in the U.S. on these crucial issues that has five key elements [6]:

Table 1

2010 national vaccine plan: U.S. department of health and human services.

Goals

1. Develop new and improved vaccines

2. Enhance the vaccine safety system

3. Support communications to enhance informed vaccine decision making

4. Ensure a stable supply of, access to, and better use of recommended vaccines in the United States

5. Increase global prevention of death and disease through safe and effective vaccination

Priorities

A. Develop a catalog of priority vaccine targets of domestic and global health importance (Goal 1)

B. Strengthen the science base for the development and licensure of new vaccines (Goals 1 and 2)

C. Enhance timely detection and verification of vaccine safety signals and develop a vaccine safety scientific agenda (Goal 2)

D. Increase awareness of vaccines, vaccine-preventable diseases, and the benefits/risks of immunization among the public, providers, and other stakeholders (Goal 3)

E. Use evidence-based science to enhance vaccine-preventable disease surveillance, measurement of vaccine coverage, and measurement of vaccine effectiveness (Goal 4)

F. Eliminate financial barriers for providers and consumers to facilitate access to routinely recommended vaccines (Goal 4)

G. Create an adequate and stable supply of routinely recommended vaccines and vaccines for public health preparedness (Goal 4)

H. Increase and improve the use of interoperable health information technology and electronic health records (Goal 4)

I. Improve global surveillance for vaccine-preventable diseases and strengthen global health information systems to monitor vaccine coverage, effectiveness, and safety (Goal 5)

J. Support global introduction and availability of new and under-utilized vaccines to prevent diseases of public health importance (Goal 5)

Develop new and improved vaccines.

Enhance the vaccine safety system.

Support communications to enhance informed vaccine decision-making.

Ensure a stable supply of, access to, and better use of recommended vaccines in the United States.

Increase global prevention of death and disease through safe and effective vaccination.

The 2010 plan emerged from discussions among many federal agencies led by the HHS National Vaccine Program Office, including input from the National Vaccine Advisory Committee and a study by the Institute of Medicine (IOM) [7]. Table 2 shows the goals and objectives from the precursor 1994 National Vaccine Plan [8]. Comparison of the 1994 and 2010 plans highlights several differences. The 1994 plan focused heavily on traditional health care systems and practices, and did not specifically mention financial incentives to any participants. Inclusion of the words “for priority diseases” in Goal 1.1 provided the only direction towards any strategic priority setting [8].

Table 2

1994 national vaccine plan: U.S. department of health and human services.

Goals and objectives

1. Develop new and improved vaccines

1.1. Develop new and improved vaccines for priority diseases

1.2. Ensure the nation’s capability to detect and respond effectively to new and emerging diseases in the United States and abroad

1.3. Enhance the process of translating technologic innovations into new vaccines

1.4. Ensure the nation’s capability to evaluate new vaccines, and to conduct prompt reviews of new and improved candidate vaccines

1.5. Promote the improvement of existing vaccines and development of new vaccines and vaccine-related technologies for other diseases of importance in developing countries

2. Ensure the optimal safety and effectiveness of vaccines and immunizations

2.1. Enhance the ability to evaluate the safety and effectiveness of vaccines

2.2. Improve the surveillance and evaluation of adverse events following vaccination

2.3. Ensure the optimal use of vaccines

2.4. Continue to ensure fair and efficient compensation to individuals injured by vaccines

2.5. Promote and support the efforts of the World Health Organization to develop and harmonize international standards and improve regulatory capabilities in countries involved in vaccine production

3. Better educate the public and members of the health professions on the benefits and risks of immunizations

3.1. Increase public demand for immunization, especially among populations at risk of underimmunization

3.2. Improve the immunization practices of all health care providers

3.3. Increase the awareness of the benefits of immunization among special target audiences (third-party payers, employers, legislators, community leaders, hospital administrators, etc.)

3.4. Develop more effective methods of communicating the benefits and risks of immunization to health care providers, patients, and parents/guardians

3.5. Continue to evaluate the benefits and impact of immunization through the use of cost-effectiveness studies

4. Achieve better use of existing vaccines to prevent disease, disability, and death

4.1. Ensure an adequate supply of vaccines

4.2. Increase immunization coverage levels for infants and children

4.3. Maintain immunization coverage for school-aged children

4.4. Increase immunization coverage levels among older adolescents, adults, and the elderly

4.5. Improve the surveillance of vaccine preventable diseases to assess the impact of immunization programs

4.6. Establish registry and immunization tracking systems

4.7. Enhance immunization coverage to strengthen national defense

4.8. Enhance immunization of international travelers who are of highest risk of acquiring vaccine-preventable diseases

4.9. Eradicate poliomyelitis globally

4.10. Promote better control of neonatal tetanus and measles, worldwide

4.11. Promote the self-sustaining capacity of immunization programs in developing countries

The 2010 plan, in comparison to earlier plans, has a much broader focus: for example, it includes both a catalog of priority vaccine targets and creation of a vaccine safety agenda [6]. Further, the 2010 plan elevated the importance of evidence-based surveillance of disease incidence, vaccine coverage and effectiveness. This specifically recognizes the importance of global surveillance and information gathering, in light of the increasingly rapid international transmission of disease seen in the late 20th and early 21st centuries. Finally, it raises the importance of interoperable health information technology and electronic health records, and urges increased support for vaccines for global health. By contrast, with the exception of two words (“and abroad”), the 1994 plan focuses on domestic interventions.

In summary, the 2010 plan has a much wider focus and greater understanding of the high inter-connectivity of the various moving parts in the world of infectious diseases and immunization than did the 1994 plan. To be fair, fifteen years of advanced research and new technologies helped to inform the 2010 plan, information that was not available earlier.

The National Vaccine Plan is not the only participant in this broad discussion. In the United States, the National Vaccine Advisory Committee (NVAC) regularly recommends ways to achieve optimal prevention of infectious diseases in humans through vaccine development and to prevent adverse vaccine reactions. Further, the Advisory Committee on Immunization Practices (ACIP) of the Centers for Disease Control and Prevention (CDC) regularly publishes guidance about deployment of existing vaccines in the U.S., using processes that include disease burden, vaccine safety, cost-effectiveness analysis and other metrics [9].

The U.S. is not unique in considering these issues. Many other international groups also deal with issues involving various aspects of planning and prioritizing, including the Strategic Advisory Group of Experts (SAGE) [10] and the Product Development for Vaccines Advisory Committee (PD-VAC) [11], both within the World Health Organization. The Vaccine Investment Strategy of GAVI (updated every five years, most recently in 2013) is oriented towards determining which vaccines that are already available or likely to be available in the near future that GAVI would support [12]. Similar roles are played at the national level in many countries, e.g., the Joint Committee on Vaccination and Immunisation (JCVI) for the U.K. National Health Service that uses cost-effectiveness as a formal metric for its analysis [13]. The WHO PD-VAC focuses entirely on advising the development of vaccines in Phase 2 and beyond dealing with highly burdensome diseases in low and middle income countries, while the other groups generally focus on advising about uses of existing or nearly-available vaccines.

The 2010 National Vaccine Plan specifically called for a vaccine research and development prioritization effort, a request that followed two earlier prioritization efforts by the IOM. In the next section, we review the priority lists created by the IOM in 1984–1985, in 2000, and then the most recent—and novel—systems-based approach to this challenge developed between 2010 and 2015 at the request of the National Vaccine Program Office.

Planning and prioritization

Planning encompasses more than prioritizing, which is necessary but insufficient to create a functional strategic plan. Plans also include (among other things) consideration of how to reach the desired end points, the mechanisms to finance those operations, consideration of contingencies, and methods to measure progress against the plans. Priority setting typically comes early on, if not as the first step. The 2010 plan included a call for a prioritization catalog, for which NVPO commissioned the IOM to create and test ways to prioritize among new preventive vaccines. As we discuss later in this article, the approach that emerged—Strategic Multi-Attribute Ranking Tool for Vaccines (SMART Vaccines)—not only provides a unique way to set priorities, but also provides a tool to assist in other phases of the overall planning process. Thus in what follows, we discuss not only the priority setting aspect of this work but the more extended uses of SMART Vaccines that emerged during its design, development, and testing.

The earlier plans developed priority lists for vaccine development as a key part of the planning process. As we discuss next, these priority lists were developed with very different approaches and results, and they were intrinsically limited in two key aspects. First, they used a single metric to establish priorities, and hence provided limited value to those who have other vaccine attributes in their planning processes. Second, the resulting lists were (and are) static, and hence subject to becoming outdated as new pathogens, medical treatments or improvements in vaccine technology emerge or re-emerge.

Prioritization based on health benefits

At the request of the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health, the IOM issued two reports on vaccine development, the first volume in 1985 for U.S. priorities [14], and the second volume in 1986 for developing country priorities [15]. The primary ranking method of these reports used the metric infant mortality equivalence values, which was described as “the number of acute morbidity days or chronic cases considered to be equal in undesirability to an infant death.” The reports made these adjustments using the median of responses from the committee members (in Volume I for the United States) and from public health professionals in a variety of developing countries (in Volume II for developing countries). Volume I estimated reduced U.S. health care treatment costs. Volume II did not attempt to estimate changes in treatment cost in developing countries, so the only cost considerations involved were the estimated vaccine development, procurement, and administration. The closest analog to these measures in current evaluation methods would be the disability-adjusted life year (DALYs), a metric that uses expert opinion to characterize tradeoffs between full health, death, and different illness states.

Volume I analyzed 14 vaccine candidates did not evaluate diseases that had yet to come into focus or emerge as public health threats, including human papillomavirus (HPV), severe acute respiratory syndrome (SARS), avian influenza, Middle East Respiratory Syndrome (MERS), Ebola, and Zika virus. Volume II evaluated 20 vaccine candidates, again omitting Ebola, SARS, avian flu, swine flu, for example, and did not comment on a number of diseases for which vaccines were subsequently developed. We note this not to criticize these efforts, but rather merely to illustrate the potentially transient value of static priority lists.

The models used in these reports had a narrow focus and did not incorporate many facets of the decision process that the 1985–1986 committees knew to be important. Volume I states:

“The committee believes that final selection of priorities should be made after decision makers have evaluated certain non-quantifiable considerations discussed in the report, but not incorporated into the model. These include the goals of the agency and its schedule for achieving them; considerations of equity in the distribution of benefits; the opportunity and need for the agency to exert influence on development; the balance of the desired portfolio of vaccine development projects; and the arguments that can be made for treating certain vaccines as unique because of their potential for restoration of public confidence in immunization programs.”

Volume II had a similar disclaimer regarding developing country priorities, omitting the statement about potential restoration of public confidence in immunization programs and adding “and certain other nonquantifiable factors relating to the diseases and alternative control approaches.” We discuss this issue further in the following section as we describe the need for an expanded approach in evaluating the value of vaccines.

Prioritization based on cost-effectiveness

In 2000, again at the request of NIAID, an IOM committee, rather than relying on a health-benefit measure, used an efficiency criterion: cost-effectiveness of vaccine candidates [16]. This choice matched a specific objective (3.5) in the 1994 National Vaccine Plan (Table 2): “Continue to evaluate the benefits and impact of immunization through the use of cost-effectiveness studies” [8].

The 2000 committee calculated the incremental cost-effectiveness ratios for 26 vaccine candidates for development and use in the U.S. alone, and adopted a four-level grouping of cost-effectiveness ratios: most favorable (saves money and QALYs); more favorable (costs < $10,000 per QALY saved); favorable (between $10,000 and $100,000 per QALY); less favorable (more than $100,000 per QALY). The committee found seven candidates in the most favorable category, nine in the more favorable category, four candidates in the favorable category, and seven in the less favorable category—in many cases with age-range limits on the recommended usage, including separate recommendations for group B streptococcus by age-group targets. The 2000 report also differed from the 1985 report on U.S. vaccine priorities by including potential therapeutic vaccines with the goal of stimulating an immunological response that would “treat” several chronic conditions. Three of their most favorable recommendations fell into the therapeutic category: insulin-dependent diabetes mellitus, multiple sclerosis, and rheumatoid arthritis.

As with the 1985–1986 studies, the 2000 report included a specific caveat about the limitations of the cost-effectiveness model, stating:

“It is common in studies like the committee’s to employ a cost-effectiveness analysis to develop a priority list, and then to remind policymakers who will make use of the list that they must, of course, also consider distributive issues of fairness or justice concerning the distribution of benefits and burdens in the ultimate priorities that they adopt.”

They highlighted distributive and fairness considerations, but did not address other aspects of vaccine attributes that might affect vaccine choices. As we review next, a comprehensive systems approach allows these issues to be formally embedded in the analysis.

Finally, the 2000 IOM committee published the spreadsheets used in their work, making them available for others to modify to conduct sensitivity testing on the assumptions adopted by the committee. In concept, the same format could accommodate analysis of vaccines not undertaken by the IOM committee, but to our knowledge, no such elaborations have occurred.

The value and uses of a systems approach

The priority-setting process

Setting priorities and plans for vaccine research and development is a subset of the larger issue of strategic planning in public health and medicine. Past efforts in vaccine priority setting and the generation of priority lists have missed the mark in several ways. Despite much progress, scientific and technical feasibility have ultimately limited the development of vaccines at the highest priority (such as HIV, tuberculosis, malaria, and broadly protective universal influenza vaccines) and other vaccines identified as priorities in the earlier priority setting processes. In addition, the development and publication of these lists did not allow the incorporation of new knowledge to inform future decision-making. Perhaps most importantly, these efforts did not incorporate valuable aspects of potential vaccines that were not included in the basic evaluation metric (health benefit or cost-effectiveness). In recognition of the limitations of these earlier vaccine priority setting approaches, the 2010 National Vaccine Plan identified the need to “develop a catalog of priority vaccine targets of domestic and global health importance.”

In late 2010, the NVPO commissioned the IOM to advance the priority setting process in a three-phase project [17], [18], [19]. During the information gathering sessions, the phase I committee heard that the 1985–86 and 2000 IOM studies—as useful and data-laden as they were—suffered from a narrowness of perspective, and indeed, the observation that the primary evaluation metrics (first, health benefits, then cost-effectiveness) differed considerably between these studies [20].

In response to these concerns, the phase I committee concluded that a flexible systems-based tool would better serve the global needs than another fixed priority list. They chose to create an adaptive planning mechanism rather than creating a fixed plan for the future. This strategy resulted in a first-of-its-kind decision support tool called the Strategic Multi-Attribute Ranking Tool for Vaccines (or SMART Vaccines). The two successor committees enhanced the model and data structures supporting SMART Vaccines 1.0 (phase II), and iteratively tested the next version SMART Vaccines 1.1 in field use with four user groups (phase III) in addition to getting feedback from various international stakeholders [17], [18], [19]. The phase III product (developed by the IOM in partnership with the National Academy of Engineering) also contained a strategic plan for the further development of SMART Vaccines from a desktop Windows application to a fully web-based version to expand usability and access [21].

With a goal of supporting a wide range of users with differing (and even competing) definitions of value, the SMART Vaccines committees created a broad list of 29 built-in attributes (plus the flexibility for seven user-defined options) from which users could make their selections. These attributes [discussed in 17–20] include health outcome measures (e.g., incident cases or premature deaths averted or various health-adjusted life year options), economic criteria (e.g., cost or efficiency measures), deployment issues (e.g., cold chain requirement, dosages, fit with existing vaccine protocols), and other specialized interests (e.g., scientific feasibility, special sub-populations of interest, military personnel issues, foreign policy goals).

Throughout this work, the core objective was to create a flexible, adaptive mechanism that could accommodate different points of view and could readily adjust for newly emerging pathogens, treatment costs, or scientific developments. Among several available options, the developers of SMART Vaccines opted to structure the planning tool around multi-criteria systems analysis (specifically, multi-attribute utility theory) that allows users to specify what is important to them in prioritizing vaccines and to state specifically the relative importance of each chosen attribute. Various modeling approaches were evaluated including the ease of use in diverse settings (especially developing countries with limited resources), and the ability to incorporate diverse perspectives in evaluating vaccines—all perceived as limitations in earlier IOM approaches.

A priority-setting approach using SMART Vaccines not only makes the data assumptions visible, but also makes absolutely clear the value choices built into the analysis. We believe that with the growing use of online social media and communication technologies, people around the world have come to both expect and demand more transparency in policy decisions. SMART Vaccines provides that possibility in the realm of vaccine prioritization [22].

Beyond priority-setting

The developers of SMART Vaccines realized that this systems approach had great value extending beyond the original intended purpose (establishing priorities for new preventive vaccines). The most obvious extended use became apparent in early testing of the tool—the desire to help select among available vaccines for general or sub-population deployment. In addition, we can identify a number of other ways in which SMART Vaccines (or generalizations to other areas of health) can assist in wider aspects of comprehensive strategic planning processes. In saying this, we again acknowledge that SMART Vaccines (and its successive wider use concepts) do not make decisions or create strategic plans, but we believe that it is a clear and transparent systems analytic approach that can widely assist in such endeavors. Potential applications are many, and include:

Scenario Evaluation. Many strategic planning processes include examination of differing scenarios to determine how robust the proposed plan when underlying conditions change significantly. This process, originating for the military (at the RAND Corporation) and corporate planning (at Royal Dutch Shell), has great potential in public health planning [23], [24]. This “what if” option is supported by the sensitivity analysis capabilities of SMART Vaccines, allowing fairly real time evaluation of the value of vaccine options in response to changes such as (a) new therapeutic options emerge, (b) disease incidence changes dramatically, (c) alternative interventions materialize to deal with the problem (e.g. genetic based vector control of mosquitos), and others. Diverse scenarios help test the robustness of a strategic plan against major environmental changes.

Program Evaluation. Evaluating public health interventions on a single-metric basis (e.g. live years saved or cost-effectiveness) is common, but is no more rational than choosing among alternatives on the same narrow basis (e.g., mortality or morbidity). The multi-attribute approach embedded in SMART Vaccines also allows for the evaluation of public health interventions once they are deployed. Evaluation and response are standard strategic plan components.

Evaluating Competing Approaches to Solving the Problem. Many problems have multiple potential solutions, often with attributes that differ in importance across populations. Systems analysis provides a rational way to consider these issues and weigh the importance of the various attributes in choosing among alternatives. Whenever these attributes cannot be readily captured in cost-effectiveness or cost-benefit approaches, the wider systems-based approach as embedded in SMART Vaccines gains value. As an obvious example, one could consider both malaria and Zika vaccines in a wider analysis that included mosquito netting, mosquito abatement (including introduction of genetically modified mosquitos into active malaria or Zika regions) or treatment options (for malaria) or birth control or pregnancy termination options (in the unique case of Zika).

Focusing Goals for Product Development. In the world of vaccines, we observe the practice of creating target product profiles to help guide vaccine R&D efforts. The SMART Vaccines approach can readily assist in sharpening target product profiles by incorporating relevant multiple attributes. This feature allows for clear evaluation of the tradeoffs of improvement along some specific attribute (e.g., number of doses required) vs. others (effectiveness of the vaccine and its consequences, program costs, etc.). The same concept can provide value for wider uses in public health and health care.

Clarify Critical Data Needs. In many strategic planning efforts, data quality inherently limits the precision and utility of the plans. SMART Vaccines, through sensitivity testing, allows planners to understand where better data are most important (e.g., those variables where SMART Scores are highly sensitive to small changes in data values).

Understand Multiple Stakeholder Perspectives. In most planning endeavors—even those in pure corporate settings—different stakeholders have different perspectives. Thinking through the relevant issues from these different viewpoints will likely enhance the value (not to mention the acceptability) of the final plan. SMART Vaccines helps provide clear and specific information about how these different viewpoints and assumptions might lead to different priorities and plan choices, and also to find areas where people with different priorities may find convergence.

Flexibility through Time. As noted, the SMART Vaccines approach has the additional advantage that new vaccine candidates can be readily evaluated as new issues emerge. The sudden and major Ebola outbreak in 2014 and the more recent outbreak of Zika in Brazil highlight the benefits of this approach—data can readily be added to evaluate an Ebola or Zika vaccine against other candidates. The Ebola outbreak also provided a clear example of how the ability to formally incorporate the values of more than one attribute of a vaccine (e.g., cost-effectiveness) can lead to markedly different rankings than single metric ranks provide [25]. Static processes cannot achieve this flexibility.

SMART vaccines: key opportunities and considerations

During the many presentations and extensive dissemination of SMART Vaccines in various countries, the IOM committees received extensive feedback for further development. We discuss these in this section.

Data needs

To implement SMART Vaccines for any particular population (its intended use), analysts must use data covering a wide range of issues—population demographics, disease burden data, and costs of treating each illness for which a candidate vaccine is evaluated [26]. SMART Vaccines has national population data built in for several dozen countries and limited disease burden data, but users must find or estimate numerous data elements to complete the model. The developers of SMART Vaccines recognized this challenge, but believed that the nature of the problem—not the software—creates the data burden. Careful review of the 1985–86 and 2000 studies show that they faced similar data challenges, and in some cases they ignored the issues when they did not believe that good data could be found. An example (from the 1986 report) is the omission of costs of treating illnesses in developing countries.

Even as extensive as current and planned future versions of SMART Vaccines allow, there remain numerous vaccine benefits that have not yet been calculated, and which would require even more extensive data. These include such benefits as the impact on social and economic development as a result of the impact of the prevention of serious infectious diseases and their short and long term consequences on family planning [27], childhood development and their implications for society more broadly [28].

There are several potential options for dealing with the issue of incomplete data. For example, one might envision generalized template data (stylized for region and level of economic development) and generalized estimates from appropriate experts to provide starting-point data for users with less-complete data systems. This could be coupled with the capability of conducting sensitivity analysis to determine which data have the most important influence on candidate vaccine rankings (given the weights used by the decision makers). SMART Vaccines also features a facility for robust sensitivity analysis, thus allowing users to identify which variables most strongly influence the resulting SMART Scores, and hence to understand where more investment in better data has the highest potential benefit, and conversely where data quality issues have little to no effect on final priority scores.

The risk of numerous priority lists

By design, SMART Vaccines allows different individuals or groups to create their own priority lists. The IOM committees heard concerns expressed that the creation of multiple lists would reduce the value of having a single list that guides the work of many stakeholders. Rather, the “catalog” of priority vaccine targets articulated in the National Vaccine Plan acknowledges that different users will have different needs and values, and thus different priorities: priorities for the U.S. population will differ from those in developing countries and, in turn, the needs of the military to keep its service members healthy. At the beginning of the SMART Vaccines work, IOM’s phase I committee heard repeatedly that the narrow focus of the earlier IOM reports reduced their value, since many potential users found that the reports (with a single evaluation metric) simply did not apply to their setting. Thus those stakeholders created their own lists informally or formally, using methods and data that were seldom made public. We believe that with SMART Vaccines in widespread use—including most desirably with a common methodology and datasets—multiple lists will still emerge, but the structure of them will be completely transparent and available to all, since differences in rankings will depend primarily on the values structures employed by different users.

While the developers of SMART Vaccines explicitly chose not to provide their own list of desired attributes, others have suggested the potential value of a consensus-informed list (and ranking) of a set of core attributes that could then lead to a convergent vaccine priority list. Further discussion of an initial effort in this direction appears in [29].

The risk of manipulation

An additional concern related to the notion that users would simply manipulate the value weights in SMART Vaccines to achieve the priorities and answers they most wanted. In fact, the transparency of the SMART Vaccines process argues the opposite. If people use SMART Vaccines in such a way, then it will be fully apparent how they set the weights, and others can use the same data with different weights to show how the manipulation occurred. The contrast to current practice is obvious: when one uses a single metric such as cost-effectiveness, the importance of other important factors remains hidden, and people or groups simply announce that taking these other factors into account, they arrived at their priorities. SMART Vaccines makes this wholly visible.

Group decisions

A number of stakeholders have pointed out that SMART Vaccines can be the basis for robust discussions about assumptions, data and values and allows for a collective decision process where a number of individuals or groups (each with different weights and value structures) are involved. Various approaches are available to combine individual preferences into group weights, all facilitated by the ability in SMART Vaccines to begin (and perhaps conclude) the process of setting weights by using a simple rank order list of the attributes’ importance. SMART Vaccines provides a well-tested method to convert rank order lists to weights as required by the model [30], [31]. Various methods to create rank order lists have been extensively analyzed in the realm of social choice theory and this literature can guide selection among alternative methods to create group rank order lists.

Conclusion

SMART Vaccines represents a major departure from previous approaches, and offers a novel platform to set priorities for vaccine development and deployment. The tool does not create a specific priority list, but rather provides a systems-based, adaptive prioritization tool that can guide and augment the planning process. It does not create a static list representing a single viewpoint, but instead, a software tool that—when augmented by appropriate population-specific data describing demographics, disease burden, available treatments, and attributes of potential vaccines—allows comparison of existing or newly emerging pathogenic threats from the differing perspectives of multiple users. Individual users or groups—by specifying their particular perspective (i.e., choosing attributes and their weights)—can create and promulgate a priority list representing their viewpoint, and can transparently modify that list under changing circumstances. SMART Vaccines can also facilitate choosing among alternative vaccine product profiles by formally incorporating the trade-offs between design options (e.g., three doses with higher effectiveness vs. one dose with wider coverage and lower costs of storage and administration). We believe that the ability to facilitate the focusing target product profiles is unique to SMART Vaccines.

SMART Vaccines can also help to facilitate decision convergence among people or organizations with differing viewpoints. By allowing use of common data and a common model for creating a final metric of value (the SMART Score for each candidate vaccine as viewed by each person or group), SMART Vaccines can clearly illuminate where differences emerge, and thus to illuminate pathways towards convergence and (perhaps) mutual agreement for a mutually-shared priority list. Following the release of SMART Vaccines 1.1, the National Vaccine Program Office has initiated follow-up efforts for subsequent versions.

Ultimately, we believe that the general systems analysis approach embedded in SMART Vaccines will have wider uses in other aspects of health care, public health planning, and related issues such as pollution control, ecology, and other health-affecting human behaviors. Moreover, SMART Vaccines has the potential to enhance the training of professionals to use and apply systems analysis in health policy [32].

At the beginning of this paper, we asked what strategic planning can do for the world of infectious diseases, and for vaccine development and deployment. Our answer contains three parts. First, systems analysis can help develop tools to clarify thinking and sharpen the ability to make well informed choices. Second, by calling attention to the relevant data needs, decision-support tools such as SMART Vaccines can accelerate the development of relevant data resources to support ongoing planning as new challenges emerge. Finally, systems-based planning can formally incorporate all of the issues that affect real-world decisions, making it fully apparent as to how these issues affect final choices. In so doing, SMART Vaccines and other systems analysis tools can help focus the attention of decision makers so that planning becomes truly a cohesive process rather than merely the production of static plans that risk becoming outdated at the moment of publication. Only then can we fully appreciate Eisenhower’s dictum: plans are useless but planning is everything.

Conflict of interest

None declared.