Pandemic H1N1 in Canada and the use of evidence in developing public health policies--a policy analysis.

When responding to a novel infectious disease outbreak, policies are set under time constraints and uncertainty which can limit the ability to control the outbreak and result in unintended consequences including lack of public confidence. The H1N1 pandemic highlighted challenges in public health decision-making during a public health emergency. Understanding this process to identify barriers and modifiable influences is important to improve the response to future emergencies. The purpose of this study is to examine the H1N1 pandemic decision-making process in Canada with an emphasis on the use of evidence for public health decisions. Using semi-structured key informant interviews conducted after the pandemic (July-November 2010) and a document analysis, we examined four highly debated pandemic policies: use of adjuvanted vaccine by pregnant women, vaccine priority groups and sequencing, school closures and personal protective equipment. Data were analysed for thematic content guided by Lomas' policy decision-making framework as well as indicative coding using iterative methods. We interviewed 40 public health officials and scientific advisors across Canada and reviewed 76 pandemic policy documents. Our analysis revealed that pandemic pre-planning resulted in strong beliefs, which defined the decision-making process. Existing ideological perspectives of evidence strongly influenced how information was used such that the same evidentiary sources were interpreted differently according to the ideological perspective. Participants recognized that current models for public health decision-making failed to make explicit the roles of scientific evidence in relation to contextual factors. Conflict avoidance theory explained policy decisions that went against the prevailing evidence. Clarification of roles and responsibilities within the public health system would reduce duplication and maintain credibility. A more transparent and iterative approach to incorporating evidence into public health decision-making that reflects the realities of the external pressures present during a public health emergency is needed.

Introduction

An effective public health response to a novel infectious disease will reduce the population impact. However, time constraints and uncertainty can compromise the response. This was seen with the 2003 Severe Acute Respiratory Syndrome (SARS) outbreak where policies were developed under scientific uncertainty due to lack of published evidence and limited experience with the virus (MacDougall, 2007). Some have suggested this uncertainty contributed to the failure to optimally control the outbreak in a timely manner in certain highly affected locations and resulted in unintended consequences such as lack of public confidence (National Advisory Commitee on SARS and Public Health, 2003). Given the history of serious pandemic influenza outbreaks and the emergence of the H5N1 avian influenza virus, concerns about a human influenza pandemic are paramount. As a result several governments and institutions have implemented extensive pandemic plans to be better prepared and minimize uncertainty.

On June 11, 2009 the World Health Organization (WHO) raised the pandemic alert to its highest phase in response to a novel strain of H1N1 influenza (Eurosurveillance editorial team, 2009). Despite extensive planning, the use of evidence to develop policies was complicated by several factors. Public health officials had to make decisions quickly, often when information was still being gathered. Coordination was necessary across multiple levels of public health (international, federal, provincial, local) and sectors (hospitals, schools, workplaces). Finally, pandemic policymaking has been shown to be influenced by political factors (Garoon & Duggan, 2008) and can be strained by antiviral and vaccine shortages, overwhelming demands on public health systems and intense public scrutiny (Timpka et al., 2009).

This study sought to examine the Canadian public health system response to the 2009 H1N1 pandemic (pH1N1). Our objective was to analyse the public health decision-making process and identify the factors that influenced the uptake and application of evidence for public health policy decisions.

Study design and methods

This policy analysis used a multiple case study design to examine policies at the federal, provincial and local levels of public health in Canada. Four policies were studied: (a) vaccine priority, (b) use of adjuvanted vaccine among pregnant women, (c) school closures as a containment strategy, and (d) recommendations on N95 respirators as a form of personal protective equipment (PPE) (Table 1 and Supplementary Table 1).

Table 1

Description of case policies studied.

Case policy	Decision required	Main issues
Vaccine priority	Which sub-populations were vaccine priority groups and the order in which they would receive the pandemic vaccine	• Supply, timing of availability (peak of second wave), jurisdictional roles and competing interests • Policies varied within Canada
Use of adjuvanted vaccine among pregnant women	Whether to switch vaccine production from adjuvanted to non-adjuvanted for pregnant women	• Pregnant women were at increased risk for complications and death from pH1N1 • Adjuvant used in the Canadian vaccine was a relatively new proprietal product and evidence about safety from randomized controlled trials was not available among pregnant women • Canada had already ordered adjuvanted vaccine, so switching production would affect its supply
Use of school closures as a containment strategy	Recommendations on whether or not to close schools	• In planning documents, school closures were a primary pandemic mitigation strategy • Limited evidence on effectiveness • Anticipated large societal costs and variability on recommendations
Use of N95 respirators as personal protective equipment (PPE)	Recommendations on the appropriate use of N95 respirators versus surgical masks to reduce infection risk among healthcare workers (HCWs)	• Research studies had yet to show the significance of airborne (in addition to droplet) transmission for pandemic influenza • Studies emerged that raised doubts on the ability of N95 respirators to prevent transmission of influenza among HCWs • N95 respirators were recommended pandemic plans

Key-informant interviews

Study participants were from five Canadian provinces (British Columbia, Alberta, Ontario, Quebec and Nova Scotia). An initial list of 41 key informants was developed by identifying officials in federal, provincial or municipal governments, provincial or local medical officers of health, members of a pH1N1 scientific advisory group, and clinical experts in infectious diseases and/or public health. A snowball sampling method was then used to identify an additional 23 participants. We initiated recruitment by email between July and September 2010 and all interviews were complete by November 2010. All consenting participants were given the choice on the policy of focus for the interview based on their role/responsibility during the pandemic. Many participants were involved in multiple policies and some spoke to more than one during their interview. Ethics approval for this study was obtained from the Health Sciences Research Ethics Board at the University of Toronto, Canada. All participants provided written informed consent.

The semi-structured interview guide was developed to capture the decision-making process with a focus on the use of evidence. All interviews were conducted by telephone by the principal investigator (LC) or research assistant (JT), recorded and transcribed verbatim. A second analyst (AC), present for all interviews, took additional notes and independently verified all transcriptions.

Document analysis

We sought public documents that detailed a pH1N1 policy recommendation related to the four policies. Primary documents included: guidance documents, statements of recommendations, interim or final government, advisory or scientific group reports and pandemic plans. Secondary documents included: information sheets, surveillance bulletins, journal publications and references from primary documents. Documents outlining recommendations were limited to those released between April 1, 2009 and May 31, 2010 while other documents were restricted to those released up to December 31, 2010. These date restrictions were in place to keep the analyses focused on pH1N1 and were determined by those responsible for monitoring these documents during the pandemic. Documents were identified through a search of an archive of published and grey literature related to pH1N1. Librarians developed this archive by performing daily local, national and international scans (www.ophla.ca). In addition, a direct search of selected public health and governmental websites was performed.

Coding and analysis

Data coding was used to abstract and synthesize data from transcribed interviews and policy documents using descriptive and iterative methods (Miles & Huberman, 1994; Yin, 1994). We used Lomas' model for decision-making (Lomas, 2000) to guide the analysis because it captures the interactive and multidirectional knowledge transfer that occurs in the decision-making process focussing on both evidence or knowledge and its transfer within a policy subsystem (Supplementary Fig. 1). It also emphasizes the importance of context-based decision-making incorporating three domains that influence the process: values, institutions and information.

We also considered Sabatier's advocacy coalition framework (ACF) as it similarly emphasizes beliefs and ideologies (Sabatier, 2007). However, this framework is less effective in describing the role of institutions. Dobrow's context and evidence utilization framework (Dobrow, Goel, & Upshur, 2004) is well-suited for examining contextual factors influencing the uptake of evidence, but has limited application to the policy and institutional subsystems where decisions take place, which was critical for pH1N1.

We also allowed thematic content to evolve independently of Lomas' framework. The principal investigator (the primary rater) developed the coding guide based on the framework and adaptations from its use in a previous analysis (Wilson et al., 2007). The interviews were distributed between the primary and secondary rater for independent coding. A subset of transcriptions was coded by both raters to evaluate inter-rater differences. In addition, the primary rater reviewed all transcripts and coding a final time to ensure consistency. A similar process was followed for the document analysis. We reviewed coded data for themes guided by the research objectives and domains in Lomas' framework as well as themes that emerged that were not part of the framework. NVivo 9 was used to assist with data collection, organization and analysis.

Findings

We conducted 40 interviews, including six pilot interviews, and included all in the analysis. The primary roles of participants during pH1N1 are summarized in Supplementary Table 2. Seventy-six relevant documents were identified with 12 covering more than one policy (Supplementary Table 3). We present our findings according to the Lomas framework and generalize themes overall, providing specific examples within policies in Table 2 .

Table 2

Main themes according to the Lomas framework by public health policy under study.

Influences	Public health policy decision
Values	Vaccine priority	Adjuvant pregnancy	N95 respirators	School closures
Ideologies of evidence
–Evidence-based	- Sequencing should be based on recent and well conducted research on likelihood and severity of infection - Scientific evaluation should be independent of groups considering logistical issues	- Limited randomized controlled trials (RCT) involving pregnant women - In epidemiological studies, pregnant women were at higher risk for serious complications if infected and thus needed protection	- RCT evidence suggested use of N95 respirators by health care workers was not necessary - Evidence supports droplet spread as the main mode of influenza transmission	- With limited pandemic experience, we have not had the opportunity to scientifically evaluate this intervention
–Policy-based	- Limited vaccine supply early in the immunization campaign was a key driver - Logistical issues related to vaccine delivery needed consideration	- Producing a non-adjuvanted vaccine due to theoretical safety concerns would delay production of the adjuvanted vaccine and therefore delay immunization program	- Regardless of the existing and emerging evidence, adhering to policies made prior to the pandemic is necessary to prevent labour disruptions	- Public/parents feared sending children to school when a case had been identified - School boards needed to consider the potential increased risk of employees working in school settings
Pragmatist	- Early epidemiological findings could facilitate timely determination of vaccine priority groups - Logistic considerations related to vaccine distributions should be balanced with epidemiological evidence	- Conflict between not making recommendations because of insufficient knowledge and risking serious complications among unimmunized pregnant women - Pregnant woman had died from pH1N1, yet current evidence did not demonstrate death or serious side effects from the adjuvanted vaccine; so it is reasonable to recommend whatever vaccine is available sooner	- Trade-off between labour disruptions versus upholding recommendation suggested by the evidence	- Modelling studies showed some benefit - Children would congregate in other places, similarly increasing risk of transmission - Resultant societal disruption
–Precautionary Principle	- Not a major influence	- No evidence of harm from the adjuvant, but in the absence of conclusive RCT evidence, supported a decision to offer non-adjuvant	- N95 respirators are superior in in vivo studies for airborne transmission - Health care workers deaths during the SARS hospital outbreak could have potentially been prevented with N95s	- No clear precautionary benefit to closing schools given lack of evidence on effectiveness, but potential for societal harms
–Interests	- Important to maintain public confidence and trust	- Pregnant women	- Balancing the interests of workers and infection control practices	- Interests of society (parents employers) and schools
–Beliefs	- Knowledge from previous pandemics and seasonal influenza were used to identify priority groups in pandemic plans, but did not directly apply during pH1N1 given the emerging epidemiology	- Any potential side effect associated with the adjuvant was theoretical	- Experience with SARS led to strong beliefs on need for N95 respirators	- School closures along with other social distancing and quarantine measures were identified as important in pandemic planning documents
Information and Information Purveyors
–Credibility	- Recommendations were made by an advisory group assembled solely for pH1N1; therefore credibility had not yet been established	- WHO recommended that pregnant women should be offered non-adjuvanted because of lack of data, creating a challenge for Canada because adjuvanted vaccine was already ordered	- An IOM committee recommended health care workers use N95 respirators when in close contact with suspected or confirmed cases of H1N1 or ILI - WHO and PHAC made the same recommendation only when in a room where aerosol-generating medical procedures are being performed, but recommended surgical masks when in close contact	- CDC’s recommendation to close schools had a strong influence
–Consensus and–Consistency	- Vaccine priority differed across jurisdictions - Variability in adherence to recommendations created challenges in public messaging	- Different recommendations between advisory groups, jurisdictions and over time created significant confusion	- Recommendations varied across jurisdictions	- Consensus from advisory bodies was reached early, but recommendations of other credible non-Canadian organizations created challenges
Institutional factors
–Formal structure	- Advisory groups evaluating evidence worked separately from those considering other factors, e.g. logistics;	- Regulatory body was at the federal level (Health Canada)	- Balance between patient safety, occupational safety and infection control	- Provincial and federal recommendations were made but local level school boards have the authority to make the ultimate decision
–Informal structure	- Prioritization strategies in other countries and recommendations from the WHO	- SOGC was the front line response for pregnant women; therefore played an important role	- Interpretation of the evidence was dependent on a groups' interests	- Excluding educational leaders from direct involvement in the decision-making process was seen as an omission - Consultation with parents and employers needed to understand the societal impact of school closures

Values

Ideologies

Three competing ideologies emerged with respect to the use of evidence: (1) the evidence-based ideology: science should be the first and foremost consideration in developing policies, and decisions made in the absence of good quality evidence can result in suboptimal policies; (2) the policy-based ideology: science is a minor component of the policymaking process and evidence is meant to inform policy rather than drive policy; and, (3) a hybrid or pragmatist ideology. Most participants associated the term evidence with the traditional research-based form of scientific evidence, which were primarily epidemiological in nature.

With a perspective derived from evidence-based medicine (EBM), participants with an evidence-based ideology placed considerable value on the peer-review process associated with scientific publications and conventional epidemiological study designs. Without a peer-review process, this group felt uneasy with proceeding to a policy decision. Furthermore, participants expressed frustration when science was re-evaluated in the policy arena, believing this compromises an evidence-based approach. Their key view is that evidence should be central and at the basis of any good public health policy decision, and that good evidence leads to good policy.

“[Information on public health measures] are published in grey literature or as evaluations or reports that were implemented… It's hard to trust in the same way because it's not peer-reviewed and not subjected to the same rigor.” (participant 25, scientific advisor)

This ideology ran into a major barrier when particular standards of evidence were deemed necessary to proceed. For example, the use of adjuvanted vaccine had not been shown to be harmful in pregnant women, but advisory groups had difficulty offering conclusive advice without evidence from randomized controlled trials (RCT) conducted in pregnant women.

“The largest and most glaring gap was data on vaccine in pregnancy. That's a major issue that we still haven't solved… we end up using vaccines and other things in pregnant women yet we exclude them from the clinical trials where they would provide the data on their safety and effectiveness.” (participant 40, scientific advisor)

In contrast, participants with the policy-based ideology shared the perspective that evidence is meant to only inform policy and not dictate policy. Under certain circumstances, they believed it was justified to give less priority to science in favour of other contextual issues. Participants with this ideology felt that scientists often lack expertise in policy-making. This ideology acknowledges that personal belief and ability to sway individuals also play important roles. Furthermore, the group stressed that the purely science-based approach over-simplifies the reality of developing policies since the determining factors were often contextual.

“in a way you generate knowledge that's no use to the field because it's inapplicable even though the science is very appropriate… you’re out of touch with the evidence mapping the policy.” (participant 69, public health official)

“In my view scientific advice into that [N95] decision would not have changed anything.” (participant 02, scientific advisor)

The pragmatist ideology can be seen as the centre of the continuum between evidence and policy-based ideologies. The participants who adopted this ideology felt science should be a primary consideration, but other factors including expert advice, clinical factors, logistics and ethics can be equally important and should be “respected.” This group was more likely to have an open-view of what constitutes evidence and tended not to make broad brush statements about either evidence or policy considerations, accepting that science can trump other considerations if compelling enough and vice versa. This group felt comfortable acting on the best available evidence and recognized that reaching consensus with respect to the science is often not possible.

“…science is a tool. We can debate the science for years but we have to make the best decision given the circumstance.” (participant 13, public health official)

“The decision-makers were all fully aware of the evidence and I think in a perfect environment would have chosen not to recommend the N95 masks. But they were working in the face of political and legal commitments as per the collective agreement… The evidence was available. It was considered but other factors ultimately drove the decision.” (participant 45, scientific advisor)

“Implementation considerations really are an incredibly important… confidence and public communication considerations are also really important considerations. They need to be respected every bit as much as the scientific and evidentiary considerations.” (participant 04, public health official)

Having not agreed upon definition of evidence results in further challenges in determining what counts as an evidence-based decision. While many participants support the pragmatic thinking, they recognize that current decision-making bodies lack the models to guide this in practice in a defensible way. In the current model, science evaluation and advisory recommendations are independent of decision-making. This separation has been recognized as essential to maintaining scientific integrity (Black, 2001) and is consistent with the evidence-based ideology. In situations where there was a lack of evidence, such as for the adjuvanted vaccine, it was difficult to proceed using an evidence-based ideology; therefore, other values and influences, such as the precautionary principle became of greater importance.

“It's another great example of the corner you get yourself into when you start making decisions that are not evidence-based. I mean the original decision to suggest that pregnant women should receive non-adjuvant vaccine, there was no evidentiary basis for that at all. It was one of these sort of theoretical precautionary principle sorts of things.” (participant 47, public health official)

“you can always say we’ll take the precautionary principle and let's just play it absolutely safe and just go with the non-adjuvant, even though it may be less effective and it was not available.” (participant 19, clinical expert)

Beliefs

In a pandemic there is uncertainty at the outset. As a result, prior beliefs on epidemiological patterns were very influential. Beliefs were well entrenched because pandemic planning has been an ongoing public health effort for several years. Beliefs were most strong with respect to disease severity, health care burden and transmission. Severity was expected to be moderate to severe, having the epidemiological characteristics of previous pandemics, particularly the 1968 Asian flu pandemic or seasonal influenza (Public Health Agency of Canada (PHAC), 2008). Participants also referenced the 1918–19 pandemic, avian influenza and SARS, indicating that historical memory played an important role in influencing beliefs. The health care burden was assumed to be high, having major resource implications and occupational safety concerns for health care workers. Transmission was assumed to occur through droplet spread with the possibility of airborne transmission. During pH1N1, emerging evidence that did not support these beliefs was more difficult to accept or incorporate into decision-making.

“One of the key considerations related to the N95 issue was the fact that we already had a pre-existing policy in place… that we would recommend that health care workers use N95 or the equivalent for protection… So you’re not going to suddenly mid-gear change your policy.” (participant 04, public health official)

“trying to change the way you respond during a time of crisis is very difficult.” (participant 51, public health practitioner)

Many participants felt that as pH1N1 progressed, attention was not paid to the severity of the disease soon enough and the subsequent scaling down of events to match the severity was delayed.

“we operated the same, whether or not it was a severe pandemic or not.” (participant 57, public health practitioner)

“We shouldn't have a war plan and stick with it right through to the bitter end, which I think is what we did this time. I actually don't know how you titre up or titre down but probably it's packaged with how we’re communicating so that there is some room to move back if the threat is less severe than you initially worried about.” (participant 38, scientific advisor)

Beliefs were also entrenched as a result of the SARS experience, despite SARS being a hospital-outbreak versus the pH1N1 community outbreak. The impact of SARS on health care workers influenced beliefs about the severity of H1N1 and how we should prepare and respond.

“Maybe over a long period of time and as things like SARS become more distant in the memories of people working in health care that might change things.” (participant 43, clinical expert)

Interests

Several competing interests factored into the use of evidence. Health care unions were interested in protecting their workers. Professional medical societies were interested in the population they serve (e.g. Canadian Pediatric Society (CPS) for children, Society of Obstetricians and Gynaecologists of Canada (SOGC) for pregnant women and educational boards needed to consider both parents and school employees). Scientific advisors were tasked with evaluating the science and providing an unbiased recommendation while staying true to the quality of the science and the needs of the overall population versus specific subpopulations. Policymakers in governmental institutions were faced with balancing these and several other interests. At different jurisdictional levels in Canada, provincial and local level decision-makers needed to balance science with the overall needs and contextual factors in their community.

“When we were looking at the dosage and the number of vaccines that needed to be given to children… both the pediatricians and the public health position were based on defensible science. But the same science was interpreted differently by the two groups depending on whether their major concern was for the one individual child in front of them or whether it was in terms of how many kids could we vaccinate in how short a period of time to give a population level of protection.” (participant 32, public health official)

“we compromised on the N95s because we felt okay, so it's better to have masks used potentially unnecessarily but actually have people willing to come to work rather than risk people refusing to come to work.” (participant 32, public health official)

Institutions

The complexity of Canada's executive, legislative and bureaucratic structures of public health complicated the policy process. This complexity has been noted previously to influence public health (Frank & Di Ruggiero, 2003). With multiple levels of action, participants reported signs of disconnect between those responsible for reviewing the evidence and those exercising decisions in their respective jurisdictions. In addition, participants stated that recommendations made at the international or national level often did not consider local issues. Internationally, the WHO and US Centres for Disease Control (CDC) were highly influential.

“the vaccine decision-making process was very complex because it involved multiple levels of public health from WHO, to PHAC, to the province and then down to the local health unit.” (participant 45, scientific advisor)

Informal institutional structures were also influential. Pandemic H1N1 was the focus of intense interest and pressure from multiple citizen and advocacy groups, particularly in relation to vaccination. Pandemic plans, recommendations from the WHO and decisions made in other countries also affected how evidence was used in Canada.

“Full consideration has been given to international activities and declarations and notices issued by the World Health Organization.” (.)

“The fact that some other jurisdictions, such as Mexico and the US, implemented measures such as school closures put more pressure on Ontario to adopt similar measures.” ()

Information and information purveyors

Information producers included scientists/researchers, advisory groups and committees/working groups. Participants frequently expressed uncertainty about who was responsible for answering what questions. This resulted in confusion on where to look for information, particularly when differing recommendations were based on the same evidence.

Information purveyors included: government, professional organizations, media and advisory organizations with and without decision-making capability at local, provincial, federal and international levels. A common theme was that the credibility of information purveyors was paramount and needed to be pre-established. Scientific advisory groups formed during pH1N1 did not have the opportunity to establish their credibility, making their recommendations difficult to accept. For example, the National Advisory Committee on Immunization (NACI) is responsible for scientific guidance on immunizations; however, another vaccine advisory group was created specifically for the pandemic and NACI not consulted until later on.

“It's no good having a scientific decision-making body that only exists in the setting of emergencies or one that is sufficiently far from government that they’re not looking at that advice and using that advice on a regular basis.” (participant 02, scientific advisor)

Credibility also posed challenges at times because recommendations coming from an established and credible organization were difficult to challenge or modify (i.e. WHO). Along with credibility, themes emerged around the challenges in achieving consensus on scientific evidence. Participants were particularly concerned about the significant amount of time spent trying to achieve consensus, commenting that long debates over scientific merits under time constraints is not helpful. It was recognized that consensus is often not feasible and therefore, different models of consensus should be considered.

“a cabinet approach, which is that you may disagree with the final decision but you go with it and you live with it.” (participant 18, public health official)

When contradictory or unclear messages were given, the ensuing confusion resulted in decreased public confidence and increased time invested to rationalize the differences.

“[WHO] made a recommendation that pregnant women should be offered non-adjuvant vaccine preferentially because there wasn't much data on the use of adjuvant, or so they claimed, in pregnant women, which set the tone for Canada… we had already ordered adjuvant vaccine for our population. So that was a bit of a challenge for us.” (participant 04, public health official)

“The untimely and ambiguous recommendations for dosing in pregnant women and young children from national and international sources put Alberta in a vulnerable position where credibility of its public health decision-makers was jeopardized because of issues beyond their control.“ ()

One of the most influential information purveyors, the media, was looked at in an overall favourable light. Unlike previous public health events, such as SARS where the media reaction was sensationalized and disruptive at times (Lewison, 2008; Washer, 2004), participants felt that most of the time the media took a balanced and responsible approach in their reporting by seeking appropriate advice from credible experts.

“I don't think the media drove any of our policies, other than forcing us to try and be as clear in our communications as possible.” (participant 32, public health official)

However, at certain points during the pandemic, such as around the immunization campaign, it was perceived that media coverage did have a strong influence.

“initially [the media] influenced the public perception of the vaccine… Originally it was in a negative way so that there were questions to the safety and the need for the vaccine. But then subsequently after the death of [child], it was for demand of the vaccine. Then all of the media attention on the lines and the long waiting times just added public pressure to go beyond the priority groups that were determined by the policy.” (participant 41, scientific advisor)

Academic journals were also cited as an influential purveyor. Study participants acknowledged that journals made an attempt to speed up their review and publishing of pandemic studies and made articles freely available. Nonetheless, the process was critiqued by some who commented that at times it disrupted the flow of information.

“there wasn't an awful lot of science being published about pandemic H1N1 in a timely enough fashion to affect decision-making. … So it would be really good to have a fast track peel-off kind of system of public access where people can review and it doesn't necessarily preclude subsequent publication …. because scientists need to be able to do that.” (participant 32, public health official)

Views on the sufficiency of information were divergent. Many participants expressed the desire for more information/evidence despite an acknowledgement that decision-makers may not be equipped to handle new or emerging information, even if the scientific quality is high.

Decision-making process

Several additional themes emerged that were not explicitly captured by the Lomas framework. Comments on the use of evidence revealed concerns around transparency (i.e. operating in such a way that it is easy for others to see what actions are performed) and explicitness (policies fully and clearly expressed, leaving nothing implied). Specifically, because criteria for the inclusion or exclusion of evidence were unclear, decision-making structures themselves were described by participants as being a mystery at times. Some participants not at the decision-making table were not confident in their ability to accept and implement recommendations because they were not privy to the decision-making structure. Scientific advisors wanted to know how scientific recommendations provided were ultimately used and balanced with other factors. If advice was not explicitly referenced, those providing the scientific advice assumed it was not considered. Some felt that decision-makers often do not cite other factors, such as logistical, political and social issues as rationales because those issues are perceived as less justifiable compared to science. This lack of explicitness can result in the unintended consequence of decisions appearing to be made without appropriate justification.

“There wasn't a lot of transparency in those decisions or at least feedback to the people who were making the recommendations… It's hard to know whether they also considered the evidence… or whether there were some other factors that were forcing them to go against the recommendation.” (participant 25, scientific advisor)

A clear documentation of the process would serve to make the process more explicit and satisfy concerns raised.

“the bottom line is a decision has to be made… You have to sometimes go right down to consensus and expert opinion or extrapolation from other types of evidence or other categories of evidence to make your recommendation… I think there has to be rigor that goes along with that and documentation and clarity on the reasoning.” (participant 45, scientific advisor)

Several participants commented on the multiple parallel processes which took place. Not only were there parallel processes for decision-making at multiple levels, but the generation/evaluation of evidence also happened at multiple levels and sometimes even within the same jurisdiction or institution. Such duplication contributed to inefficiencies and confusion in the recommendations.

“all of us were scrambling across all of our different domains to understand as best as we could what could be learned at any given moment. In retrospect it was so fundamentally inefficient.” (participant 16, scientific advisor)

Participants noted that First Nations groups were at increased risk of mortality and morbidity and warranted vaccine priority, but overall did not speculate on the broader causes of the disproportionate mortality and morbidity. This issue was not the focus of the study or the policies examined, which could be an explanation for why this was not discussed in more detail by the participants.

Synthesis

In contrast to previous public health events such as SARS (National Advisory Commitee on SARS and Public Health, 2003), the Creutzfeldt–Jakob disease tainted blood scandal (Wilson et al., 2001) and the Walkerton Escherichia coli outbreak (Schuster et al., 2005), a unique aspect of pH1N1 was the advance planning. Substantial investments had been made in pandemic planning since SARS, and the identification of H1N1 in Mexico in early 2009 initiated further preparations for its anticipated spread. Advance planning did minimize uncertainty, but it also generated assumptions that created conflict when handling emerging evidence. Furthermore, evidence was interpreted differently according to the three ideologies of evidence.

According to Fishbein and Ajzen's theory of reasoned action, pandemic plans are manifestations of the intentions for a pandemic, a conscious action plan which functions as a major determinant of future behaviour (Fishbein & Ajzen, 1975). Pandemic plans are a complex collection of assumptions based on various types of evidence. This creates a box in which policymaking begins and sometimes stays. As the outbreak unfolded it became apparent that data on severity did not match the models and underlying assumptions of most plans. Decision-making, however, continued to be framed by these models. Pandemic plans are intended to provide practical guidance and not indoctrinate ideology; however, this analysis shows that the latter prevailed. This finding is consistent with the theory of cognitive dissonance (Festinger, 1957) which explains the difficulties in adjusting when there is a discrepancy between prior held beliefs and new information based on the existing situation.

Other decision-making theories can offer insight into policy choices made despite the evidence. Challenging the content of the pandemic plans, for example on the use of N95 respirators, was avoided in certain jurisdictions given the significant conflict that would arise. This is consistent with conflict avoidance as described by the conflict model of decision-making (Janis & Mann, 1977). A decision that involves a potential conflict provokes a degree of stress, the excess or absence of which is in turn a major determinant of the final decision. The N95 decision in Ontario was a result of “defensive avoidance” such that the risks to labour disruption would not be worth challenging a recommendation from the pandemic plan. The school closure policy went against a priori guidance, which largely endorsed school closure as a containment strategy. According to the conflict model, the (largely societal) risks associated with school closures outweighed the risks of staying with the decision. These two policies contrast in how differences in antecedent conditions, available evidence and resulting decisional-conflict, guided the policy actions. It is possible that the nature of the situation, and not theory, explained the N95 decision as the political and non-evidentiary considerations were considered legitimate. Compared with N95, very little epidemiological evidence existed for school closures with support mostly coming from modelling studies. However, the potentially large societal costs of closing schools supported a change in direction. The vaccine sequencing decisions also largely agreed with pandemic plans although the manifestation of the policy was extremely heterogeneous across Canada and within provinces. With the adjuvanted vaccine for pregnant women, little to no a priori information existed and therefore cognitive dissonance did not apply. Instead, the precautionary principle was largely used to justify decisions.

There are some limitations to this study. As interviews were conducted a few months after the pandemic was declared over, recall bias is a possibility. Verification of our findings with the document analysis is not susceptible to this recall bias and supported similar themes around the role of beliefs, information purveyors and ideologies. Nonetheless we acknowledge that hindsight may influence responses. All participants work within the public health or health care system. As a result, the views of individuals outside the public health system were not represented. While this is appropriate given our research question, these findings represent only one particular stream of perspectives. One of the limitations of using the Lomas framework in this analysis is the unidirectional flow between information domains and institutional structures for decision-making. During pH1N1, this exchange was multidirectional and several individuals were both producers of information and decision-makers within the formal structures, making it difficult to conceptually separate their unique influences. Lomas' framework also does not capture Canada's hierarchical, jurisdictional and institutional levels of decision-making and their interactions. Finally, the placement of evidence ideology in the values component of the model can be questioned as it is reflective of a normative stance versus a value trait.

Conclusions

Rationalism demands that decision-makers choose the best alternative from a set of options (Betsch, Haberstroh, & Hohle, 2002). However, under the congruence of multiple pre-planning and entrenched assumptions, the precautionary principle and evidence-based paradigms, a complex situation emerges whereby the decision-making process cannot be easily tracked or justified according to a straightforward set of criteria. Pre-planned approaches have many benefits including clarifying accountability and strategy during uncertainty. However, caution needs to be exercised when pre-planned approaches are in place to ensure policy options are not constrained by plans. We suggest that pandemic plans need to be written and used in a way that accommodates dynamic responses to fit an emerging situation. This includes minimizing scenarios based on single assumptions and conducting a range of sensitivity analyses.

An emergent public health situation that affects a large proportion of the population will necessarily require a balance between various factors. Both evidential and policy considerations are important and improvements in addressing both effectively and explicitly within decision-making processes are needed. We recommend an iterative scientific review that re-considers science/evidence based on the policy landscape. The challenge with this approach is that in an EBM paradigm this can be thought of as compromising evidence. We have noted throughout our analysis that this is already happening; therefore, being more explicit about the process will allow for more critical investigation. Functionally, public health needs a model that can support these types of decision constructs, but at the same time ensure rigor. One modifiable construct that would be beneficial to amend going forward is to reconsider what constitutes evidence. This analysis suggests a more comprehensive and inclusive concept of evidence that recognizes the variety of contexts in which evidence is generated and used (Upshur, VanDenKerkhof, & Goel, 2001), and that is more appropriate during an emerging public health event. This can be facilitated by increased collaboration and discussion among researchers and policymakers to ensure the relevant evidence is generated. Such collaborations would also facilitate greater credibility of information purveyors, which is particularly important when evidence does not fit prior expectations.

Credibility, transparency and explicitness in the decision-making process will improve understanding and uptake of policy decisions. In particular, our analyses revealed that when and why evidence/advice is or is not considered needs to be clearly articulated. This includes outlining steps, logic, key assumptions, limitations and trade-offs considered, evidentiary or otherwise. The role of ideological perspectives and prior beliefs should also be made explicit given how influential they are in defining decisions from the outset. We acknowledge that detailed recording of evidence-based or consensus-derived decisions to ensure transparency may be idealistic in the face of time and other pressures during a crisis. One possible recommendation could be a more rigorous use of Incident Management System (IMS) processes which would serve not only to improve processes but also to ensure better documentation. At the minimum, critical post-outbreak analyses of detailed real-time records of the meetings and transcripts should occur to improve future public health actions. Finally, with many different groups addressing the same questions, inefficiencies still exist. The public health system needs to leverage existing relationships rather than create new structures during a pandemic. In addition, better clarity on who is responsible for answering what questions is needed to avoid duplication and mixed messages. Addressing inefficiencies will require trust and collaboration, supporting the themes identified around credibility.

This analysis reflects the tension that occurs during emergency public health decision-making and highlights several important considerations for public health planning. Going forward, we must ensure that our decision-making structures evolve in light of the challenges experienced during pH1N1.

Funding/support

This study was supported by the Institute of Population and Public Health and the Knowledge Synthesis and Exchange Branch of the Canadian Institutes of Health Research (CIHR). LCR was supported by a CIHR postdoctoral fellowship in public health policy. KW is supported by a Canada Research Chair in public health policy.

Disclaimer

The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by the Ontario Agency for Health Protection and Promotion is intended or should be inferred.