Using structured expert judgment to assess invasive species prevention: Asian carp and the Mississippi-Great Lakes hydrologic connection.

Recently, authors have theorized that invasive species prevention is more cost-effective than control in protecting ecosystem services. However, quantification of the effectiveness of prevention is rare because experiments at field scales are expensive or infeasible. We therefore used structured expert judgment to quantify the efficacy of 17 proposed strategies to prevent Asian carp invasion of the Laurentian Great Lakes via the hydrologic connection between the Mississippi and Great Lakes watersheds. Performance-weighted expert estimates indicated that hydrologic separation would prevent 99% (95,100; median, 5th and 95th percentiles) of Asian carp access, while electric and acoustic-bubble-strobe barriers would prevent 92% (85,95) and 92% (75,95), respectively. For all other strategies, estimated effectiveness was lower, with greater uncertainty. When potential invasions by other taxa are considered, the effectiveness of hydrologic separation increases relative to strategies that are effective primarily for fishes. These results could help guide invasive species management in many waterways globally.

Introduction

Recently, authors have argued that prevention is the most effective strategy for protecting native biodiversity and avoiding unwanted economic damages from invasive species.[1,2] Further, prevention focused on dispersal pathways (e.g., waterway connections, canals, ships’ ballast, organisms in trade) is likely to be more cost-effective than that focused on individual species because a given prevention strategy can be effective against multiple species.[3,4] However, for most pathways, little information exists on the effectiveness of prevention strategies[5] because of the expense or infeasibility of experimental tests at realistic field scales. Therefore we use structured expert judgment (SEJ)[6] to evaluate the uncertainty around the effectiveness of alternative strategies to prevent invasions through the Chicago Area Waterway System (CAWS) that connects the Mississippi River and Laurentian Great Lakes watersheds (Figure 1).

Figure 1

The Laurentian Great Lakes-St. Lawrence and Mississippi River basins. The Chicago Area Waterway System (CAWS; white star), located in northern Illinois at the southern tip of Lake Michigan, is a man-made hydrological connection of the two watersheds. Map by S. R. Mysorekar, The Nature Conservancy.

While human-made waterway connections have clear socio-economic benefits, these hydrologic connections are often also conduits for species invasions and disease outbreaks worldwide.[5] In Europe, canals are the most important pathway for aquatic nonindigenous species along the Rhine River.[7] In North America, the canals that connect the Mississippi River and Great Lakes basins have enabled the interbasin passage of a number of species such as sea lamprey (Petromyzon marinus) and zebra mussel (Dreissena polymorpha)—which have significantly damaged ecosystems in both watersheds.[8,9] As the Great Lakes and Mississippi River basins have the highest diversity of freshwater fishes[10] and mussels[11] in the world, the continued interbasin transfer of nonindigenous species presents a major threat to global aquatic biodiversity.

Two of the species of greatest concern as potential invaders into the Great Lakes from the Mississippi River basin via the CAWS are Asian carps (bighead; Hypophthalmichthys nobilis, and silver; H. molitrix).[12] Originally introduced in Arkansas in the 1970s, their invasion throughout the Mississippi River Basin has been characterized by exponential increases in biomass and damages to ecosystems, human safety, and fisheries.[12] Should Asian carps gain access to the Great Lakes, there is potential for establishment and concern they may change food webs and significantly impact the multibillion dollar fishing industry.[13,14] To prevent passage of Asian carps into the Great Lakes via the CAWS, the U.S. Army Corps of Engineers operates a system of electric barriers in the CAWS.[15] Other prevention strategies are also under consideration, including physical hydrologic separation of the Mississippi and Great Lakes basins, which would cost $3.5–9.5 billion USD.[16]

Whether any fish deterrent strategy can provide an effective and long-term method for preventing passage between watersheds is uncertain.[17] Research on fish deterrence technologies for Asian carps has been conducted in laboratory confinements, outdoor raceways, and a small river system, with limited, short-term studies in the CAWS.[17−19] The empirical information required to rigorously evaluate the long-term effectiveness of prevention strategies at the relevant field scale does not exist and is likely not feasible to acquire with field experiments. Further, the time required to develop, implement, and conduct multiple large-scale field trials—even they were feasible and ethical—might preclude the utility of this information because an invasion may occur within that time frame. Despite extensive analysis and review, Great Lakes natural resource managers remain unsure as to the optimal approach for Asian carp prevention between the Mississippi and Great Lakes watershed.[20]

With this study we use a performance-based method of SEJ, which aggregates expert knowledge to quantify the uncertainties associated with invasive species pathway management.[6] SEJ has been widely used in a variety of applications including consequence assessment for chemical substances, nuclear accidents, probabilistic hazard and risk assessment and increasingly in conservation biology.[21−25] SEJ has been successfully used to quantify uncertainty in scenarios when scientific consensus is confounded by divergences of opinion or lack of data. Thus providing the information required for decision makers to reduce economic damages and in some cases, loss of human life associated with urgent hazards.[25] Specifically, we use SEJ to assess the efficacy of alternative strategies to prevent Asian carp dispersal via the CAWS, one of the largest human-made water connections in North America. This analysis not only provides important information for the management of the CAWS, but results are also applicable to many canal systems globally. In addition, we provide a novel and tractable approach for using SEJ to address the management of species invasions under great uncertainty.

Materials and Methods

Several approaches are available for the elicitation and aggregation of individual experts’ judgments, most of which seek to find homogeneity or create a single combined aggregate of a group of expert assessments.[26] Here we used a structured, performance-based method to elicit and aggregate expert judgments (“the Classical Model”).[6] Assessments by individual experts were obtained with 2–3 elicitors present, in which each expert quantified his response as the 5th, 50th, and 95th percentiles of his subjective probability distribution for each target variable. A typical assessment lasted four hours. Each expert was then given different scores according to individual performance on a set of calibration variables, that is, variables from the experts’ field of specialization whose values were not known at the time of elicitation, but were realized post hoc. For example, we asked experts to quantify the Asian carp biomass removed from the CAWS by commercial fishing crews during 2012; the effort was ongoing at the time of elicitation, and results were published afterward by the U.S. Army Corps of Engineers (USACE) in 2013. Expert assessments were then aggregated into a single combined outcome, determined through methodologies described below.

To select experts, we identified individuals with expertise in fisheries biology and specific experience in the Great Lakes, or Asian carp species, or both. From this list, we contacted 11 experts (Table 1), all of whom agreed to participate. The choice of the number of experts is supported by simulations showing that adding experts above 10 does not lead to increased performance of combined assessments.[27] Prior to an in-person interview, each expert received the elicitation questionnaire (Supporting Information part A), background information concerning expert elicitation, and a booklet containing biological information about the Great Lakes, and bighead and silver carp (all materials are available upon request). Experts were encouraged to utilize any additional information sources or considerations not included in the provided background information or booklet that would help them estimate their responses. We interviewed each expert individually, in person, between May 1 and June 15, 2012. Each interview began with a brief presentation explaining the goals of the study, a statement of assumptions, and an explanation of calibration and target variables.

Table 1

List of Experts Elicited with Title, Affiliation, and Brief Description of Expertisea

name	title, affiliation and expertise
Duane C. Chapman	Research Fish Biologist, United States Geological Survey, River Studies: Invasive Carp Research Program. Chapman is affiliated with the Asian Carp Working Group, Asian Carp Rapid Response Team, Mississippi River Basin Panel on Aquatic Nuisance Species and the American Fisheries Society.
Joseph V. DePinto, Ph.D.	Senior Scientist, Limnotech. A former professor of environmental engineering, DePinto conducts aquatic ecosystem structure and functioning research, and designs education and management programs, with emphases on the Great Lakes region.
Tomas O. Höök, Ph.D.	Assistant Professor of Fisheries and Aquatic Sciences, Purdue University, Department of Forestry and Natural Resources. Focuses on fish and fisheries ecology in the Laurentian Great Lakes.
Timothy B. Johnson, Ph.D.	Research Scientist, Ontario Ministry of Natural Resources, Great Lakes Fisheries Division. Johnson’s expertise is in bioenergetics models, specifically for Lake Erie, and has studied the biology of invasive round goby.
Roger L. Knight	Lake Erie Fisheries Program Administrator, Ohio Department of Natural Resources, Division of Wildlife. Serves on the Lake Erie Committee and the Council of Lake Committees (Great Lakes Fisheries Commission).
Stuart A. Ludsin, Ph.D.	Assistant Professor, The Ohio State University Department of Evolution, Ecology and Organismal Biology. Ludsin’s expertise is on mechanisms that regulate fish population and community structure and dynamics, food web interactions and natural resource management.
Charles P. Madenjian, Ph.D.	Research Fishery Biologist, United States Geological Survey, Western Basin Ecosystems Branch, Lake Michigan Section. Madenjian is a quantitative fisheries biologist and has focused on fish bioenergetics modeling in the Great Lakes.
Peter Meisenheimer	Executive Director, Ontario Commercial Fisheries Association. A biologist who represents commercial fisheries in Ontario, currently a member of the Canadian Committee of Advisors of the Great Lakes Fishery Commission and Chair of the Ontario Species at Risk Public Advisory Committee.
Mark A. Pegg, Ph.D.	Associate Professor, School of Natural Resources at the University of Nebraska Lincoln. Pegg specializes in fisheries management, the impacts of aquatic nuisance species including Asian carps, and restoration ecology.
Kevin Reid	Ph.D. candidate, University of Guelph, and Assessment Manager and Fisheries Biologist-Technical Advisor Ontario Commercial Fisheries Association.
Brian J. Shuter, Ph.D.	Professor, Department of Ecology and Evolutionary Biology, University of Toronto and Research Scientist Aquatic Research & Development Section Ontario Ministry of Natural Resources. Shuter focuses on food web dynamics, population ecology and growth/production models for fish and zooplankton.

The experts are listed in alphabetical order which does not correspond to responses presented in this study.

The elicitation questionnaire consisted of 84 questions about bighead and silver carp establishment in the Great Lakes. For the present study, we focused on a subset of questions about the efficacy of 17 Asian carp deterrence strategies proposed for use in the CAWS (Table 2).[13] We asked experts to quantify the percentage of Asian carps that would be prevented from accessing Lake Michigan as a result of implementing one of the 17 deterrent strategies at a time. Of the 84 questions in the overall questionnaire, 20 were calibration variables used to evaluate the performance of each expert. Experts were aware which questions were calibration questions.

Table 2

Physical, Chemical, And Thermal Deterrent Strategies Proposed for Asian Carp Prevention in the Chicago Area Waterway System (CAWS), Strategies and Descriptions Based on USACE (2010)

deterrent or barrier method	description
hydrologic separation (SEP)	permanent separation of Great Lakes and Mississippi River basins
electric (ELE)	maintaining a direct current electric field in the water column
thermal (THR)	maintenance of high or low water temperature
CO2 barrier (CO2)	maintenance of elevated carbon dioxide concentrations
hypoxia (HYP)	maintenance of hypoxic conditions
chlorine (CHL)	maintenance of chlorination
pH (pH)	manipulation of ph
hydrogun (GUN)	use of hydrogun, seismic cannon, sonic cannon, or similar devices
pheromones (PHE)	as a repellant, or as an attractant, to contain and/or control
block net (BLK)	nets to physically block passage
strobe light (STR)	using deterrent flashing lights
bubble curtain (BUB)	pumping compressed air through a diffuser to create a continuous dense deterrent curtain of bubbles
acoustic curtain (AC)	arrays of underwater loudspeakers or sound projectors to produce a diffuse omni-directional deterrent field of sound
acoustic + bubble (AB)	combination of bubble and acoustic barriers
bubble + strobe (BS)	combination of bubble barrier and strobe light
acoustic + strobe (AS)	combination of acoustic barrier and strobe light
acoustic + bubble + strobe (ABS)	combination of acoustic, bubble and strobe

The product of the calibration and information scores determines the performance-based weight that each expert receives. The calibration score is the probability that the divergence between the expert’s probabilities and calibration variable realizations might have arisen by chance. For the set of calibration variables, each expert provides quantile information to test the hypothesis H0: “This expert is well calibrated”. If an expert gives 90% confidence bands for the set of calibration variables, then it might be anticipated that about 10% of actual realizations (e.g., the true outcomes of the calibration variables) will fall outside his chosen bands. Thus, for an expert assessing 20 calibration variables for which realizations become known post hoc, three or four outcomes outside the confidence bands is expected. However, if 10 or more of the 20 variables fell outside the expert’s bands, it is not likely that so many outliers resulted by chance. If an expert is perfectly accurate statistically, then his/her calibration score is uniformly distributed on the [0,1] interval, where scores near 1 indicate strong agreement between the expected number of realizations falling into the inter quantile intervals, and scores near zero indicate strong disagreement. In this sense, low scores are critical. In classical hypothesis testing, on which the Classical Model is based, a score of 0.05 would be sufficient to reject the hypothesis that an expert is statistically accurate.

The information score is the degree to which the expert’s distribution is concentrated, relative to a user-selected background measure. This is to say that the width of an expert’s confidence band determines his or her information score. This information score is calculated for each expert at each variable and then averaged over all variables to get the overall information score. See Supporting Information part B for further details on performance measures and calibration.

In addition to evaluating each expert’s performance, we also derived two alternative aggregates of expert responses: equally weighted (EQ), in which the score of each expert is weighted equally; and performance-based (PB) in which weighted experts (those whose calibration score is above a specified cutoff level, α (see Supporting Information part B for derivation of α) are pooled. Additionally, we recorded notes on what each expert revealed during the interview about the rationale associated with quantitative responses provided for each target variable.

Expert names and affiliations are provided herein, but not associated with their responses. Thus, ordering of the experts presented in Table 1 is alphabetical and does not correspond to the ordering of results. The identification of responses associated with individual experts are privately maintained by the authors of this study. All analyses were carried out using EXCALIBR,[28] software that is freely downloadable at http://risk2.ewi.tudelft.nl/.

Results

The experts’ calibration scores ranged from 2 × 10–6 to 0.53, with 9 of 11 experts scoring above 0.05 (Table 3). Higher calibration scores indicate better performance in assessing the calibration variables accurately. All experts assessed 15 calibration variables, with the exception of Expert 8 who assessed only 11, reducing the effective number of items for all experts to 11. The EQ had a calibration score of P = 0.3126 (Table 3), indicating that we would not reject the hypothesis that EQ’s probability assessments were accurate. Both weighting schemes (EQ, PB) returned good calibration and high information (mean relative information; all variables: 0.5789, 3.798). Higher mean relative information scores indicate better performance in assessing the calibration variables precisely. The PB combination was better calibrated and more informative than the EQ combination; therefore we put the greatest emphasis below on PB results. The PB combination was comprised of responses from a single expert (expert 4) due to this expert’s high calibration (Table 1) and the strictly proper scoring rule (see Supporting Information part B).

Table 3

Performance and Combination of Expert Judgmentsa

		mean relative information
expert ID	calibration score	all variables	calibration variables	normalized weight
1	0.1815	1.395	0.6121
2	0.1227	0.677	0.6648
3	0.0056	2.832	1.4700
4	0.7606	3.798	0.8562
5	0.6660	2.148	0.8400
6	1.93 × 10–06	1.481	1.3810
7	0.0595	1.839	1.1580
8	0.6150	4.334	1.0860
9	0.5276	2.547	1.2880
10	0.2587	2.603	0.8282
11	0.5276	2.517	0.8071
EQ	0.3126	0.5789	0.2943	0.0920
PB	0.7606	3.798	0.8562	0.6513

Calibration score (2nd column) is the likelihood that the realizations of calibration variables correspond with the expert assessments. Mean relative information measures the degree to which an expert’s uncertainty distribution is concentrated around the true answers to a set of variables (either to all variables (3rd column), or to the calibration variables 4th column). Decision makers: EQ = equally weighted combination, where all experts’ responses are pooled with equal weights; and PB = performance-based combination, where experts are weighted according to a selected cut-off level for calibration for which the normalized weight (5th column) of the combination is maximal.

Both combinations (PB, EQ) indicated that hydrologic separation at the CAWS is the most effective prevention strategy, with median estimates of 99% for the PB and 98% for the EQ result (Figure 2a,b). Uncertainty ranges (5, 95th percentiles) for the PB (95, 100) and EQ (78, 100) results for hydrologic separation were all small compared to any other prevention strategy considered, indicating high certainty among experts.

Figure 2

Equally weighted (EQ; a) and performance-based (PB; b) expert assessments of the percentage of Asian carp prevented access to the Great Lakes as a result of implementing 17 proposed fish deterrent strategies in the Chicago Area Waterway System (strategy acronyms from Table 2). Dots indicate median; lines denote the 5% and 95% intervals.

Similarly, both combinations indicated that electric barrier was the second most effective strategy, ranging from a median of 88% to 92% of Asian carps prevented. The PB indicated that the acoustic-bubble-strobe (ABS) combination of deterrent technologies was similarly effective to the electric barrier option, with a median result of 92% prevented, but with a wide uncertainty range in comparison to electric barrier or hydrologic separation (Figure 1b). The equally weighted median result for ABS indicated a much lower proportion of fish prevented (60%). Pheromone treatment was indicated as the fourth best option according to the PB, with CO2 and ABS being similarly low in their expected effectiveness for the EQ decision maker. The similarity in the PB decision maker among the third to sixth approaches (THR, CO2, HYP, CHL, pH, GUN) is due to this combination being comprised of the responses of a single expert, whose estimates of uncertainty for all of those strategies were equivalent. This suggests that the expert did not view the effectiveness of any of these strategies as being meaningfully distinguishable from one another.

According to experts, none of the fish deterrent strategies would be 100% effective in preventing Asian carp passage between watersheds. However some methods were estimated to be more effective than others. All experts indicated that hydrologic separation would be the most effective at reducing the greatest proportion of Asian carps, due to a lack of mechanical failures other than extreme flooding events. All experts also acknowledged and considered as part of their rationale that Asian carp introduction to Lake Michigan could occur through other pathways (e.g., bait contamination, movement by birds).

Experts indicated that an electric barrier—the only method currently in place—is likely to be less effective than hydrologic separation. This expectation was attributed to the possibility of mechanical failures of the electric barrier, which is also expected for other strategies that require a constant power source (i.e., ABS, thermal). Likewise, in situ materials such as block nets and associated maintenance were expected to be subject to breakdown. Experts indicated that chemical, thermal or other curtain barriers (i.e., chlorine, bubble, acoustic, strobe) were highly dependent on two factors: the ability to consistently maintain the barrier with appropriate magnitude, and the behavioral responses of fishes. Variation in hearing or vision of Asian carp, their tolerance to physical or chemical cues, and habituation to these cues were identified as the main sources of failure even if all barriers functioned full time.

Discussion

Uncertainties about the success or failure rates of species prevention strategies may lead to delayed decision making by managers or policy makers—potentially enabling further damages from invasion.[29,30] Here we have provided a previously missing and key aspect of the information needed for risk management—the expected effectiveness of prevention strategies. Experts identified hydrologic separation as the most effective strategy for preventing Asian carp passage, with alternative prevention strategies having greater uncertainty around lower estimated effectiveness. SEJ provides a relatively rapid and transparent way to provide essential information to managers and policy makers. Given the similar biological and engineering issues present in many of the world’s canals, the experts’ responses here may also be useful to the management of waterways elsewhere.

To manage risk, natural resource managers often seek to evaluate the value of any management action against the likelihood of invasion, and the potential damages to ecosystem services that may be caused by the invader.[31] While this study identifies and quantifies the uncertainties associated with the effectiveness of Asian carp deterrent strategies, it does not address three important aspects of any species invasion that managers would desire: (1) the relationship between the number of Asian carps (i.e., propagule pressure) and the probability of establishment, (2) whether Asian carp, if established, would cause ecological damage in the Great Lakes, and (3) the costs associated with the alternative prevention strategies we considered.

The CAWS is the primary focus of efforts to prevent Asian carp introduction into the Great Lakes because the CAWS provides a direct connection to the Illinois River, home to the densest populations of Asian carp known to exist.[32] Proximity to a source of dispersing individuals increases propagule pressure, and therefore the likelihood that an invasive species will establish.[33,34] However, it is the dynamics of initial populations that actually reach the Great Lakes that would determine establishment success.[35] Demographic stochasticity, density dependence and interaction with the environment are some factors that complicate the ability to estimate invasion risk from propagule pressure alone.[36,37] Despite evidence that abundant spawning habitat, food availability and suitable climate exist for Asian carps in the Great Lakes,[12,38,39] major uncertainties remain about the likelihood of how persistence, survival and recruitment within the Great Lakes would be quantitatively related to propagule pressure. It is unlikely that additional empirical work can be done at appropriate spatial scales to resolve this question within a time useful to management decisions.

Whether Asian carp would cause significant ecological or economic damage in the Great Lakes is also uncertain. Observations of rapid growth and dispersal, and reductions in planktivore populations of river systems indicate that Asian carps may also similarly impact Great Lakes food webs, and subsequently, commercial or recreational fisheries.[14,40] In contrast, assimilations with little or no measurable impact to the Great Lakes ecosystem and resource limitation have been suggested as reasons why Asian carps may not cause ecological harm.[41] The uncertainties remaining about how damaging Asian carps would be may be narrowed by additional research using empirical studies and foodweb models.

Costs associated with the implementation of some of these alternative strategies are known or would be fairly easy to quantify, as suggested by these examples: one electric barrier located in the CAWS cost $13 million (USD) to install and $0.9 million per year to maintain;[42] the Minnesota Department of Natural Resources requested $12–19 million to install an ABS barrier at Lock and Dam 1 on the Mississippi River with an annual operating cost of $0.25 million;[43] maintaining a hypoxic barrier in the CAWS would cost approximately $0.25 million per day, or over $91 million per year;[15] and hydrologic separation is estimated to cost $3.5–9.5 billion with low annual maintenance cost.[16] Thus hydrological separation, the strategy identified with high certainty as the most effective strategy, is also likely to be the most expensive strategy, at least in initial expenditures. However, at least two other considerations suggest that the initial expense of hydrological separation would be at least partially offset by other advantages relative to the other strategies.

First, initial expenditures of hydrological separation would be effective over a much longer time frame relative to all the other methods, which would have shorter lifespans and/or higher periodic maintenance costs. The time frame over which decision-makers assess costs and benefits will be a major determinant of cost-benefit calculations.[1]

Second, we asked experts to estimate the effectiveness of the 17 fish deterrent strategies against Asian carps only, but many other animals and plants are poised to disperse through the CAWS in one or the other direction.[44] In contrast to hydrologic separation, the other 16 strategies are designed primarily to deter the movement of juvenile or adult fishes (e.g., ABS, electric barriers, pheromones, block nets), and are not suitable to prevent the passage of mollusks, crustaceans, macrophytes, or gametes or larvae of fishes and other taxa suspended in the water column. The implementation of prevention strategies that inhibit the movement of multiple species within a pathway, rather than strategies that focus on single species, are likely to be more effective and have greater societal benefits.[45] Thus the costs of each of the alternative strategies reviewed above would more reasonably be weighed against the avoided damages from many species—not just Asian carps. With projected increases in species introductions, global transportation and canal construction, the management of these pathways (including the coupling of deterrent strategies with surveillance programs and response plans to reduce the likelihood of invasion) is necessary to reduce the unwanted effects of species invasions.

Although the results presented here are specific to Asian carp passage from the Mississippi River Basin to the Great Lakes, this approach can serve as the framework for addressing the scientific unknowns associated with any species invasion, or uncertainties associated with new environmental risks. For example, SEJ is valuable method for identifying key uncertainties that may point the way to future empirical research programs. Rather than allocating resources to understand whether less effective methods are marginally more effective than one another (i.e., hydrogun vs bubble curtain), natural resource managers can instead focus efforts on maximizing the projected effectiveness of hydrologic separation or electric barriers. By separating expert responses and the identity of experts, the Classical Model of SEJ provides a means for experts to quantify scientific uncertainties on important societal issues without the risk of political confrontation, and for policy makers to utilize this information to make risk-based decisions in a transparent and defensible way.