Independent Scientific Review under the Endangered Species Act.

The directive from Congress in the Endangered Species Act obliging the US Fish and Wildlife Service and National Marine Fisheries Service along with other federal agencies to use the best available scientific information in their determinations-and calls from stakeholder communities to show that they have done so-have led the federal wildlife agencies to seek external, expert review of their determinations with increasing frequency over time. In the present article, we survey the agency determinations that may be subject to independent science review and the technical tasks embedded in those determinations that can benefit from such review. We go on to identify common failures in scientific review that compromise the quality and reliability of agency determinations and then describe the attributes of independent scientific reviews that enable the agencies to discharge their statutory duties while seeking to conserve threatened and endangered species and the ecosystems on which they depend.

In retrospect, Congress' prescience in directing the US Fish and Wildlife Service (USFWS) and National Marine Fisheries Service (NMFS) to use the best available scientific information to inform its regulatory determinations under the US Endangered Species Act (ESA) of 1973 was, in a word, remarkable. That durable standard—crafted to change over time to reflect scientific advances—has contributed to manifold conservation success stories. Scientific information fuels implementation of prohibitive and affirmative policies and actions carried out under the authorities of the ESA. Reliable characterization of the status and trends of species is required to make listing and delisting determinations, resource needs and landscape use by listed species must be quantified to develop recovery plans, and species–habitat relationships need to be analyzed to inform consultations between federal agencies when listed species may be affected by federal agency actions. Congress anticipated that science and the input of scientists would be essential in implementing nearly all of the Act's provisions. Less clear is whether the authors of the statute understood that formal review by outside experts of agency determinations under the Act would become common practice.

Considering the substantial economic and social impacts that can accompany implementation of ESA prohibitions, together with concerns by the environmental community that species protections are frequently inadequate, one can appreciate the demands of stakeholders that agency determinations be subjected to review by outside experts. Recognizing the value of scientific review, the federal wildlife agencies nearly a quarter century ago issued a “peer review policy” that targets listing recommendations and draft recovery plans (USFWS and NMFS 1994). The policy, updated by memo in 2016 (USFWS 2016), directs the agencies to “solicit the expert opinions of three appropriate and independent specialists regarding pertinent scientific or commercial data and assumptions relating to the taxonomy, population models, and biological and ecological information for species under consideration for listing” and use “the expertise of and actively solicit independent peer review to obtain all available scientific information… during the development of draft recovery plans for listed plant and animal species.” Over the past two decades the agencies increasingly have relied on scientific review to inform a wide range of decisions under the ESA (described in some detail below). Regulatory peer review—outside input into agency determinations that affect public policy—has become an important means of assuring that the wildlife agencies' determinations are based on reliable knowledge and therefore are lawful (Ruhl and Salzman 2006). Particularly in those cases when agency determinations have significant societal impacts, we contend that independent science reviews of such determinations should share a common set of attributes to assure they facilitate the use of the best available scientific information in agency decision-making.

Review of the methods of and inferences drawn from scientific investigations is a critical aspect of modern scientific inquiry (see Ziman, 1968, 1969). Science narrowly defined is the product of application of the scientific method. But the vernacular definition of the term “science” in its applications under the ESA encompasses both reliable knowledge and the actions of scientists, including provision of expert opinions (Murphy and Weiland 2016). Scientific review, that is, evidence-based judgment by scientists, is frequently referred to as “peer review,” the label typically applied to scholarly review by actual professional peers. The term peer review does not reflect particularly well the type of review intended by the wildlife agencies, which is to obtain expert assessments of agency documents that have been generated by skilled staff who have synthesized data and analyses from the primary scientific literature and other information sources.

Peer review under the wildlife agencies' purview addresses the review of draft decisions or manuscripts by subject-matter experts in relevant fields of inquiry who did not contribute to the development of those decisions or manuscripts. Recognizing “the importance of peer review for the credibility of agency scientific products,” an Office of Management and Budget (OMB) information-quality bulletin (OMB 2005) established government-wide “standards for when peer review is required for scientific information and the types of peer review that should be considered by agencies in different circumstances.” Providing detail that the wildlife agencies' peer review policy lacked, the bulletin states “Peer review typically evaluates the clarity of hypotheses, the validity of research design, the quality of data collection procedures, the robustness of the methods employed, the appropriateness of the methods for the hypotheses being tested, the extent to which the conclusions follow from the analysis and the strengths and limitations of the overall product.” The bulletin also addresses a number of critical elements of peer review, including the timing and scope of review, selection of reviewers, public participation, and disposition of reviewer comments. The list of tasks included in the bulletin does not appear to be so different from that before a faculty committee with respect to a doctoral student's dissertation or an editorial board vetting manuscripts submitted to a scientific journal. In those contexts, scientific review can be conceived of as a complement to reproducibility by offering an independent view of the technical matter and a second opinion, both of which increase confidence that the knowledge being conveyed is reliable.

The value of scientific review in the context of government policy decisions is real, but is tempered by its institutional design. It rarely is an effort to reproduce the analysis or information synthesis that is the subject of review. As a consequence, in virtually every case, there are elements of the agency determination that are not scrutinized in the course of the review. Moreover, a number of reports and studies have confirmed that the less-than-comprehensive nature of the scholarly peer review process can lead to reviews that may miss major shortcomings in the underlying decision or manuscript (e.g., Ferguson et al. 2014, Harris 2017). Accordingly, the existence of a scientific review can erroneously imply that the agency decision was based on reliable knowledge or the manuscript provides information that meets a best science standard, irrespective of the comprehensiveness and quality of the review (see Smith 2006, Bohannon 2013), or whether the agency responded to the review (Dudley and Gray 2012).

Although scientific review in the context of decision-making under the ESA shares certain attributes with academic peer review, the two differ in material respects. Notable among them are the facts that agencies whose determinations require review often seek, and in many cases provide, the funding for those reviews, select the outside experts who do the reviewing, and selectively respond to comments in the reviews, creating an entirely different relationship (or power dynamic) between the reviewers and the wildlife agencies than exists between a student and the student's dissertation committee, or an author and the journal editorial board that might consider publishing a scientist's work. Scientific reviews commissioned by the wildlife agencies frequently are neither as rigorous nor as independent as OMB’s guidelines intended (Greenwald et al. 2012).

Recognizing that reviews of agency determinations frequently are not well characterized as peer reviews and that those reviews are sought from experts from outside the wildlife agencies, we use the term “independent scientific review” (following Meffe et al. 1998) to refer to reviews undertaken in support of implementation of the federal ESA. In the present article, we describe and evaluate independent scientific review as it pertains to (a) agency determinations and other products of wildlife-agency deliberations under the ESA that benefit from review by outside technical experts, (b) the “scientific” activities undertaken and analytical approaches, modeling techniques, and application of assessment methods that require such review, and (c) the recurring shortcomings of past scientific reviews. We then present the characteristics of credible independent scientific reviews and offer recommendations that, if they are implemented in full, should address the concerns of critics of wildlife agency-generated reviews and increase appreciably the ability of those agencies to fulfill their statutory obligation to use the best available scientific information available in decision-making (see Murphy and Weiland 2016).

Agency determinations under the Endangered Species Act that benefit from independent scientific review

The central regulatory features of the ESA are (a) the processes whereby species are listed as threatened or endangered and critical habitat is designated for them, (b) the protections afforded to such listed species, principal among them the prohibition on “take” of listed species, and (c) the processes available to obtain authority to conduct activities that are expected to harm (or “take”) listed species. The ESA also includes provisions for the development of recovery plans for listed species and for delisting species where the protections afforded by the ESA are no longer needed. Agency determinations made under the authorizing provisions of the ESA benefit from independent scientific review, which may involve the institutional capabilities and protocols of the federal wildlife agencies (box 1).

Box 1. Institutional processes that the wildlife agencies follow in implementing independent scientific review (NMFS 2017, USFWS 2017). The processes are consistent with guidelines developed by the OMB (2005).

Fish and Wildlife Service

The USFWS has implemented a formal “peer review” process for influential scientific documents. That process recognizes the need for “contractor support to conduct a variety of scientific activities” including review of “documents, data, models, study plans, proposals, and other forms of scientific information.” The agency follows the guidelines for federal agencies spelled out in the Office of Management and Budget's Final Information Quality Bulletin for Peer Review. Part of the peer review process is to provide information online about how each peer review is to be conducted. The agency makes convenient use of three support contractors to identify expert reviewers and conduct peer reviews of documents proposed rulemakings, such as listings, status reviews, management plans, recovery plans, biological opinions, and scientific papers, including models, analyses, and data. The agency has a peer review checklist that includes a description of the document subject to review, the peer review process, public participation, and a point of contact.

National Marine Fisheries Service

The NMFS established the Center of Independent Experts (CIE) to routinely provide external and independent expert reviews of the agency's influential science used for policy decisions. The impetus for the establishment of the CIE was to strengthen NMFS’s science quality assurance in accordance with the mandate of the Magnuson-Stevens Fishery Conservation and Management Act, although the agency uses the CIE for reviews conducted under the ESA. Peer review requests that include expertise requirements and terms of reference for each peer review are submitted to the NMFS’s Office of Science and Technology. The CIE conducts an independent selection process to provide highly qualified experts who adhere to rigorous peer review standards, such as independence from the science under review and strict conflict of interest standards. The reviewers may be required to participate during panel review meetings or conduct desk reviews in which travel is not required. The contract deliverable is a CIE peer review report from each CIE reviewer that is independently reviewed by a CIE technical team. Although most CIE reviews are related to fisheries stock assessments, past CIE engagements include reviews of listing decisions, recovery plans, biological opinions, and other determinations and documents that are required to use best available science.

The provisions of the ESA that yield significant agency determinations, other than those respecting enforcement of the Act, are included in sections 4, 7, and 10 of the statute. Section 4 includes provisions for the listing of species as threatened or endangered. Listing decisions, which are made on the basis of biological considerations and absent economic considerations, are important because they trigger other key regulatory provisions of the Act, namely the interagency consultation requirement in section 7 and the prohibitions delineated in section 9. Acknowledging concerns that imperiled species tended to be conferred protection only when their numbers had dropped to extremely low numbers, a National Research Council committee proposed “quantitative risk standards,” which it defined as “the probability of extinction within a specific timeframe” to serve as the basis for listing and assigning the status conditions “threatened” and “endangered” to species that warrant protection under the Act (NRC 1995). That proposal encouraged the wildlife agencies to make listing decisions on the basis of “a set of objective criteria for assigning species to risk categories” and assigning the degree of threat to species on the basis of “probability of extinction, trends in abundance, population size, number of populations, and geographic extent”—scientifically justified criteria. Section 4 also includes provisions for conducting periodic status reviews that may result in reclassifying a species from endangered to threatened, or vice versa, or delisting a species that is no longer in need of the Act's protections. These actions can increase, reduce, or eliminate the extent to which key regulatory provisions of the ESA operate to protect the species. Furthermore, the Act (16 U.S.C. 1533[f]) expressly authorizes USFWS and NMFS to engage qualified persons from outside the wildlife agencies in the development and implementation of recovery plans. Independent science review has been employed in the service of each of these agency actions. For example, recent independent scientific reviews addressed the proposed listing of the wolverine (Gulo gulo; Woods et al. 2014), the proposed delisting of the valley elderberry longhorn beetle (Desmocerus californicus dimorphus; Atkins 2013), the status of the pallid sturgeon (Scaphirhynchus albus; USFWS 2005), and a recovery plan for the Mexican wolf (Canis lupus baileyi; see Beier et al. 2017).

Section 4 also addresses the designation of and revisions to critical habitat, a legal construct that is partially informed by the ecological concept of habitat, therefore can benefit from independent scientific review. The Services routinely designate for listed species critical habitat areas that do not meet the ecological definition of habitat—that is, the landscape areas that support a species and the resources and conditions of those resources necessary for its survival and reproduction. For example, the designation of critical habitat for the dusky gopher frog, which is presently the subject of litigation, includes lands that are not currently habitable by the species (Weyerhauser Co. v. U.S. Fish and Wildlife Service, 139 S. Ct. 361 (2018)). The Services may exclude from critical habitat areas that do meet the ecological definition of habitat to the extent they are not “essential for the conservation of the species” or do not require “special management considerations or protection” (16 U.S.C. § 1532(5)). Whether an area is designated as critical habitat is important because it is considered expressly during the interagency consultation process and because some federal and nonfederal agencies take critical habitat into account when implementing their authorizing legislation, such as the National Forest Management Act (by the US Department of Agriculture Forest Service) or the Coastal Act (by the California Coastal Commission). Examples of scientific reviews of critical habitat designations where the agency determinations were controversial include those for Preble's Meadow jumping mouse (Zapus hudsonius preblei; USFWS 2009) and Canadian lynx (Lynx canadensis; USFWS 2013).

Section 7 requires federal agencies to consult with the Services when actions they authorize, fund, or carry out may affect listed species or designated critical habitat. The provisions can lead to formal consultation, which obliges the wildlife agencies to prepare a biological opinion and, where take of the listed species is anticipated to occur, an incidental take statement (authorizing “take” of listed species incidental to otherwise lawful activity). For unusual circumstances where the Services make a determination that a proposed action is likely to “jeopardize the continued existence” of the species or result in the destruction or adverse modification of critical habitat, the Services are required to suggest “reasonable and prudent alternatives” to avoid jeopardy to the species or adverse modification of its critical habitat, and can be taken by the federal agency or applicant in implementing the agency action. Biological opinions, incidental take statements, and reasonable and prudent alternatives constitute agency determinations that can have material consequences for a wide range of human activities, as well as for listed species. Reviews of agency determinations under section 7 of the Act have taken various forms ranging from targeted reviews (with respect to a number of species of fish) conducted by National Research Council committees (National Research Council, 2002, 2010) to a review of a biological opinion for the Stellar sea lion (Eumetopias jubatus) administered by the National Marine Fisheries Service's Center for Independent Experts (Stewart 2012).

Section 10 provides that nonfederal entities may prepare and seek Service approval of a habitat conservation plan (HCP) to authorize take of listed species incidental to otherwise lawful activities, where there is no federal involvement in the activities. These plans vary widely in scope and complexity. In some cases, they allow take of a single species on less than a hectare. In other cases, they allow take of dozens of species across hundreds of thousands of hectares spanning multiple states. So-called low-effect HCPs usually do not permit levels of take that warrant independent scientific review, but many spatially extensive HCPs allow take of multiple listed species and their habitats thus warrant such review. Scientific review may occur during both the planning and implementation phases of HCPs. Examples of HCPs that were reviewed during the planning phase include the Santa Clara Valley Habitat Conservation Plan, a landscape plan that encompasses over 500,000 acres in central California and covers 18 species (reviewed by Independent Science Advisors 2006) and the Sacramento–San Joaquin Bay–Delta Conservation Plan, a largely aquatic plan that was intended to cover 54 species including fish, mammals, birds, reptiles, amphibians, invertebrates, and plants, which ultimately was abandoned for reasons independent of the review (see Independent Science Advisors 2007).

Scientific and technical tasks that benefit from independent advice and review

Decision-makers, agency staff, and stakeholders frequently refer to the technical information that is used to inform agency determinations under the ESA as “science.” But, of course, science is a process, the means of obtaining reliable knowledge through application of the scientific method (Hilborn and Mangel 1997, Murphy and Weiland 2016). The technical information that shapes wildlife-agency determinations takes myriad forms. It can include informal and formal narratives regarding the life history, distribution, and trends in abundance of a species; anecdotal accounts and raw data from surveys or monitoring, ranging from presence/absence surveys on individual parcels or in discrete river reaches to population- or species-wide census efforts; the output of analyses from ongoing or previous research in the gray and peer-reviewed literature; conceptual and numerical models of species–environment relationships; and data, analyses, and inferences drawn from putative proxies for the species or its habitat. The agencies generally are responsible for determining which technical information constitutes the best available science (box 2).

Box 2. Types of information that should be based on data and analyses that can be considered to meet a best available science standard and can accordingly be a focus of independent scientific review.

It is understood that most such data are lacking for most listed species. Nevertheless, depending on the identity and substance the determination, any one or more of these information sources may play a deciding role in the agency's findings and prioritization of subsequent conservation actions.

Information on the listed species

Information on the taxonomic identity of the species and relatedness to other species

Data on the genetics of the species, including effective population size, spatial variation, and other measures of genetic diversity

Data on the distributional range of the species and its local abundances (or relative abundances)

Data on the environmental correlates of the species' historical and current landscape occupancy and abundance or densities of life stages

Data on survival and reproduction of the species across its range

Data from which population and metapopulation dynamics can be inferred

Data on life history that can inform population viability analysis or other demographic modeling efforts

Data on salient behavioral attributes of the species, including interspecies interactions

Information on the species' habitats

Data and observations that document species–habitat relationships, emphasizing ecological and behavioral data on habitat and resource use

Data pertaining to the environmental correlates of landscape occupancy

Data on the geographic extent of habitat and variation in habitat quality

Contemporary and historical data that reflect spatial and temporal patterns in resource use by the species

Inferences pertaining to limiting environmental factors

Data on prey, predators, competitors, and symbionts that affect the species

Information on stressors to the species and its habitat

Information on natural and anthropogenic environmental stressors and the mechanisms by which they affect the species and its habitats

Data on the spatial and temporal contexts and scales on which environment stressors operate to affect the species and its habitats

Information on nonnative, invasive species that may deleteriously affect the species by causing changes in the structure or ­composition of its habitats or in disturbance processes

The types of information on imperiled species, their habitats, and the threats to both that allow inference to the status and trends of those species are diverse, frequently are difficult to obtain, and often are equivocal in application. Analytical methods that can inform agency determinations have advanced rapidly since the ESA was enacted. Best standards and practices have emerged over the past decade (box 3), but the data necessary to use those analytical methods often do not exist. For example, the Fish and Wildlife Service lacks data regarding the status and trend of the Hawaiian hoary bat, despite the fact that the species has been listed as endangered for nearly 50 years (USFWS 2011). Likewise, there is a paucity of data regarding the status and trend of the spider relative, the Bone Cave harvestman, which is endemic to Texas and also listed as endangered (USFWS 2015: 16–17). Even for species that have been the subject of intensive surveys and sampling over time, such as the threatened delta smelt, data on key environmental stressors, such as predation rates, are wanting (see Hamilton and Murphy 2018). In many cases reviewers are often confronted with unpublished data of unknown quality and must evaluate their accuracy.

Conceptual ecological models that relate population dynamics to environmental drivers through ecological links and pathways

Conceptual management models that link prospective management actions to responses of species and their habitats

Life-cycle models that reflect life-stage-specific uses of habitats

Demographic models and population viability analyses

Ecosystem models that incorporate ecological and hydrological regimes, sediment dynamics, fluvial geomorphology, fire, and other physical processes

Spatially explicit landscape-scale models that consider the extent, configuration, and quality of habitat

Hypothesis testing in effects analysis, and other analyses that support agency selection from among management alternatives

Evaluations of putative surrogate, indicator, and proxy measures

Designs of surveys of species, their resources, and environmental stressors

Common failures of independent reviews under the endangered species Act

For several reasons, scientific review can do more harm than good if it is conducted improperly. It can yield advice that is contrary to species conservation needs or may function to impose societal costs without concomitant ecological benefits. Improper review may create a false sense of confidence or doubt in the program, plan, or project subject to review. There are a number of ways that poor institutional design can compromise scientific review. Table 1 describes recurring attributes in the design and implementation of reviews that can produce flawed, unreliable outcomes.

Table 1.

Attributes of the science review process that can produce flawed, unreliable outcomes.

Design attribute	Design flaw	Example
Selection of reviewers	Allowing the entity subject to review to select the reviewers	The same staff who drafted a proposed rule to designate critical habitat for a listed species selected the reviewers of the proposed rule (USGAO 2003, USFWS 2009, USFWS 2013)
Qualifications of reviewers	Mismatch between the tasking and reviewer qualifications; employing improper conflict of interest guidelines	An individual appointed to a panel that reviewed proposed listing rule did not have the expertise necessary to respond to the four questions in the task statement (NCEAS 2014) An individual appointed to a panel that reviewed a biological opinion was dismissed from the panel solely because an article she wrote and disclosed prior to selection was published in a peer-reviewed journal while she was serving on the panel (McGuire 2010, Taugher 2010). The four reviewers of a delisting proposal included one whose consulting work and one whose research funding depended, in part, on the species remaining listed (Atkins 2013)
Specification of the task	Allowing the entity subject to review to specify the task	The agency acting alone drafted the review questions for a panel reviewing genetic data on a listed species (AMEC Foster Wheeler Environment and Infrastructure, Inc. 2015)
Development of the record for review	Allowing the entity subject to the review to determine the record for review	The agency provided narrow records for review when seeking outside technical input regarding proposed listings of species (NMFS 2012)
Sufficient time	Providing reviewers insufficient time to complete the task	The agency provided a review panel less than a week to review an extensive draft biological opinion for a complex project (PBS&J 2008)
Communication with the reviewers	Direct communication occurred between the resource agency and reviewers absent oversight	The agency communicated directly with reviewers when seeking input regarding proposed listings of species (NMFS 2012)
Deliberation among the reviewers	Allowing the entity subject to review to participate in or observe deliberations	The agency listened in on a review panel as it engaged in deliberations regarding the adequacy of the scientific basis of a proposed listing rule (NCEAS 2014)
Timing of the review	Conducting a review too early or late in the planning process	The State of California provided an incomplete draft habitat conservation plan to a panel; the panel noted the plan was “incomplete in a number of important areas,” and identified a number of “critical gaps” in the plan finding that the “plan is missing the type of structure usually associated with current planning methods in which the goals and objectives are specified, alternative measures for achieving the objectives are introduced and analyzed, and a course of action is identified on the basis of analytical optimization of economic, social, and environmental factors” (NRC 2011)
Response to the review	Not requiring the entity subject to review to provide a written response to the review	The action agency and wildlife agencies failed to provide substantive responses to panel reviews regarding implementation of large-scale biological opinions (Bureau of Reclamation et al. 2015)

Reviews with shortcomings in institutional design are exemplified by a review of the status of gray wolf (Canis lupus) in the wild. A group convened by the National Center for Ecological Analysis and Synthesis performed the review at the request of USFWS. The purpose of the review, broadly stated, was to assess the USFWS’s proposed rule “to remove the gray wolf from the List of Threatened and Endangered Wildlife but to maintain endangered status for the Mexican wolf by listing it as a subspecies” (NCEAS 2014). The report states, “Our primary goal in selecting reviewers was the individual scientist's ability to perform the task.” But with respect to the scientific questions that the panel was charged with answering, one of the five reviewers explained, “I cannot comment on the scientific substance underlying the 4 questions, because this is not my field of scientific study.” This should have disqualified that potential reviewer at the outset of the review (NCEAS 2014). The statement is remarkable given that the panelists were selected on the basis of their ability to perform the review, and the fact that the panelists were asked to respond as individuals rather than as a committee.

The scientific review of the listing determination for the bearded seal (Erignathus barbatus) also illustrates a flawed institutional design. NMFS organized the review in response to concerns that climate change could threaten the species' habitat; the species was proposed for listing despite the fact that its numbers appeared to be stable. NMFS selected the reviewers, described the task, and determined the record for review (National Marine Fisheries Service 2012), none of which is an appropriate action by the agency that made the determination under review. The report on the so-called special independent peer review, which was included in the administrative record for the decision by NMFS, was written by NMFS. Furthermore, the actual review consisted of two individual reviewers' responses to questions posed by the Service. Because of the pervasive involvement of NMFS in the review, it cannot be characterized as either independent or objective.

A third example of a review characterized by a shortcoming in institutional design involved the biological opinions for continued operations of California's Central Valley Project and State Water Project in California. Annual reviews are performed by a standing independent review panel and coordinated by the Delta Science Program (within California's Natural Resources Agency). The first review occurred in late 2010, following the completion of the USFWS’s biological opinion for the delta smelt (Hypomesus transpacificus) in December 2008 and the NMFS’s biological opinion for salmonids and the green sturgeon (Acipenser medirostris) in June 2009. In the first review, the panel noted that the Services assessed the success of conservation actions (referred to as reasonable and prudent alternatives) by measuring abiotic targets rather than the responses of the listed species, and urged the Services to refocus on the latter (Anderson et al. 2010). The Bureau of Reclamation and the Services acknowledged and seemed to accept the recommendation, but failed to make any management changes in response to it (National Marine Fisheries Service et al. 2011). As a consequence, in successive reviews, the panel repeated its concerns. In its 2013 report, the panel pointed out that all of its previous reviews emphasized the need to link ­success or failure in meeting physical targets prescribed in the reasonable and prudent alternatives to the responses of the listed species (Anderson et al. 2013). In its 2014 report, the panel observed that the effectiveness of the reasonable and prudent alternatives “in terms of biological responses has remained elusive under all conditions” (Anderson et al. 2014). The failure of the agencies to respond adequately to the reviews is striking and problematic.

Attributes of successful science advice and review

Recognizing the diversity of scientific reviews, Meffe and colleagues (1998) opined that there is no single, set format that independent scientific reviews should follow. Although we share that view, we offer nine common attributes of successful scientific review and rules of engagement that should be followed to maximize the likelihood that policy in a determination under the Act will produce the desired outcome.

Panel to function as a deliberative body with at least three panelists

Expert reviewers should serve as a panel and deliberate as a panel. There may be cost savings associated with soliciting discrete reviews from individuals who do not meet or otherwise interact through the Center for Independent Experts, as the National Marine Fisheries Service has often done (see box 1). But that approach precludes deliberation, which has substantial benefits, among them facilitating exchange among panel members with diverse areas of expertise and providing a forum for resolving disagreements and refining conclusions. We think that a panel should have no fewer than three members, but usually would benefit from more. The size of the panel should match technical (scoping) needs, which include a requisite complementary breadth of skills, expertise, and experience. A panel typically should include members with familiarity in experimental design, quantitative ecology, ecological theory, and species–habitat relationships. The review should be a report from the panel as a whole, although it may be appropriate for individual reviewers or groups of reviewers to provide divergent views, including on issues of disagreement where the panel is unable to reach consensus.

Panel balance in perspective

In some circumstances concern has been expressed regarding perceived bias of prospective panelists who otherwise may offer requisite diverse expertise to a review. As the National Academies has explained, “Potential sources of bias are not necessarily disqualifying for purposes of committee service” (National Academies 2003). The appointment of members with divergent backgrounds or perspectives may be necessary to create a panel with essential skills and balance. We recommend that the selection of panelists focus on actual conflict of interest rather than bias. In the context of independent scientific review, conflict of interest is a financial, professional, or personal interest that reasonably could be expected to significantly impair the objectivity of a potential panelist or create an unfair competitive advantage for the potential panelist or their home institution.

Use of a third-party neutral to administer review

When practicable a third-party neutral or equivalent intermediary between the resource agency and the scientific review panel can contribute to establishing panel independence, impose reasonable sideboards on direct communication between the resource agency and the reviewers, and thereby enhance stakeholder confidence in the outcome of the review process. Staff typically serves this role on National Research Council committees and panels. The role can be played by a regional scientific body (e.g., the Delta Science Program in the Sacramento–San Joaquin Delta, California), or by an independent entity or individual (e.g., the third-party neutral brought into the Missouri River Recovery Program process described below). The intermediary should be involved in development of the charge and decisions about the scope of review materials.

The task description

The charge or task statement provided to the panel should be crafted to query the foundational approaches taken, the pertinence and quality of data and analyses employed, and the conceptual and quantitative modeling approaches used to support the determination. The task description should steer reviewers away from making policy recommendations, and should restrict them to critically evaluating and interpreting the information—including observations on data and model limitations and uncertainties—used to support the determination. It is appropriate for panelists when reviewing a document or determination that presents alternative management-action scenarios, for example, to rank them in terms of relative extinction risk to a target listed species.

Scope of materials to be provided to reviewers

The review panel should be provided the administrative record and materials that explain fully the policy and management context and the need for the document requiring review. Technical presentations to the review panel (which can include agency experts) can be invaluable, allowing reviewers direct access to subject-area experts who may be the sources of salient applied information. Background material that may be necessary to understand the circumstances that require the agency determination should include supporting materials that are cited in the document to be reviewed and other technical materials that have contributed to development of the product subject to review. In addition, scientific information that is omitted from the list of references, but bearing on the determination, should be provided to the panel (Pullin and Stewart 2006). Stakeholders affected by the determination should contribute to background materials presented to the review panel.

The agencies need to show their work

The reviewers should be provided the supporting materials that are necessary to allow them to discern the stepwise process that the agency followed in order to generate its determinations, its interim findings, or its conclusion(s), and how relevant technical information was synthesized, interpreted, then integrated into that process. In some cases, a decision document will incorporate the analyses, findings, and conclusions; in other cases, the review panel will need to be provided multiple information sources in addition to the decision document.

Adequacy of time and resources for the review

The review panel should be afforded sufficient time and resources to complete a comprehensive assessment of the determination and supporting documentation. Scientific reviews can seldom be afforded months to complete, but a thorough review of a determination and supporting technical material, and delivery of an integrated panel response, procedurally will take at least a couple of weeks to a month. A biological assessment or biological opinion and supporting technical material, for example, may run into hundreds of pages. They may include a breadth of literature citations, numerical models, complex analyses, and logic chains linking ecological phenomena with the risk assessment on which a management action scenario and an accompanying monitoring scheme is based.

Timing of review relative to agency action

The scheduling of independent scientific review must be appropriate to the review tasking. In some cases, reviewers are engaged to provide feedback on early phases of a plan or project, for example, reviewing goals and objectives, or conceptual ecological models. Where this is the case, reviewers are properly engaged well before decisions and determinations are made, when changes to the trajectory of a plan or project will not have substantive adverse programmatic implications for the agency. In other cases, reviewers are asked to assess agency determinations at or near competition. By and large an agency determination can only be evaluated when a complete draft is available. When an incomplete agency determination is provided to reviewers, benefits from the review cannot extend to missing sections. So, for example, past reviewers have noted that where a species status review does not include a listing recommendation (Fleishman 2017), or a conservation plan does not include an effects analysis (NRC 2011), those absences amount to critical gaps in the documents. A review early in the process can have value to the ultimate agency deliverable, but it should be followed with review at or near completion of the determination or document. A review carried out in later stages of the development of a determination or document should include all relevant material and allow time for the agency to respond to the review, including by undertaking new modeling or other analyses if necessary.

Agency response to the review

The review panel's charge should describe how the agency intends to respond to the review, and the agency should be held accountable for responding accordingly. The agency should respond in writing to individual review responses and document whether and how the review input, both affirmative and critical, influenced or did not influence the final determination. All recommendations, prescriptions, and other substantive input from the review panel require acknowledgement, including justification by the agency for nonresponses to or rejections of critical reviewer comments.

The nine attributes of successful independent scientific review should be viewed necessarily as a unit. Reviews consistent with all nine attributes can be expected to contribute to agency efforts to use the best available scientific information when making decisions. Imposing the requirement for reviews to include the nine attributes will increase the likelihood that decisions subject to review are based on reliable knowledge, applied using generally accepted analytical tools and contemporary modeling approaches, and are lawful. The requirement also can be expected to increase the legitimacy of such decisions. And, most importantly, it can serve the policy ends of achieving the optimal amount of regulation at the least cost to society.

At the same time, not every wildlife agency determination must undergo a high level and intensity of review. The wildlife agencies engage in thousands of informal consultations annually that have neither the societal costs nor the potential to affect listed species that would justify fully engaging in the nine process attributes described above. But there are many dozens of activities annually, ranging from listings or delistings of immediately imperiled species to consultations regarding large-scale land planning or water management efforts to long-term, landscape-level habitat conservation plans, for which an elevated level of scientific review should be required.

At least one ongoing conservation-planning process, the Missouri River Recovery Program, appears to incorporate all nine of the attributes of successful independent scientific review. This program was initiated by the US Army Corps of Engineers in 2006 to comply with its obligations under a biological opinion intended to avoid jeopardy to the pallid sturgeon (Scaphirhynchus albus), Piping Plover (Charadrius melodus), and Least Tern (Sternula antillarum) while maintaining congressionally authorized uses of the river, including flood control, navigation, irrigation, hydropower, water supply, water quality, fish and wildlife, and recreation (USFWS 2003). In fulfillment of its obligation, the agency used a structured decision-making process, developed conceptual ecological models, management hypotheses, effects analyses, and a science and adaptive management plan, and prepared an environmental impact statement, a biological assessment, all under the purview of a standing scientific review panel (box 4). The subsequent biological opinion, rendered by the USFWS, was also reviewed by the panel for its technical adequacy.

Box 4. Independent scientific review under the Missouri River Recovery Program.

The Missouri River Recovery Implementation Committee (MRRIC), a stakeholder advisory group engaged under the authority of the US Army Corps of Engineers (Corps), convened an independent scientific advisory panel (ISAP) in 2011 to assess the efficacy of managed spring–pulse releases from Gavins Point Dam in mitigating negative effects of river operations on pallid sturgeon, Piping Plover, and Least Tern, which were prescribed in Biological Opinions issued by FWS to enhance and sustain habitat quality and availability for the three listed species. The six-person panel reviewed and interpreted available information, concluded that the spring–pulse management actions had been inadequate in volume and duration to benefit the species and their habitats, and that the then-existing monitoring program could not measure the effectiveness of the actions. The panel recommended that the Missouri River Recovery Program (MRRP) carry out a structured analysis to assess the effects and associated costs of alternative management actions on the listed species within a risk-analysis framework that would also explicitly consider socioeconomic factors.

The Corps and FWS agreed to implement a new species management plan “using the structured effects analysis as proposed by Murphy and Weiland (2011), employing an adaptive management framework, and abiding by the requirements of the National Environmental Policy Act” (Fischenich et al. 2016). MRRIC and the agencies retained the ISAP to provide independent scientific review of the technical elements of the effects analysis and development of an adaptive management plan. Over 5 years the ISAP provided reviews of draft, interim, and final programmatic technical products that included the following:

A description of a structured decision-making approach that would serve as the institutional framework for developing a MRRP process, including adaptive resource management, that would meet the Act's directive that management actions be informed by the best science available

An evaluation of other species-conservation approaches applied elsewhere in the country and their applicability to the MRRP

Syntheses of existing scientific data, analyses, and models pertinent to effects analyses and risk assessment

Species-specific conceptual ecological models, which set forth biological and physical relationships pertinent to addressing the management challenge

Management hypotheses that are evaluated to assess effects of candidate management actions and contribute to the parameters for the quantitative effects analysis

Analyses of the effects of the management-action alternatives on habitat availability and the demographic responses of the species

The science and adaptive management plan, including its identification of objectives, key uncertainties, conservation targets, decision criteria, and proposed governance

Proposed monitoring designs, which are intended to provide long-term trend data and assess the performance of habitat restoration and maintenance projects, and need to be consistent, complementary, and cost effective across the multiple-species adaptive management plan.

On the strength of the independent scientific review process, the program can be lauded as using the best science available to develop management alternatives, to evaluate the effects of those alternatives on environmental resources and human uses of the river, and to modify management as necessary. The program's science and adaptive management plan identifies specific roles for the panel on an ongoing basis, including engagement during annual science workshops, review of critical technical products, and as an interface with stakeholders on science issues.

Independent scientific review of the Missouri River Recovery Program has been continuous over the past 7 years, but in other circumstances the duration of scientific review may be very much less, just a month or months, in order to inform a discrete agency determination. Scientific review of the Missouri River Recovery Program followed the enactment of federal legislation (Water Resources Development Act of 2007, Pub. L. No. 110–114, § 5018) and has a sustained federal funding commitment. Independent scientific review typically will need to be implemented with fewer resources. Nonetheless, scientific review—particularly of agency determinations that have significant societal impacts—should not be tailored in response to budgetary constraints in a manner that abandons any of the nine science-review attributes.

Science review has increasingly become a commonplace component of the decision-making process undertaken by USFWS and NMFS as they discharge their duties under the ESA. Capable scientific review offers the promise of improving the quality of agency determinations. For critics of those agencies, review can serve as a check on abuses, whether by political appointees seeking to advance Administration priorities or line staff seemingly unaccountable to senior civil service or political appointees. For agency proponents, scientific review can supplement resources stretched thin by ambitious Congressional mandates coupled with lean Congressional appropriations. This promise can be better fulfilled if USFWS and NMFS design the reviews to produce useful and defensible outcomes through implementing the nine recommended attributes of competent and reliable independent science review.