Comprehensive environmental assessment: a meta-assessment approach.

With growing calls for changes in the field of risk assessment, improved systematic approaches for addressing environmental issues with greater transparency and stakeholder engagement are needed to ensure sustainable trade-offs. Here we describe the comprehensive environmental assessment (CEA) approach as a holistic way to manage complex information and to structure input from diverse stakeholder perspectives to support environmental decision-making for the near- and long-term. We further note how CEA builds upon and incorporates other available tools and approaches, describe its current application at the U.S. Environmental Protection Agency, and point out how it could be extended in evaluating a major issue such as the sustainability of biofuels.

Introduction

To paraphrase Bob Dylan, the times, they are a-changin’ for the field of risk assessment. Risk assessors have been hearing a crescendo of calls seeking more holistic and systems-based approaches as well as greater transparency and stakeholder involvement, all with an eye toward supporting the near-term needs of risk managers as well as achieving long-term sustainable trade-offs between risks and benefits.[1−3] Few would quarrel with these notions, but the question is: how do risk assessors do these things?

We highlight here recent efforts that illustrate an approach to addressing such objectives through Comprehensive Environmental Assessment (CEA). CEA provides both a systematic framework for organizing complex information and a structured process for reaching transparent judgments about the implications of such information. As a meta-assessment approach, CEA combines various features and outcomes of existing assessment methods, including life-cycle based approaches, decision support techniques, cost–benefit analysis, and other such tools, along with the basic risk assessment paradigm.

In this article we briefly describe how CEA has been applied at the U.S. Environmental Protection Agency (EPA) in identifying and prioritizing research directions for engineered nanomaterials, using selected case study examples. We also note how CEA could be applied in evaluating a major issue such as the sustainability of biofuels. More detailed descriptions are available in other publications.[4−8]

Past Practice

Various assessment approaches have historically been used to evaluate environmental issues, particularly the traditional risk assessment (RA) paradigm[9] and life cycle assessment (LCA).[10] As commonly applied in evaluating health risks of specific chemicals, the RA paradigm lacks the scope of LCA. Whereas an LCA may focus in detail on emissions associated with the “cradle-to-grave” stages of a product’s existence, RA typically limits the exposure analysis component of risk characterization to the use of a chemical or product and may disregard “upstream” (e.g., manufacturing, distribution) or “downstream” (e.g., disposal or recycling) stages of the life cycle.[11] On the other hand, LCA does not typically have the temporal or spatial resolution of RA, nor does it usually quantify risks in the manner of RA.[12]

Proposals and efforts to combine RA and LCA[12−18] appear to be a step in the direction of more holistic, systems-based assessment approaches. But how does one go about simultaneously satisfying the goals of being holistic, systematic, stakeholder-engaging, responsive to risk managers, supportive of sustainability, and so forth? And, in particular, how does one do these things in a transparent manner?

Comprehensive Environmental Assessment

One answer to the above questions is an approach known as CEA. To understand CEA, it is first important to realize that it consists of both a framework and a process, as will be explained in more detail below. Although these two aspects of CEA overlap, the framework basically provides a holistic, systematic way to organize information about complex environmental issues, whereas the process affords a transparent and structured means to engage diverse expert and stakeholder perspectives in judging the implications of such information. In addition, CEA is a meta-assessment approach that builds on existing assessment methods and, where possible, the results of available analyses.

The CEA Framework

Although there are many more layers of detail underlying the two-dimensional summary of the CEA framework depicted in Figure 1, the figure illustrates key features of the framework, beginning with the product life cycle. Next, the possibility of releases of the primary material of interest and associated materials (including waste byproduct) to environmental media—air, water, sediment, and soil—must be considered for each stage of the product life cycle. Such releases could transfer through and between environmental media, and depending on various conditions, could undergo transformations that result in secondary substances.

Figure 1

Although this “10 000-meter” two-dimensional view can only highlight key topics, the CEA framework provides a holistic, systematic way to organize information about complex environmental issues.[53]

Environmental conditions include variables that could affect the fate of these primary and secondary substances, namely physical (e.g., temperature, wind/water currents), chemical (e.g., pH, reactions with other chemicals in the environment), biological (e.g., natural organic matter, bacteria), and social (e.g., human behavior patterns) factors. Note that environmental media and conditions are relevant throughout the chain of events encompassed by the CEA framework, including the potential for exposure to primary and secondary substances.

Exposure and uptake can be cumulative (involving multiple associated substances) and/or aggregate (involving a single substance via multiple pathways), and could be relevant to human populations, other living organisms (biota), and nonliving resources, such as structures and landscape features that comprise the abiotic natural environment.

Ultimately, the type and magnitude of both direct and indirect effects on various receptors and other types of impacts are evaluated in relation to exposure-dose potential by all relevant pathways. Note that there are always multiple “impacts” that can be deemed desirable or undesirable depending on the specific context or an individual’s perspective. Where does one draw the line on the extent of impacts to be included in a CEA? This is where the process aspect of CEA comes into play.

The CEA Process

The CEA process is distinctly more than convening a workshop or small group meeting to talk about the “issues” surrounding a topic. As outlined in Figure 2, the CEA process is designed to yield transparent judgments about the implications of information contained in the CEA framework. Compiling information in the CEA framework is a fundamental first step, but searching and interpreting the scientific literature and other sources of information are only the beginning of the CEA process. Next, the information in the framework is reviewed using a collective judgment procedure with an eye toward (1) the scope of a CEA, (2) the completeness and accuracy of information in the framework, and (3) the implications of information and information gaps compiled in a CEA framework document. Note that the goal is not simply to identify every conceivable concern pertaining to an issue but to determine priorities among information gaps (for research planning) or “risk trade-offs” (for risk management).

Figure 2

The CEA process builds on information compiled in the CEA framework and uses collective judgment to prioritize issues in support of research planning and/or risk management.[53]

By using a collective judgment process with diverse experts and stakeholders (explained below) to examine the above three elements, CEA helps address challenges related to scope and transparency. First, the scope of an assessment is subject to review along with the completeness and accuracy of information compiled in the framework document. Clearly, “comprehensive” cannot extend to infinity, yet it is also important to avoid arbitrarily setting boundaries that could omit significant impacts or other considerations. This does not mean that the scope of an assessment must be changed as a result of such review, but the CEA process does enhance transparency by engaging a broad set of independent experts and stakeholders who can point out whether any potentially significant questions are being omitted from the scope. Risk assessors and risk managers may then choose to expand the scope, or not, as appropriate relative to time, financial, legal, and other constraints.

As noted above, the intent is not simply to compile extensive information but to weigh the relative importance of the various implications of this information and provide guidance to decision makers such as research planners and risk managers. For issues such as emerging technologies, which may have relatively limited information available, the implications may point to gaps and uncertainties in the extant knowledge base.[4−8] In these cases, the CEA process can provide guidance to those who manage research planning and implementation. For more mature issues that have a substantial scientific knowledge base, the focus of the CEA process may be to provide guidance to risk managers in deciding which risks and benefits warrant more attention.[19] Both pathways are presumed to lead to outcomes (e.g., research results, monitoring data) that would figure into an iterative process, as depicted in Figure 2 by arrows that return to the CEA framework. Such a process is also consistent with the concept of adaptive management.[1,20,21]

Collective judgment based on diverse perspectives is a key feature of the CEA process. Diversity can be defined in various ways, but for our purposes it has primarily implied heterogeneity in the areas of technical expertise and institutional affiliations of the persons participating in the judgment process. Not only is there reason to think that some types of judgments are improved by having broader participation than afforded by a small set of narrowly focused experts,[22−24] such a process also allows greater transparency (discussed below). As to how this is done, it is critically important to use a structured procedure that helps ensure all participants have an equal opportunity to contribute their views, as opposed to the more typical “free-for-all” discussions of many workshops, which can often be dominated by a small number of outspoken individuals, even when moderated by a designated leader or facilitator.

Various formal methods can be used for this purpose. Nominal group technique (NGT), for example, has been used to identify and prioritize information gaps for selected EPA nanomaterial case studies[5,7] and for a series of topics addressed by the National Water Research Institute (e.g., (25−27)). NGT is a relatively simple method to have a set of individuals consider and select their preferred choice(s) from a number of alternatives or options.[28,29] A variety of other methods are also available for use in conjunction with a CEA. The choice of method should be based in part on the state of the science related to an issue, where issues that have a well-developed knowledge base may lend themselves to more quantitative and analytic techniques such as multicriteria decision analysis (MCDA),[30] expert elicitation,[31] and variations on these and other approaches.[14,32,33] Where empirical data are lacking, a less analytic approach such as NGT or Delphi may be more appropriate.[34]

The choice of methods will be discussed separately in greater detail (manuscript in preparation), but it should be noted that some analytic methods (e.g., expert elicitation)[31] can be rather labor-intensive, which may effectively limit the number of participants who can be used in reaching a collective judgment. Also, regardless of the method employed, care should be taken to avoid making assumptions that can lead to premature structuring of the judgment process, which is especially a concern with newly emerging or data-limited issues.

CEA as Meta-Assessment

What distinguishes CEA from other approaches already available in the risk assessor’s tool box? As meta-assessment, CEA builds on other analytic and evaluative tools, including variations of LCA, health risk assessment, ecological risk assessment, cost–benefit analysis, decision science methods, and statistical meta-analysis. To the extent that some aspect of a topic has already been subject to evaluation using any of these approaches, CEA would include the results of prior assessments or analyses. For example, like LCA, CEA incorporates a product life-cycle perspective, and if a conventional LCA has been applied to an issue, the results of one or more LCAs can be incorporated into a CEA of that issue. Note, however, that CEA differs from most LCAs in including environmental fate, exposure-dose, and human health, ecological, and other impacts as equal, integral components of the basic CEA framework (as depicted in Figure 1). Moreover, whereas RA and LCA typically focus on quantitative data and analyses, CEA accords consideration to both quantitative and qualitative information. In characterizing the risks and benefits associated with an issue, for example, a CEA might include quantitative health reference values for a given chemical while using qualitative or, at best, ordinal estimations of the potential releases of the chemical. Even though a quantitative risk estimate might not be feasible, CEA could nonetheless provide useful guidance to risk managers in weighing which options to pursue.

CEA also shares the inherently comparative nature of cost–benefit analysis and of many LCAs, for the goal of CEA is to provide risk managers a clear sense of the most important trade-offs related to a technology option. A key point here is that CEA identifies priorities through a collective judgment process. Many assessments and research plans may point to a multitude of concerns or possible actions, but CEA provides a means of prioritizing for both research planning and risk management purposes (Figure 2).

Another sense in which CEA is a meta-assessment is that it not only includes results of previous assessments and analyses, but it also enables seemingly contradictory or inconsistent results to be weighed in reaching an overall judgment (see discussion of biofuels below).

CEA and Transparency

Calls for more transparency in assessments often accompany recommendations for greater stakeholder engagement. In our view, these are two separable albeit sometimes related objectives. Although we assert that the CEA process is facilitated by having the participation of a diverse cross-section of experts and stakeholders, such involvement does not in itself ensure transparency of the assessment.

By transparency we mean an explicit process for making judgments and reaching conclusions. Although a great deal of information is often included in LCAs and RAs, the basis for deciding which factors are given more weight or attention and how assumptions are made or justified is often not explained in any detail. “Professional judgment” frequently figures into pivotal aspects of such assessments. Peer reviewers may question the inputs and outcomes of an assessment, but the peer review process itself does not guarantee that the “black box” between input and outcomes will be opened. Nor is transparency achieved merely by asking for public comment or “input” at various stages of preparing an assessment.

A more useful way of thinking about transparency may be that it can provide a record that would enable an independent observer (e.g., a scientific historian) to understand at a later time how the conclusions of an assessment were reached and justified. CEA affords such a record not only in the framework document as a catalogue of information considered for the assessment, but in the collective judgment aspect of the CEA process. By using a formal procedure that provides an explicit record of key facts, values, and other information cited by the participants themselves and, even more importantly, an objective measure of the participants’ collective judgment of priorities, such as a rank-order list derived from a multivoting process, the CEA process makes the resulting assessment much more transparent.

Applications of CEA

As noted above, CEA can be used for both research planning and risk management purposes. We illustrate these two types of applications of CEA by briefly describing (1) case studies of nanomaterials used in developing a long-range strategic plan for nanomaterials research and (2) a proposed assessment of the sustainability of biofuels. Although these two purposes can overlap, at least in cases of assessments that reveal information gaps (which they typically do in the form of uncertainties), the nascent state of the science for nanomaterials[35−38] suggests that a more appropriate use of CEA for this area is to identify and prioritize research directions to support future assessment efforts.

Nanomaterials

CEA has been used in a series of EPA case studies of selected nanomaterials, to date including nanoscale titanium dioxide (nano-TiO2) in water treatment and topical sunscreen,[4] nanoscale silver (nano-Ag) in disinfectant spray,[6] and multiwalled carbon nanotubes in flame-retardant textile coatings.[8] The case studies focus on specific examples because the novel and often unique properties of different nanomaterials make it difficult if not impossible to consider them in the abstract or with generalities.

The organization of the EPA case study documents[4,6,8] follows the CEA framework depicted in Figure 1. For example, a chapter on the product life cycle includes information on feedstocks, various manufacturing processes, potential use patterns of the products, and likely disposal routes. A chapter on environmental fate discusses transport and transformation processes and the potential for nanomaterials to end up in wastewater treatment plant sludge or other environmental compartments. Other chapters discuss the potential for exposure, uptake, and effects in humans and other biota. Notably, other types of impacts (e.g., resource depletion, climate change) encompassed by the CEA framework are included in case studies when available information indicates that such effects might be significant issues.[8] Factors such as the size and surface charge of the nanomaterials and environmental conditions such as pH and organic matter composition are highlighted as well.

As shown in Figure 2, the case study documents provide a starting point for the CEA process. Although some details vary among case studies, the collective judgment step of the CEA process used groups of about 25 individuals who were selected to represent a range of technical fields (manufacturing, environmental fate, exposure, ecology, toxicology, risk management, etc.) and sectors (academic, civic, government, industry). The participants reviewed the draft case study documents prior to convening for a multiday workshop. In accordance with a formal NGT round-robin procedure,[5,7,28] each participant stated the issue(s) they considered the most important to address in order to conduct an assessment of the nanomaterial in question using the CEA approach. Participants thus heard a variety of viewpoints about key information gaps and could thereby potentially appreciate the overall state of the science from a broader perspective than their own particular expertise.

The participants then consolidated closely related or overlapping issues and voted on these topics using a 10-point weighted voting procedure, which yielded a prioritized list of research topics. Not surprisingly, the top priorities included many of the issues discussed in the draft case study documents,[4,6] including priorities pertaining to the need for better characterization of physicochemical properties, effects (ecological as well as human health), and exposure, although the specific needs were much more detailed than can be summarized here.[5,7] In addition, a few other issues not originally identified in the draft documents emerged through the workshop processes (e.g., mode of action information to guide nano-TiO2 toxicity testing, determining the half-life of nano-Ag in the environment, and developing standard reference materials for use by researchers). Thus the collective judgment process provided additional insights that would help shape a more complete research strategy to support future CEAs of nanomaterials.

Sustainability of Biofuels

Biofuels comprise multiple fuels (e.g., ethanol, biomass-based diesel [biodiesel], and various mixtures of ethanol and gasoline [e.g., E-10 to E-85]), as well as an extensive range and depth of technical issues that do not lend themselves to simple “good/bad” characterizations. Given the relatively mature state and complexity of the science surrounding some biofuels, the CEA approach could well be applied to identify and prioritize risk–risk and risk–benefit trade-offs of various biofuel options and thereby help inform policies and risk management decisions regarding their sustainability.

Focusing a CEA on the sustainability of biofuels (or any other technology or product) implies a concern about potential environmental impacts not just in the near term but over a long-range time frame and attention to not only environmental impacts but social and economic aspects as well.[1,39−41] To the extent that it is feasible for any assessment to weigh social and economic implications, CEA is also capable of doing so. But in determining the scope of a CEA (as discussed above) a judgment has to be made about the feasibility of encompassing social and economic considerations. For commercial products and technologies, proprietary interests may make it difficult to meaningfully assess economic implications, for example. However, we assume for the sake of illustration here that a CEA of the sustainability of biofuels could feasibly include economic and social dimensions along with environmental aspects. That said, in actual practice we would expect some sharpening of the scope of a CEA of biofuels by, for example, limiting consideration to a single fuel rather than all biofuels or perhaps by narrowing the focus to choices among alternative feedstocks. Another focus for CEA might be identifying and prioritizing metrics or ways of measuring sustainability from among the many indicators that have been used or proposed to date.[1]

Although a detailed description of the many issues and steps involved in a CEA of biofuels is beyond the scope of this article, we would note that compiling the extensive scientific literature on biofuels using the CEA framework has been facilitated by the first in a series of Biofuel Reports to Congress.[19] However, the CEA process would need to be applied to evaluate the scope and completeness of this report as a first draft of a CEA framework document on the sustainability of biofuels. Using a diverse group of individuals in a structured collective judgment process to evaluate this information not only would potentially expand the coverage of the document but would, as explained above, increase the transparency of the assessment.

Applying collective judgment in a CEA meta-assessment of prior assessments and analyses also provides a robust and effective means to overcome “paralysis by analysis” in the face of unquantifiable impacts, uncertainties, and contradictory information. For example, net energy gains or losses related to ethanol as a biofuel have been a prominent issue for several years.[42−44] Models and other analyses of this issue have produced seemingly inconsistent results and conclusions.[44−46] While debate on this point has persisted, other types of impacts have also emerged as noteworthy, including effects on resources (e.g., quality of water, air, soil), land use patterns, climate change, and several other major categories of impacts (e.g., health effects, biodiversity, invasive species).[42,47−49] A CEA meta-assessment of these issues would not only help resolve the confusion created by competing analyses, it would also help risk managers take decisive action by prioritizing where to focus mitigation and monitoring efforts if policy factors (e.g., energy security) override environmental sustainability considerations.

Further Actions

We have briefly highlighted some of the ways in which CEA responds to recent calls for assessment approaches that are more holistic, systems-based, stakeholder-engaging, transparent, and responsive to near- and long-term risk management needs, including sustainability. Although still being refined and extended, CEA has already been used in implementing recommendations related to risk assessment research in the EPA Nanotechnology White Paper.[35] Additionally, it has formed part of the EPA Office of Research and Development (ORD) Nanomaterials Research Strategy[36] and is aligned with recent efforts within EPA to have a more “integrated, transdisciplinary” research program in the ORD.[41] Beyond EPA, the CEA approach is consistent with and provides a means to address many of the goals and objectives identified in both the National Nanotechnology Initiative Strategic Plan[37] and the NRC Research Strategy for Environmental, Health, and Safety Aspects of Engineered Nanomaterials.[38] In addition, other organizations have taken steps to adapt CEA as an aid in evaluating or planning research on environmental issues.[50−52]

By integrating assessment tools, including decision science methods, in a meta-assessment approach, CEA also provides a holistic, transparent, and responsive means to assess complex and difficult topics such as the sustainability of biofuels, genetically modified organisms, climate change, and biodiversity. Thus, CEA can provide scientifically rigorous input to individuals and organizations responsible for making research planning and risk management decisions related to such issues.

As a means for identifying priorities for research planning and risk management, CEA provides a basis for moving beyond analysis to action. A quote attributed to Leonardo da Vinci seems apropos here: “Knowing is not enough; we must apply. Being willing is not enough; we must act.” Implementing the CEA approach is essential to beginning to provide more sustainable solutions to the major environmental challenges facing society.