Post-Normal Science in Practice at the Netherlands Environmental Assessment Agency.

About a decade ago, the Netherlands Environmental Assessment Agency (PBL) unwittingly embarked on a transition from a technocratic model of science advising to the paradigm of "post-normal science" (PNS). In response to a scandal around uncertainty management in 1999, a Guidance for "Uncertainty Assessment and Communication" was developed with advice from the initiators of the PNS concept and was introduced in 2003. This was followed in 2007 by a "Stakeholder Participation" Guidance. In this article, the authors provide a combined insider/outsider perspective on the transition process. The authors assess the extent to which the PNS paradigm has delivered new approaches in the agency's practice and analyze two projects-on long-term options for Dutch sustainable development policy and for urban development policy-the latter in somewhat more detail. The authors identify several paradoxes PBL encounters when putting the PNS concept into practice. It is concluded that an openness to other styles of work than the technocratic model has become visible, but that the introduction of the PNS paradigm is still in its early stage.

Introduction

On January 20, 1999, Dr. Hans de Kwaadsteniet, a senior statistician at the Netherlands Environmental Assessment Agency (PBL)[ 1 ] made news in the Netherlands. After years of trying to convince his superiors that the agency’s environmental assessments leaned too much toward computer simulation at the expense of measurements, he went public with this criticism by publishing an opinion article in the national newspaper Trouw (de Kwaadsteniet 1999). His article was supplemented with an interview that resulted in the headline “Environmental Institute Lies and Deceives” on the newspaper’s front page. His specific claim was that—being based on a virtual reality of computer models—the agency suggested too high an accuracy of the environmental figures published yearly in its State of the Environment report.

Soon after the publication by de Kwaadsteniet, an intense and long-lasting media debate ensued in the Netherlands.[ 2 ] The affair reached the floor of the Dutch Parliament within a matter of days. Facing the Parliament, the Minister of the Environment, Jan Pronk, defended the integrity of the agency. In return for an agreement that the agency would organize more regular external reviews of its activities and improve its communication of uncertainty, the Minister granted the institute additional funding for its monitoring activities. As is described in this article, the response of PBL to the affair was to introduce changes in its working style at the science–policy interface.

The de Kwaadsteniet affair illustrates some of the difficult choices that have to be made by “boundary organizations”—that is, organizations that reside at the boundary between science and policy—in addressing very uncertain issues with high stakes involved. Should scientists communicate about uncertainties with decision makers and let the decision makers decide on how to deal with these uncertainties or should scientists instead give in to the demand often expressed by decision makers to offer “certainties,” in the form of “best available scientific information” (thus, in effect repressing uncertainty)? Underlying the de Kwaadsteniet affair are different views on the role of science in policy.

A traditional perspective on these views is offered by Yaron Ezrahi (1980), who discussed two opposing views: the “utopian rationalist” and the “pragmatic rationalist” view. The “utopian rationalist” or “technocratic” ideal of science advice entails that the policy-making process should assimilate scientific information to a maximum extent. Utopian rationalism may be appropriate in those rare situations of consensus on both knowledge and values that pertain to a policy problem. The technocratic ideal reflects the notion of science speaking “value-free” truth to political power that gained institutional currency in the nineteenth century. The notion of value-free science itself was based on the expectation that the impartiality and objectivity of scientists could help overcome political conflict (Proctor 1991). In the twentieth century, however, it became clear both that science cannot be value-free and that politics deals increasingly often with issues that are clouded with uncertainty, including value diversity. The presence of conflict among scientists, both epistemic and social, makes it hard to provide politicians with neutral advice. There is often considerable room for scientists and policy analysts to make choices in the assumptions of their analysis.

Next to the “utopian rationalist” model, Ezrahi identified a “pragmatic rationalist,” or “democratic” ideal of science advice that accepts, within limits, the inevitability of political ingredients in science advice. Pragmatic rationalism considers technocratic science advising to be mistaken for many policy problems. Uncertainty and value controversy ask for science to contribute to political debate by representing different legitimate perspectives on policy problems. The approach is to recognize and identify the controversies, uncertainties, and ambiguities to open up discussion and stimulate the process of deliberative decision making. The post-normal science (PNS) paradigm that is studied in this article can be understood as a strategy for the production of relevant knowledge, in line with the “pragmatic rationalist” view on the science–policy interface. Funtowicz and Ravetz (1993) introduced the term “post-normal science” for issue-driven knowledge production in a context of hard political pressure, values in dispute, high decision stakes, and high epistemological and ethical systems uncertainties.

But how can this PNS paradigm be put into practice? The experiences of the Netherlands Environmental Assessment Agency may shed some light on this question. After briefly summarizing the theory of PNS in the next section, this article describes how, since the de Kwaadsteniet affair—which caused a severe disruption in the agency’s standard practice—the agency unwittingly embarked on a transition from the technocratic model of science advising to PNS paradigm. The seeds of the agency’s transition to PNS were already present before or during the de Kwaadsteniet affair (Kloprogge and van der Sluijs 2006). Two emblematic projects in this regard were the TARGETS project (Rotmans and de Vries 1997)—which was centered on a computer model that deals with value diversity and uncertainty by allowing multiple model routes depending on a typology of so-called perspectives—and the COOL project (van de Kerkhof 2004)—a participatory Integrated Assessment project aimed at supporting development of long-term climate policy in the Netherlands and run by a consortium, in which the PBL participated to bring in its expertise. The agency has recently described itself as “an interface between science and policy,” which is “independent” and aims to report “the full range of scientific opinions” (MNP 2006). How far the agency has come to incorporating the PNS paradigm is critically assessed in this article.

The assessment is structured by referring to three key elements in the PNS paradigm:As will become evident to the reader, three of the authors of this article (AP, JvdS, and MH) have themselves been involved both in elaborating the theory of PNS and in the introduction of PNS in the practice of PBL. With advice from the initiators of the PNS concept, the agency introduced a Guidance for “Uncertainty Assessment and Communication” in 2003 and, with advice from political scientists, issued a “Stakeholder Participation” Guidance in 2007. These guidance documents offer methods and practical tools to incorporate the PNS approach in the agency’s projects. Two of the authors of this article (EK and AC) were not directly involved with the elaboration and introduction of these guidances; still, one of these (EK) later joined the agency and the other (AC) has acted as a consultant to the agency for the past few years. Thus, the article combines insider and outsider perspectives and it must thus largely be regarded as a self-evaluation.

The management of uncertainty: PNS acknowledges that uncertainty is more than a technical number or methodological issue. Ambiguous knowledge assumptions and ignorance give rise to epistemological uncertainty;

The management of a plurality of perspectives within and without science: Complex problem solving requires scientific teamwork within an interdisciplinary group and joint efforts by specialists from the scientific community and from business, politics, and society;

The internal and external extension of the peer community: An extended peer community includes representatives from social, political, and economic domains that openly discuss various dimensions of risks and their implications for all stakeholders.

To illustrate the paradoxes encountered when the PNS paradigm is implemented into practice, the ideas behind the two guidances are described and two PBL projects are examined. A major example where especially the second element of the PNS approach—the plurality of perspectives—became clearly visible to the outside world, was the agency’s “First Sustainability Outlook” (2004). The second example, examined in somewhat more detail, addresses in particular the third element and concerns a project on long-term options for urban development in the Netherlands which ran from 2008 to 2010. We end this article by discussing the paradoxes the PBL encounters when putting the PNS concept into practice.

The Paradigm of PNS

The interface between science, policy, and society, or “science–policy–society interface” has been the topic of study of a whole body of political science and social studies of science literature (e.g., Jasanoff 1990; Funtowicz and Ravetz 1993; Gibbons et al. 1994; Hisschemöller and Hoppe 1996; Gieryn 1999; Guston 1999, 2000; Miller 2001; Nowotny, Scott, and Gibbons 2001; Collins and Evans 2002; Siebenhüner 2004). Most of these studies have addressed questions pertaining to democratization and stakeholder participation.

The science–policy–society interface is typically not regarded as a static given but instead can be seen as a highly dynamic process in which all kinds of linkages are continuously being formed and broken. The concept of “boundary work” (for an introduction see, e.g., Gieryn 1999; Jasanoff 1990) is used by many studies in the literature to capture this dynamic. Boundary work results in a demarcation and positioning of domains (for instance, science and policy or scientific disciplines) by defining who is seen as competent in specific areas of a policy problem. The boundaries between the domains of competence involved in a debate usually remain subject to continuous renegotiations. This boundary work is work done by all actors to both define and cross the boundaries between science, policy, and society.

Social scientists do not arrive at an equivocal view on the science–policy–society interface, neither on how it works nor on how it should work. In the current article, we focus on the normative theory of “post-normal science,” which entails a reflective way of doing boundary work. The “post-normal science” paradigm (Funtowicz and Ravetz 1993) has its roots in the mid-1980s. Recognizing that the technocratic interactions between science and policy making on controversial risks issues were often unproductive in cases where the decision stakes and system uncertainty are very high (notably in the case of nuclear energy), Sylvio Funtowicz and Jerry Ravetz proposed distinguishing a new type of risk assessment called “total-environmental assessment” (Funtowicz and Ravetz 1985, 228). This is a form of risk assessment in which the “total environment”—that is, the complete context—of a risk issue is taken into account as much as possible.

Similar to the risk assessment of nuclear energy in the 1980s, current day complex issues faced by the Netherlands Environmental Assessment Agency (PBL) are hard to tackle with normal scientific procedures. Activities of the PBL, such as constructing environmental outlooks, sustainability outlooks, or assessing options for sustainable long-term urban development, involve deep uncertainties and value-ladenness, which cannot be dealt with in a “normal science” mode. The PNS paradigm is based on an acceptance of fundamental uncertainty in science and policy and can be regarded as an initial impetus toward addressing the “complexity” of some pressing real-world problems. In comparison with the situation of “normal science,” in which a linear view of the world prevails, PNS respects processes of emergence. Analysts within the PNS framework know that the future is fundamentally unpredictable. While for natural systems, this unpredictability can be mathematically explored using chaos theory involving laws of nature, social systems are even more fundamentally unpredictable since they involve human reflexivity. When we use the term “complexity” here, we mainly refer to the latter element of reflexivity.

Where normal science aims at establishing the ultimate truth or the final resolution of a scientific puzzle, PNS recognizes that as long as both scientific uncertainties and decision stakes are high, such an aim is in principle unachievable. Indeed it can be misleading and create false expectations to act as if the role of science in such issues is just to get the facts right. Instead, PNS aims at common commitments to reflective approaches for dealing with complex policy issues. The PNS approach also allows room for “narratives” as a means of addressing complexity. Narratives provide important additions to factual claims, “because they are not about objective reality, but are statements of what is significant” (Allen and Giampietro 2006, 595).

PNS is often misunderstood as something that replaces normal science. It should instead be seen as a societal problem-solving strategy that partly draws on normal science but emphasizes that for addressing complex policy problems more is needed to shape the science–policy–society interface. Facts are still necessary but no longer sufficient. The guiding principle of normal science—discovering the true facts—must be modified to fit the post-normal principle to achieve the goal of quality (Funtowicz and Ravetz 1991). Traditional methods of quality assurance do not apply under those circumstances. Apart from testing knowledge for validity and reliability, it should also be tested for “social robustness.” The knowledge production is to be organized in a way that increases the social robustness and guarantees scientific quality at the same time. On this argument, Funtowicz and Ravetz (1994) defined the key elements of PNS. These include the appropriate management of uncertainty, the plurality of perspectives and commitments, and the internal and external extension of the peer community.

In the remainder of this article, we review the elaboration of the PNS paradigm in PBL’s guidance documents and two projects to reflect on the transition of PBL from interfacing science and policy in normal “getting the facts right” mode toward the post-normal “working deliberatively within imperfections” (Funtowicz 2006) mode.

Dealing with Uncertainty and Value Plurality

PBL’s Uncertainty Guidance

The government-funded Netherlands Environmental Assessment Agency (PBL) supports national and international policy makers by providing integrated assessments on topics such as sustainable development, energy and climate change, biodiversity, transport, land use, air quality, and spatial planning. After the crisis described in the beginning of this article, the PBL, or, more accurately, its immediate predecessor within RIVM, was reviewed by an international expert committee in 2000, which recommended to start a project to “systematically address terminology, methodology, interpretation and communication of uncertainty” (RIVM 2000). A subsequent review at the national level led to the proposal to develop a protocol or guideline (van Asselt et al. 2001).

In 2001, a project was started at PBL to develop, in collaboration with Utrecht University and involving an international team of uncertainty experts, a Guidance for Uncertainty Assessment and Communication (Petersen et al. 2003; Janssen et al. 2003; van der Sluijs et al. 2003; van der Sluijs et al. 2004; Petersen 2006; van der Sluijs et al. 2008). In the project, an early choice was made to de-emphasize (though not neglect) two dominant understandings of how to deal with uncertainty at the science–policy–society interface—the “deficit view” and the “evidence evaluation view” (explained below)—and to focus more on the further development of the “post-normal view” (van der Sluijs 2006).

Within the deficit view, uncertainty is considered a deficit of available knowledge. Uncertainty is seen as a temporary problem that will disappear automatically if more research would be performed. In this view, management of uncertainty equals reduction of uncertainty and there is a strong belief that science is ultimately able to provide certainty. Where uncertainty can not yet be reduced, uncertainty is treated as if it were a fact (usually a percentage number that represents for instance two standard deviations over a mean value) that needs to be discovered and correctly quantified. In monodisciplinary domains there is a whole science on how to calculate the “true” uncertainty in for instance measurement data (e.g., Taylor 1982). One tendency typically seen in the deficit view is the production of ever more complex and detailed models, because precise calculation is seen as key to truth. The techniques applied include Monte Carlo, Bayesian belief networks, and other quantification techniques. A pitfall of this paradigm can be that a false certainty is created, because the numbers obtained from these models can suggest more knowledge than there actually is. The deficit view is often encountered in monodisciplinary normal science. At PBL, the deficit view was especially prevalent among modelers and economists.

The evidence evaluation view considers uncertainty to be a problematic lack of equivocalness. When science speaks with multiple voices to policy, conflicting certainties may emerge. The solution proposed is a comparative evaluation of individual research results, focused on building scientific consensus. The focus shifts from establishing certainty to evaluation of evidence to establish gradations of certainty. Multidisciplinary expert panels such as the Intergovernmental Panel on Climate Change (IPCC) have been established for this purpose. This view focuses on generating robust conclusions and widely shared interpretations of the available limited knowledge. A pitfall of this paradigm can be that matters on which no consensus can be reached continue to receive too little attention, while, in fact, this dissension is often highly policy-relevant. Especially weak signals of early warning of (new) risks are likely to be overlooked or underaddressed here, because it often is impossible to reach a consensus interpretation on such issues. The evidence evaluation view was recognizable in PBL’s multidisciplinary assessments of complex policy problems.

The post-normal view that was further developed in PBL’s Uncertainty Guidance sees uncertainty as intrinsic to complex systems and thus as a permanent phenomenon. In further contrast with the deficit view, it sees uncertainty as more than a number. It stresses that uncertainty also results from the new ways by which knowledge on complex policy issues is produced. For instance, the use of computer simulation models, scenarios, and extrapolations all critically depend on the validity of the assumptions that unavoidably need to be made. Most of such assumptions can in principle not be validated (see also Konikow and Bredehoeft 1992; Oreskes, Shrader-Frechette, and Belitz 1994). This implies that if knowledge is produced that is conditioned on (the unknown validity of) assumptions, uncertainty is unavoidably coproduced. The post-normal view acknowledges that not all uncertainties can be quantified and that in complex issues, unquantifiable uncertainties can well be more relevant and salient than the part of uncertainty for which we have enough knowledge to quantify it in some reliable way. It calls for an approach that openly deals with deeper dimensions of uncertainty, such as those stemming from problem framing, choice of system boundaries, indeterminacy, ignorance, assumptions, value loadings, underdetermination (the same data allowing for several interpretations and conclusions), and even institutional dimensions.

The Uncertainty Guidance proposes a more qualitative and reflective approach to uncertainty (van der Sluijs et al. 2008). Techniques that are applied to deal with this are Knowledge Quality Assessment tools such as the NUSAP system (van der Sluijs et al. 2005) and risk management (including production of knowledge) as a deliberative (participative) social process. A pitfall of the PNS paradigm is that uncertainty can be highlighted to such an extent that we may forget how much we actually do know about the risk concerned and on which aspects there is, in fact, consensus. Within the Guidance, the PNS paradigm therefore complements and does not replace the consensus approach and the uncertainty quantification approach.

In the post-normal domain, the traditional forms of scientific quality control such as peer review and validation run into their limitations. For instance, if scientific insights required for decision problems cannot be produced without computer simulation models, the resulting knowledge cannot be validated (Oreskes, Shrader-Frechette, and Belitz 1994). To maintain quality, a modeler has to be a good craftsperson (Ravetz 1971). In cooperation with PBL, Utrecht University developed several Knowledge Quality Assessment tools to assist modelers. Discipline can be maintained by controlling the introduction of assumptions into the model and maintaining standards of “good practice.” A new method was developed to identify, review, and prioritize assumptions to assess the potential value-ladenness of important assumptions and to deal with these potentially value-laden assumptions in an explicit and transparent manner (Kloprogge, van der Sluijs, and Petersen 2011). Furthermore, Risbey et al. (2005) developed a heuristic that encourages self-evaluative systematization and reflexivity on pitfalls. It takes a reflective approach to the management of quality by providing guidance on good modeling practices and at the same time providing diagnostic help as to where problems may occur and why. It recognizes that models are tools, not truths. A model is not good or bad but there are “better” and “worse” forms of modeling practice. A crucial notion is that models are “more” or “less” useful when applied to a particular problem. Reflective approaches to quality can thus provide insurance against pitfalls in process and insurance against irrelevance in application. Risbey et al. (2005) applied their approach to PBL’s global energy model TIMER.

The Uncertainty Guidance distinguishes six parts of environmental assessments that merit separate attention: (1) problem framing; (2) involvement of stakeholders; (3) selection of indicators; (4) appraisal of the knowledge base; (5) mapping and assessment of relevant uncertainties; and (6) reporting of the uncertainty information. The Uncertainty Guidance tackles all three key elements of PNS theory mentioned in the introduction: the appropriate management of uncertainty and of quality; the plurality of perspectives; and the extension of the peer community. The Guidance is set up as a checklist with pointers to hints and actions of good practice.

In the view on the science–policy–society interface that underlies the Uncertainty Guidance, the shaping of this interface is strongly related to the framing of the policy problem. In the first step of the guidance, the problem and its context and history are outlined, by identifying major issues, past work, the level of contention, and the (expected) role of the assessment in the policy or decision-making process. The assessor is explicitly asked to consider various views/perspectives on the problem and to pay attention to the problem’s interconnectedness with other problems. She or he is asked to be specific on what knowledge is needed with regard to the problem and into which research questions this is translated. Possibly relevant aspects that are not dealt with in these research questions have to be indicated. Moreover, it should be outlined what role the study is expected to play in the policy process and what the relation is with previous studies on the subject (policy context and problem history).

Although the Uncertainty Guidance is not fully used within all PBL projects yet, it is increasingly used, attitudes toward dealing with uncertainty in performing and reporting environmental assessments—the first PNS element—have changed, and communication on uncertainty in PBL reports has improved over this period (Wardekker et al. 2008). Below we will briefly illustrate how also the second PNS element—dealing with value plurality—has received a more prominent role in PBL.

PBL’s First Sustainability Outlook

The Uncertainty Guidance’s advice on explicitly addressing different perspectives was most visibly put into practice in an important agency project that was concluded in the year after the guidance was introduced: the First Sustainability Outlook (MNP 2004; de Vries and Petersen 2009). In this project, after much heated internal debate on how global sustainable development should be conceptualized and operationalized for a country such as The Netherlands, a method was developed to explicitly deal with divergent perspectives on the problem, which built on the experiences gained by the agency in the TARGETS project. Such explicit treatment of underlying value commitments proved to be a radical break with the way environmental assessment were conducted previously in the agency.

Starting point of the method is that a sustainability assessment should investigate the ability to continue and develop a desirable way of living vis-à-vis later generations and life elsewhere on the planet. Evidently, people hold different values and beliefs about the way societies sustain quality of life for their members. The first step, therefore, is to analyze people’s value orientations and the way in which they interpret sustainability problems, that is, their beliefs. The next step is to translate the resulting worldviews into model-based narratives, that is, scenarios. The qualitative and quantitative outcomes are then investigated in terms of associated risks and opportunities and robustness of policy options. This method was applied, using extensive surveys among the Dutch population, in the First Sustainability Outlook. The resulting archetypical worldviews became the basis for four different scenarios for policy analysis, with emphases on the domains of transport, energy, and food. The goal of the agency’s Sustainability Outlooks is to show that choices are inevitable in policy making for sustainable development, to indicate which positive and negative impacts one can expect of these choices (trade-offs), and to identify options that may be robust under several worldviews.

The use of different worldviews provides a way to deal with the plurality of perspectives on sustainability problems. Obviously, for a policy assessment on sustainable development to be able to play a significant role in structuring the policy debate, stakeholders need to be engaged and feel represented in the worldviews used. In this project, the level of participation was still kept relatively low. Scenario experts were consulted on the scenario methodology and consultative sessions were held at four ministries, the social–economic council, a government strategists group, a high-level environmental network organization, a group of university professors and three nongovernmental organizations (NGOs). Still, the methodology of worldviews as a tool for strategic policy making proved to be helpful in policy exercises that were performed after the assessment had been published.

Dealing with an Extended Peer Community

PBL’s Stakeholder Participation Guidance

In the post-normal domain, scientific and technical discourse is no longer restricted to expert communities but needs to be inclusive of nonspecialist participants (stakeholders and citizens; Funtowicz and Ravetz 1993). This requires the involvement of an extended peer community and participation of stakeholders and citizens not only in the phase where solutions are debated but also in the assessment process that precedes it. Since no particular expertise can deliver certainty for policy issues in the post-normal domain, no expertise can claim a monopoly of wisdom and competence. Indeed, relevant wisdom is not limited to scientific specialists and public officials.

Stakeholders can contribute to knowledge in a number of ways: by co-framing the problems addressed by the scientists (e.g., what are the relevant aspects of a problem that need be included), by acting as a source of nonscientific knowledge, and as a source of critical reflection in quality control of the scientists' work (e.g., by challenging the validity of assumptions made in assessment studies).

Thus, PNS requires a dialogue based on the recognition of a plurality of legitimate perspectives and areas of expertise vis-à-vis the problem, each with its own contributions of relevant evidence, commitments, and insights. Its goal should not be to establish which single voice is “right,” inevitably making the others “wrong” (Funtowicz and Ravetz 1994). Instead, a rich and diverse body of (sometimes conflicting) potentially policy-relevant evidence is coproduced and matures by criticism across the perspectives involved in the inclusive dialogue.

This PNS mode of interfacing science and policy can be summarized as “working deliberatively within imperfections” (Funtowicz 2006; Funtowicz and Strand 2007; van der Sluijs and Funtowicz 2008). This requires new interdisciplinary contacts and integration (internal extension of the peer community) on one hand, and new ‘‘knowledge partnerships'’ with policy makers, NGOs, industry, media, and the public (external extension of the peer community) on the other hand, to meet the challenges of quality control in the assessment of complex risks.

After the Guidance for Uncertainty Assessment and Communication was published in 2003, the Netherlands Environmental Assessment Agency decided to add a more detailed Stakeholder Participation Guidance (Hage and Leroy 2008; Hage, Leroy and Petersen 2010) to its toolkit for environmental assessors, to facilitate both types of peer community extension. It is organized around a number of guiding questions:One of the most important things when organizing participation is to formulate clear aims. The Netherlands Environmental Assessment Agency does not consider stakeholder participation in science-for-policy as an aim in itself for each project. But for cases in which stakeholder participation is found important, the Stakeholder Participation Guidance offers advice on how to organize it.

Why do you want participation?

What should the participation be about?

Who do you want to involve?

How much participation do you want?

What form are you choosing?

The “Sustainable City” project, which explores long-term options for urban development in the Netherlands, is the first “PBL-only” project in which stakeholder participation plays a central role. It started in 2008 for a duration of two years. Below we present some insights into the relation of the project to PNS practices (addressing not only the extension of the peer community but also the other two key elements of the PNS paradigm: managing uncertainty and a plurality of perspectives).

PBL’s Sustainable City Project

The aim of PBL’s Sustainable City Project was to generate integrated options for long-term urban development policies in the Netherlands. The project started from the assumption that today’s policy on urban development is too fragmented and not well considered concerning side-effects and trade-offs. By integrating different scientific disciplines and different kinds of knowledge, the project tries to identify those policy options that will trigger sustainable urban development. The central element of the project consists of stakeholder dialogues concerning three main topics: health, livability, and energy. The procedure was modeled to the participatory backcasting approach (see Quist and Vergragt 2006; Carlsson-Kanyama et al. 2003; Carlsson-Kanyama et al. 2008). The primary idea of the project is to ask stakeholders from different functional areas (health, energy, and livability) to deliver their ideas about the “sustainable city” in the year 2040, within their field of expertise. The project team made the choice to start from three different perspectives to identify synergies and trade-offs between these three aspects of sustainability.

The choice for the participatory approach was triggered by the growing awareness at the agency that “wicked problems” such as sustainable development need another methodological approach than the traditional ones. The project aimed at method development to address this type of problems, using an analytic–deliberative approach to address sustainability in an urban context. An analytic–deliberative approach offers an opportunity for creating “a decision framework that stresses not only technical information, but also the explicit input of values, insights and tradeoffs” (Petts 2004, 116), which can be obtained with stakeholder involvement. At the same time, an overall philosophical discussion about the conceptualization of “quality of life” emerged at the agency after the publication of the First Sustainability Outlook (for a brief description, see previous section). A highly abstract definition of quality of life was considered as not useful for this project and it was decided to take a “hands-on” approach to sustainability in the urban context. Knowledge about the important issues for urban areas was judged as poor, because there are hardly scientific models that can assess sustainability at a local level. Also for this reason, the project leader decided to involve stakeholders: to improve the knowledge base by asking the stakeholders to bring in their knowledge of the urban context.

Three rounds of workshop were conducted. In the first workshop, stakeholders were asked for their ideas to make an imaginary city respectively more healthy, livable, and energy-neutral and to enhance the quality of life. The group of stakeholders consists of practitioners, (local) policy makers, architects and builders, NGO representatives, and scientists. In between the two workshops, the stakeholders were given the opportunity to elaborate their statements made in the workshop by participating in an internet-facilitated discussion. In the second workshop, the stakeholders were asked to identify pathways to the future, using the participatory backcasting method.

The project team assessed the values, beliefs, and knowledge claims on the future orientation of sustainable urban life and their implications for spatial planning.

A mixed assessment approach is applied consisting of scenario analysis and narrative analysis. Narrative analysis served as input to development of the policy scenarios. Scenario analysis was done by means of urban systems simulation modeling and impact assessment on health, livability, and energy. A selected set of stakeholders was invited to peer review the analysis process and its outcomes, paying particular attention to the modeling assumptions. Narrative analysis offers insight into normative and institutional perspectives on sustainable city planning. The stakeholders support that sustainable development at the urban level asks for a transition to a different governance structure, where new and diverse alliances of local citizens, business, and organizations are responsible as producers and consumers of societal value. The role of the national government should be to provide clarity and consistency by offering a challenging and inclusive future vision, while implementation is bounded to local and regional levels where economic, ecological, and social sustainability can enhance and strengthen one another.

At the third, final workshop the assessment findings were demonstrated on the basis of the model outcomes of the scenario analysis. The question was addressed of how the national government can support local government and private parties in developing sustainable urban environments, knowing that existing structures and positions will have to change into new alliances.

Can the project “Sustainable City” be classified as post-normal? As probably most of sustainability research projects, the “Sustainable City” project contains post-normal elements but is actually a hybrid package of different kinds of topics with different kinds of uncertainties and different degrees of value-ladenness. The project design gives an important role to stakeholder participation to enhance the quality of knowledge by bringing scientists and practitioners together and making use of practical knowledge. However, during the design and assessment process of the project, the project team had to face different dilemmas that revealed a tension between post-normal and normal science. This is most obviously expressed by noting that the project members only realized later in the project how important the narrative elements of the stakeholder dialogue actually were for reconciliation of the deliberative and analytic processes. Using rigorous methods such as scenario design and system-bounded models only partially allowed for incorporation and processing of the creativity and out-of-the box ideas that were generated by the stakeholders. This reductionist approach is inherent to the use of such methods. The seeds of opening up this project to a more PNS-like approach were present in PBL’s organizational adoption of the Uncertainty and Stakeholder Participation Guidances and was also due the participation of two of the authors (MH and EK) within the project team to help guide the project toward more openness. The process of opening up is illustrated with reflective fragments of the project in the following paragraphs.

Framing of knowledge

One of the central questions was how far one could and should control the stakeholder interaction for the benefit of the projects' research question. The initial idea was to give the stakeholders the same clear-cut definition of the issue at stake as the definitions the models are based on. This should also avoid unnecessary discussions and create a common starting point of debate. However, such a restriction would do no good for the motivation of the participants and the dynamics of the workshops. That is the reason why the project team decided to give the stakeholders the opportunity to co-frame the issue at stake. The first workshop endorsed this expectation: the participations did not want to discuss about criteria and cause–effect relationships but did want to express their perspective about what the “good life” could look like. An example from the first health workshop can illustrate this: the discussion shifted from definitions of health and effectiveness and cause–effect relationships of health-improving measures in urban design to quality of life in general, happiness, and social interaction.

Use of stakeholder knowledge

A major challenge of the project was to realize a dialogue between the (mostly quantified) scientific knowledge of the project team (the models used for health, energy, and livability) on one hand, and the practical knowledge of the stakeholders on the other hand. At the beginning of the project, stakeholders were considered as a source of knowledge that can be used like other sources of quantifiable information. But it turned out that the initially expected role of the stakeholders—to “feed” the models of the scientists involved with quantifiable information—was hard for them to fulfill. If the project team reduced the contribution of the stakeholders to the limited parameters of the models, a lot of valuable information would have been lost. For this reason, the project design opened up to a more narrative approach, allowing for a debate on what the “good life” could be instead of prescribing such concepts from a scientific perspective.

Scale of knowledge

Another methodological problem of the project was the tension between generalized and contextual knowledge on the topic of urban development. The Netherlands Environmental Assessment Agency reports to the national government. The knowledge production process of the “Sustainable City” project is therefore oriented to produce knowledge that facilitates sustainable urban development on a national policy level. This should be generalized knowledge, independent from a particular local context. That is why the project team chose to ask the stakeholders to create future visions of an imaginary city and not an existing one. The project team was afraid of getting a discussion on “pavement level” concerning a local context that would not deliver useful information. However, contextual knowledge is essential for obtaining insight into policy processes and feasibility of strategies. Additional case study research would probably have delivered valuable insights into the feasibility of the options for urban development. The tension perceived in this project illustrates the general problem of the agency to make use of case study methods. The agency is respected to produce context-independent knowledge that is generalizable at a macro level. Knowledge produced by qualitative empirical social research does seldom meet this expectation.

Stakeholder selection

The selection of the stakeholders took place from an expert perspective. Participants should deliver ideas and policy options from their field of expertise. They should also be able to think on a higher level of abstraction to deliver context-independent knowledge. This resulted in a selection where participants were mostly highly educated, high income class, and white. Considering the value-ladenness of the issues at stake, other criteria of stakeholder selection with a better representation of the different stakes and societal perspectives on the whole might have been more appropriate. This could have strengthened the knowledge base and the quality of the dialogue in terms of social robustness.

When we evaluate this ongoing project, we see signs of a shift from technical and analytic research to analytic–deliberative research where model-driven knowledge is reconciled with stakeholder knowledge, using quantitative and qualitative approaches in liaison. As a first conclusion, we see that managing plural perspectives and opening up to “extended peer review” requires less a revision of the traditional scientific knowledge than a shift in focus of the research design: the Sustainable City project opened up to a more narrative approach. As a consequence, however, the project suffered from a lack of focus. A lesson learnt from this project is that involving an extended peer community into a debate on wicked problems requires more intensive framing to allow for in-depth analysis of the various problem perspectives.

As a second, more general conclusion, one might say that this project demonstrates how PBL is seeking to reposition itself into a more post-normal reality while in a way maintaining the status of a “normal” scientist who is able to produce generally valid long-term options for urban development in the Netherlands.

Discussion and Conclusion

What we have learned over the past few years is that the institutional challenges of implementing practices of PNS, for example, through “guidances,” in a government agency with several hundreds of employees should not be underestimated. It entails much more than disseminating the documents through an organization. For example, PBL’s top management has ordered and subsequently endorsed the guidances; PBL’s methodology group led the development of the guidance; the use of this guidance is now mandatory as part of the agency’s quality assurance procedures; and the staff is actively trained to acquire the necessary skills. They are slowly, but steadily, starting to adopt or more PNS-like mind-set. In addition, a methodological support unit is available in the agency to assist and advise in assessment projects. The required process of cultural change within the institute was consciously managed over the whole period of 2003 until now.

One requisite for successfully implementing the PNS-based guidances is that the agency staff will actually have to understand and share the basic underlying theory. It is not obvious that this is already fully the case. Some evidence for obstacles to dealing with a plurality of perspectives—in particular, a plurality of normative views—was found, for instance, in a socratic conversation held with a small group of crucial staff members (Petersen, Melse, and Cath 2006). Three questions were addressed in this conversation.

The first question posed to the agency staff was “Under what conditions can normative perspective plurality be made fruitful for the staff?” There are many ways to deal with normative perspective plurality, with both pros and cons. And there are different levels at what such complexity can be dealt with (individual, project, and organization). According to the group, the important issue to discuss in this regard was how different perspectives are represented within the agency as compared with the perspectives that are held outside the organization. Three positions were taken: (1) some claimed that the population of employees should be diversified to include more people from the political right; (2) others thought that that cannot be realized and that instead different roles should be played (you do not need to be conservative to act like one), which should be considered as a competence of the professional; and (3) again others disagreed with both previous positions and claimed that the real problem is not a lack of different perspectives, but an abundance of pluriformity, which needs to be dealt with.

A second question followed up on the previous discussion: “Suppose that there is a dominant frame within the agency; can this be remedied by playing roles?” Some participants recognized that the agency staff perform their analyses from a relatively narrow frame. In how far there is room for perspective plurality and dealing with real complexity is a matter that touches upon the mission of the organization? Is it necessary or useful for policy evaluations to consider all complexity and perspective plurality? Is the “post-normal science” method needed for all products that the agency delivers? Others gave examples of playing roles for the purpose of their regular policy evaluation tasks: for example, a Lower House session was organized about air quality, which did produce new insights (“We all agreed with each other and with discussion we did not get any step further within the agency, so we took different roles.”). Such a role play would also have been useful in other cases is noted by participants. Still, a few of the participants would very much like to see the director to make a definite choice after having heard the different viewpoints. Although the majority of people in the group thought it is necessary, there was still some reminiscent fear to communicate externally about different perspectives and uncertainties concerning environmental problems.

The discussion was therefore concluded with a third question: “Shall we put different perspectives next to each other and stay neutral, or shall we make a choice for one position, or—third alternative—are we pragmatic and does it not really matter where we end in dealing with perspective plurality?” The participants preferred to stimulate plurality and note that from internal discussions new insights can emerge and creative solutions can be found. Actually, the policy makers themselves will have to decide under uncertainty and they may appreciate us for giving them uncertainty information (“One of our products is to offer pluriform assessments”). It was concluded that dealing with pluriformity and complexity should be done when it is necessary, but that it cannot be put in a “protocol.”

The current situation at the Netherlands Environmental Assessment Agency manifests a number of paradoxes. While some of the agency staff claim that they work according to a PNS approach, the actual mode of working of most of the agency staff has only slowly begun moving in the direction of the PNS paradigm. In the First Sustainability Outlook, for instance, hardly any stakeholder was involved—four archetypical perspectives were formulated, which were not derived in a bottom-up process or tested for their representativeness and usability. Still the presentation of value-laden aspects of a problem that could also have been treated in a more “objective” manner constitutes progress in the direction of PNS. And the “Sustainable City” project did open up to a post-normal strategy, even though some members of the project team still aimed for “normal” outcomes with assumed general validity for national policy making. Furthermore, given the institutionalized role of the Netherlands Environmental Assessment Agency at the Dutch science–policy interface and regular reorganizations (the latest due to a merger), the modest progress made in the direction of a PNS strategy should be considered a substantial result. It is not clear how much further the agency could go even, without losing some of its credibility in the policy domain (based on the image of “normal science”). In February 2010, the agency was charged with evaluating the quality of the Working Group II contribution to the Fourth Assessment Report by the IPCC in 2007. PBL approached this task by performing a limited evaluation within the PNS paradigm. PBL’s scientifically educated policy analysts, who are not experts in climate-impacts science, represented the extended peer community, as well as several interested Dutch individuals who submitted comments on the IPCC report to PBL’s public registration Web site. Through its evaluation, published in July 2010 (Meyer and Petersen 2010), PBL reflected on the qualitative judgments and value commitments that necessarily permeate the IPCC assessment process. The PNS approach taken by PBL to evaluating the IPCC report—by critically focusing on issues of transparency as outsiders—was found risky vis-à-vis PBL’s own credibility. The IPCC could have publicly denounced PBL as not being sufficiently competent in the area of climate-change impacts (a “normal science” criticism of the evaluation report). Until policy expectations toward the role of PBL ask for socially robust knowledge in addition to the scientific rationale, possibly other institutions are better suited to take the PNS approach to its extremes.

Even the modest progress toward PNS observed for the agency involved a considerable implementation effort. The new strategy for dealing with uncertainty and stakeholders has been introduced mainly by way of a training in the order of two workdays, but this initiation in PNS paradigm is not sufficient to reach the level of in-dwelling (cf. Polanyi 1958) needed for thorough understanding. A change in working procedures is a slow process that naturally is met with resistance. Change is facilitated by external factors (such as societal expectations toward PBL) and internal factors such as availability and acquaintance with new practices for knowledge production. The training is focused on recognizing and acknowledging uncertainty, and as such can be considered useful. However, for the agency to fulfill its self-proclaimed “knowledge brokerage” function (the agency calls itself publicly a “bridge” between science and policy), it runs into the problem of an effective maintenance of multidisciplinarity and complexity.

Knowledge that is disconnected from values, circumstances, and concrete persons is in danger to reduce complexity to the extreme. Multidisciplinarity does not just require an ability to see from more than one point of view. It has to provide an ability to let the different perspectives (in science, policy, and society) communicate with one another, influence one another, and reach some sort of coherence. However, by emphasizing “science” and denouncing “postmodernism” (cf. Funtowicz and Ravetz 1985), there is the risk in the PNS paradigm of accepting the truth claims of what the stakeholders or practitioners acknowledge and take for real. The risk of self-reproducing truth claims—in particular with respect to social reality—is a clear and present danger when practicing PNS in a context of normal science expectations. The methodological challenge ahead, not only for the organization studied in this article, is to find answers to the question how can we reach understanding which addresses complexity even deeper. As said, it remains questionable whether an agency such as PBL can go even further than the PNS paradigm, given its institutional context and the obstacles that are already encountered in moving toward the PNS paradigm. But even though the introduction of the PNS paradigm is still in its early stage in PBL, we can conclude that an openness to other styles of work than the technocratic model has become visible in PBL’s practice.