Politics of policy learning: Evaluating an experiment on free pricing arrangements in Dutch dental care.

In Dutch healthcare, new market mechanisms have been introduced on an experimental basis in an attempt to contain costs and improve quality. Informed by a constructivist approach, we demonstrate that such experiments are not neutral testing grounds. Drawing from semi-structured interviews and policy texts, we reconstruct an experiment on free pricing in dental care that turned into a critical example of market failure, influencing developments in other sectors. Our analysis, however, shows that (1) different market logics and (2) different experimental logics were reproduced simultaneously during the course of the experiment. We furthermore reveal how (3) evaluation and political life influenced which logics were reproduced and became taken as the lessons learned. We use these insights to discuss the role of evaluation in learning from policy experimentation and close with four questions that evaluators could ask to better understand what is learned from policy experiments, how, and why.

Introduction

In many healthcare systems, market mechanisms are introduced on an experimental basis to try to contain costs and improve the quality of services (Grit and Zuiderent-Jerak, 2015; Harrison and Calltorp, 2000; Selker and Wasser, 2014). In line with this development, scholars have begun paying attention to how policy learning takes place through experimentation (Picciotto, 2012) and the role evaluation should play in the process (Arkesteijn et al., 2015; Martin and Sanderson, 1999). We contribute to this nascent field by studying a policy experiment ‘in practice’.

In the literature, two distinct epistemic approaches can be identified that focus on learning in relation to policy experiments (Picciotto, 2012). In the first, policy experiments are considered tests, designed to examine solutions to relatively concrete problems with well-established rigorous methods (Ludwig et al., 2011). In the second, policy experiments are seen as protective spaces for reflexive learning (Sabel and Zeitlin, 2012). These approaches have conceptualized the role of evaluation differently. Emphasis is placed either on the summative ex post measurement of outcomes, or on the formative ex durante monitoring of changes and the fostering of dialogue between stakeholders (Arkesteijn et al., 2015).

Although these approaches to experimentation, learning and evaluation seem conceptually and operationally irreconcilable, they can – in practice – be entrenched in a single experiment (Regeer et al., 2016). An excellent example is the experimental introduction of price liberalization in Dutch dental care.

On 1 July 2011, Healthcare Minister Schippers decided to introduce this experiment to encourage entrepreneurship, improve service quality, stimulate product innovation and contain costs (Schippers, 2011c, 2011d). The experiment should continue for three years, with regular evaluation and adjustments made, when and where necessary. Depending on the results, structural changes in the financing system for Dutch dental care could be implemented (Schippers, 2011a, 2011b, 2011c). At first sight, the experiment appeared to be an opportunity for reflexive learning.

But instead of lasting three years, the experiment was cancelled after six months, mainly because of an observed price rise. As opposed to reflexive learning, a tester-oriented logic had become dominant. According to this view, the experimental dental care market had been put to the test, but rising prices demonstrated that the market had failed (Kuiken, 2012). Although generally agreed to be a failure, it did produce learning. According to healthcare economists, politicians and consumer organizations, the failure proved that transparency on quality – a criterion considered lacking in dental care – should be deemed a prerequisite for any healthcare market to function properly (Varkevisser, 2012).

Nowadays, the importance of quality transparency has become mainstream in Dutch policy discourse. Nevertheless, at the time of the experiment, its cancellation, the lessons learned and the role of evaluation were highly contested. In line with this observation, we take a constructivist approach and argue that policy experiments with market mechanisms are not neutral testing grounds (Muniesa and Callon, 2007). To better understand how we learn what from policy experiments, it is important to make explicit the processes that produce the lessons learned (Van Assche et al., 2014) and the role that evaluation has in these processes (Martin and Sanderson, 1999).

Following Van Assche et al. (2014), we argue that special attention should be paid to the different logics – unique sense-making models based on particular distinctions, observational procedures, conceptual mappings and causal inferences – that are reproduced in the policy experiment and its evaluation. To understand how and why some logics get reproduced and inform lessons learned, whereas others get ignored, even forgotten, special attention should be paid to how different logics and practices imply and reinforce one another (Van Assche et al., 2014; see Butler, 2010; Muniesa and Callon, 2007). To reveal that process in the Dutch dental care experiment, we pose the following research questions:

What experimental and market logics were present in the experiment? Which of these formed the basis for the lessons learned and why was this the case?

Guided by these questions, we reconstruct the experiment in Dutch dental care. This reconstruction contributes to the literature on evaluation and policy experimentation. A shift is occurring in this literature, from conceptualizing the role of evaluation as ex post reporting of outcomes, to making explicit the processes through which policy experiments produce their outcomes (Dixon-Woods et al., 2011; Martin and Sanderson, 1999; Pawson and Tilley, 1997). Now, the experiment, its context and its outcomes have become objects of scrutiny and learning. In this article, we take this development a step further, aiming to understand an experiment’s observed outcomes – and lessons learned – by revealing different logics reproduced during the experiment by different actors involved. In line with this approach, we close our paper with a methodological note directed to scholars of evaluation and evaluators who are particularly dedicated to making explicit – or to intervene in – the processes through which lessons are learned (Arkesteijn et al., 2015; Dixon-Woods et al., 2011; Regeer et al., 2009, 2016). But before presenting our findings, we first develop the theoretical lens through which we scrutinize our case: the policy experiment in dental care.

Experiments and politics of policy learning

Two distinct epistemological approaches can be identified in which learning by experimentation is discussed (Picciotto, 2012; Vaessen and Raimondo, 2012).

The first approach conceptualizes learning by experimentation as a testing ground. In this line of reasoning, the experiment functions as a time–space in which to test a predefined policy intervention under controlled circumstances (Ludwig et al., 2011). The policy intervention is introduced into a small, controlled setting that includes a limited number of variables (Duflo and Kremer, 2005; Oakley, 2007). Ideally, the effects of the intervention are measured and compared to a control setting which has not had the intervention introduced (Campbell, 1998). Often, though, control settings are absent or difficult to arrange, and the post-intervention situation is compared to the pre-intervention situation (Duflo and Kremer, 2005; Oakley, 2007). The results from these tests are evaluated ex post and used to inform ex ante predictions on the effects of scaling up a policy intervention (Greenberg and Morris, 2005; Wolpin, 2007). In this approach, the role of evaluation is distant and summative, geared towards measuring impact with predetermined indicators and placing a value on whatever is measured by relating it to predetermined programme objectives (Martin and Sanderson, 1999).

The second approach conceptualizes learning through experimentation as a process of reflection and improvisation in a multi-stakeholder context (Friedman, 2001; Patton, 2008). Here, experimentation is about creating opportunities to obtain new ideas and insights and not testing predetermined policy formats or programme theories (Sabel and Zeitlin, 2012). To do so, the experiment is conceived as a protective time–space that is informed by – but detached from – the stabilizing forces that impede change. Examples are formal rules and routines, existing infrastructures and dominant material investments (Arkesteijn et al., 2015). These experiments allow stakeholders to interact, bring different knowledge and practices together and reflect on the results of the interaction. Reflection can lead to proposing further steps, implementing these and a new round of reflection. In this line of experimentation, objectives are intangible to some extent and redefined over the course of the experiment (De Wildt-Liesveld et al., 2013). Here the role of evaluation is engaged, reflexive and formative (Martin and Sanderson, 1999). Some scholars describe evaluators as change agents who critically engage in formulating problems, implementing preliminary interventions, inductively mapping the impact and formulating new steps to be taken, informed by the primary lessons learned (Arkesteijn et al., 2015; Regeer et al., 2009).

Both approaches to experimentation and evaluation can be considered ideal types that further their own ideas about the role and design of experiments and how to interpret and evaluate their results. In practice, however, policy experiments are often conducted in vivo (Muniesa and Callon, 2007), meaning that interventions are introduced into real-scale, real-life settings, involving an undefined number of stakeholders and participants with their own ideas of what the experiment is for and how it should be designed. If we want to understand how and what we learn through policy experiments, it is essential to gain insight into the ways in which experiments actually unfold in practice. Here, lessons from constructivist literature are of great value.

This literature specifically focuses on experimental practices (Latour, 1987; Muniesa and Callon, 2007) and the social construction of markets (Callon, 1998; Mackenzie et al., 2007; Mitchell, 2008). It uses the concept of performativity to address the process of experimentally introducing economic propositions, drawn from economic theories, into socio-material arrangements and through selective observation and demonstration, stabilizing them into the social-material setting. Here, the key mechanism is that the observed outcomes of an introduced economic proposition are starting points for further action and thus inform policy learning in a certain direction (Mackenzie et al., 2007). Meanwhile, the original proposition is reproduced through newly proposed and implemented action (Van Assche et al., 2014). In revealing this process, constructivist scholars have been able to trace how a particular market turned out the way it did and the role of an economic proposition in that process.

The experimental introduction of an economic proposition derived from a particular economic theory is rarely an exclusionary event, reserved for the reproduction and stabilization of that single economic proposition (Butler, 2010). Policy experiments usually involve an undefined number of stakeholders and participants who bring into – or project onto the experiment – their own languages, conceptual mappings and causal inferences (Muniesa and Callon, 2007; see Garrety and Badham, 2004). Besides market logics, other logics are also introduced to – or are part of – the socio-material setting into which an economic proposition is introduced. Examples are the different logics for what an experiment is and how it should be evaluated.

If we shift our attention from what is actually learned from the policy experiment to the experimentation process, then it becomes apparent that diverse propositions, logics and practices not only imply policy learning, they are also implied in reproducing one another in the course of the unfolding experiment (Butler, 2010; Van Assche et al., 2014). In other words, certain ways of approaching experiments fit better with certain ways of approaching markets, certain ways of approaching healthcare, and certain acts of ‘capital P politics’; the latter referring to the formal or informal organization and enactment of government (Marres, 2013). For example, different conceptualizations of the market can become part of a ‘capital P political’ debate and certain acts of leadership, framing and parliamentary presentation can influence which conceptualization becomes dominant during evaluation.

Different logics can thus create conditions for one another to be reproduced, stabilized and taken as starting points for further action (Butler, 2010; Van Assche et al., 2014). Different logics can also create conditions that halt another’s reproduction. Logics might thus also be implicated in each other’s undoing (Butler, 2010). What is learned from the experimental introduction of market mechanisms depends on how different concerted logics are reproduced, in concurrence, at the cost of other logics. Inspired by Butler (2010, on performative politics), we refer to this process as the politics of policy learning.

In this paper, we pay close attention to how the different market and experimental logics implicated one another during the unfolding experiment. Additionally, we demonstrate how some concerted and implicated logics constituted what was learned from the experiment, at the cost of other logics. Evaluation and ‘capital P politics’ played an important role in this process. Reconstructing a politics of policy learning, we argue, is therefore key to a critical and fuller understanding of the epistemological and political dimensions behind the experimental creation of healthcare markets. Only when we understand the different logics and evaluative and ‘capital P political’ practices implicated in policy experiments can we better understand the experiment’s outcomes and consider whether and how these should inform the governance of future healthcare markets (see Butler, 2010).

Methodology

Below we present our reconstruction of the policy experiment on free pricing arrangements in Dutch dental care. This experiment provides an excellent opportunity to reveal a politics of policy learning that is inherently part of policy experiments, especially because of the participating stakeholders’ strikingly different approaches to the market and the experiment.

Please note, we [the authors] did not participate in this policy experiment as researchers or evaluators. Our findings are based on a historical reconstruction. Reconstructing past events through a combination of archival data and interviews – the latter pointing to the inclusion of contemporary interpretations of past events – faces the problem of dealing with retrospectively constructed meaning in a new and different actuality (Dirksmeier and Helbrecht, 2008). Considering this caveat, we argue that our findings must be read as a contemporary picture of past events. Our findings are, however, important beyond their historical value because they reveal a politics of policy learning that has methodological implications for evaluators ex post evaluation – or ex durante participation in – policy experimentation. We come back to this point in our discussion.

Our study is based on document analysis and semi-structured interviews. Data gathering and analysis was a cyclical, iterative process in which document selection and analysis complemented the conducting and analysis of interviews (Marshall, 1996). In this process, letters and policy documents helped us to identify the specific logics through which the experiment was observed and described and to place events and interviewees’ interpretations in the context of time (Aminzade, 1992). In total, we analysed four policy documents, five organization websites, 14 letters to and from the House of Representatives and 31 newspaper articles. Policy documents and letters were obtained from the digital archives of the Netherlands Healthcare Authority (NZa) and the Ministry of Healthcare, Welfare and Sports (VWS). News items were selected on the keywords: ‘experiment’ [experiment], ‘dental care’ [mondzorg], ‘free prices’ [vrije prijzen], ‘rising prices’ [prijs stijging(en)], ‘price liberalization’ [prijs liberalisering], ‘dentists’ [tandartsen].

To obtain multifaceted insight into how the policy experiment unfolded, we conducted interviews in addition to our document analysis. We used purposive sampling (Marshall, 1996). Here, we used documents and news items to identify actors involved in the experiment. Striving for inclusive actor representation, we furthermore followed a strategy resembling stakeholder analysis. We identified actors who had an interest in the experiment, were affected by it, or who influenced the decision-making process. Additionally, we asked respondents to point out relevant others (Varvasovszky and Brugha, 2000).

The interviews (n = 13) were conducted between March and September 2015. We approached politicians (N = 2), representatives of professional organizations (N = 4), dentists (N = 4), consumer organizations (N = 2) and a member of the NZa (the Dutch healthcare inspectorate responsible for the development of healthcare markets) (N = 1). We chose to conduct semi-structured interviews. We asked interviewees to reflect on their objectives for the experiment, their observations on its design, what they aimed to achieve and how they interpreted the outcome. All interviews were audiotaped, transcribed verbatim and coded. We member-checked our interpretation of the interviews by sending our reconstruction of the experiment, highlighting quotes and insights from each interview, to our respondents. Below, the official public texts by the actors are reported with actual names. However, interviewees’ responses are anonymized. All quotes are translated from the Dutch.

During our analysis, we focused on the use of concepts (e.g. creating or guarding), the drawing of distinctions (e.g. is or is not an experiment) and the proposition of causal inferences (e.g. if this, then that). A helpful starting point was our attempt to identify what kind of experiment it actually was, according to those involved. From there, we identified how descriptions of the experiment resonated with descriptions of the market (the object of experimentation). Lastly, we traced how these descriptions were reproduced in (a) the market scans, (b) decision-making in the Dutch House of Representatives and (c) in the lessons drawn from the experiment.

Empirical reconstruction

In the following sections, we reconstruct how the experiment in Dutch dental care unfolded. We follow a chronological structure, but identify different logics along the way. In the first three subsections, we focus on the object of experimentation: price liberalization in dental care. Here we identify two market logics. In two subsequent subsections, we focus on the experimental logics reproduced during the experiment’s unfolding. Thereafter, we pay attention to how evaluation and political life were implicated and informed the lessons learned.

Prelude to the experiment

The roots of this experiment trace back three decades. A key point in time is the release of the Dekker report in 1987 (Helderman et al., 2005). The Dekker committee advised the Dutch government to introduce a system of regulated competition in the healthcare sector, in which healthcare insurers would negotiate with healthcare professionals on price, quality and volume. Although several similar recommendations and initiatives followed, it took until 2006 to institutionalize market mechanisms through the ‘Law for Ordering Markets in Healthcare’ (Wet Marktordening Gezondheidszorg [WMG], 2006). The goal was to provide rules and guidelines in the:

Development, ordering and surveillance of healthcare markets to stimulate an effective and efficient healthcare system and contain the development of costs. (WMG, 2006)

Dental care was one of several subsectors to explore the possibilities of price liberalization. Professional organizations in this sector were important proponents of this. For years, they had been searching for ways to gain the freedom to implement innovative, sometimes more expensive but qualitatively better, techniques and services. According to them, the sector had a growing variety of products and services. Even so, it was still financed through an average-based system. For each of 400 identified standardized diagnosis-treatment combinations, the state calculated a standard maximum remuneration. This had huge implications:

I would like to use an operation microscope in my practice. It’s a beautiful instrument, but expensive. So, we don’t use it. That’s a pity because it would improve quality and save time. (Representative of a professional organisation, personal communication, 5 March 2015)

Because of the hitherto state-controlled financing structure, dentists claimed they were unable to use new techniques. After all, how could they use occasionally more expensive technology if the state forced them to charge fixed prices? Even when private patients were willing to pay more, dentists were unable to charge extra costs. Due to the financing structure, the sector had ‘slipped into a one-size-fits-all rationality’ (Representative of a professional organization, personal communication, 5 March 2015). Price liberalization, these professional organizations felt, would provide a way out this conundrum.

For the professional organizations, the WMG law introduced in 2006 offered new prospects. It coined the possibility of price liberalization (NZa, 2006), concurring with the development of healthcare markets (WMG, 2006).

Recommendation for a market experiment in dental care

A semi-governmental inspectorate, the NZa, was established in 2006 to coordinate the healthcare market development process, describing their role as ‘the creator and guardian of healthcare markets’ (NZa employee, personal communication, 24 March 2015).

The NZa quickly published a programme theory, in which they defined the market as an ‘instrument’ capable of doing something (NZa, 2006: 7). It could improve quality, stimulate diversity, solve regional scarcities and lower costs. But the market also needed to be contained. After all, its workings were associated not only with desirable effects such as lower costs, but also with undesirable effects such as supplier-induced demands (NZa, 2006). The NZa therefore set out to ‘proactively establish conditions for the market to work’ (NZa, 2006: 1). These conditions included ‘rules of the game’ (NZa, 2006: 8) and the presence of:

transparency on a product’s content, quality and price;

accessibility & capacity/distribution (no scarcity or entrance barrier);

equal bargaining power between market players.

The first year after the WMG law was introduced, much emphasis was placed on identifying these conditions in various healthcare subsectors, including dental care. The NZa advised the Health Minister not to liberate pricing in dental care due to (a) an identified scarcity of dentists, (b) the existence of an entrance barrier and (c) a lack of transparency in the price, content and quality of the services provided (NZa, 2006). In other words, the dental market did not meet the necessary conditions for price liberalization to work well.

However, the NZa’s mission was not only to guard markets, but also to create markets. In this role, the NZa argued that not all conditions had to be present at the start of price liberalization (NZa, 2006: 19):

We don’t want to suggest that if a market isn’t transparent yet, that is automatically a reason to advise the Minister against liberating price setting.

This line of reasoning became more prominent in the reassessment of the dental care sector in 2009. Now, the NZa distinguished between the prerequisites for price liberalization and conditions that could be established after liberalization was initiated. They referred to the latter as ‘in-growth’ conditions (see Online Appendix).

In 2009, the NZa still identified regional scarcities in the provision of dental care and still observed a lack of transparency in quality, content and price (NZa, 2009: 12-13). But reasoning from a creator logic, now the lack of transparency and the presence of scarcity became the focal points for change. In a move that turned their 2006 argument on its head, the NZa argued that ‘some identified issues might be resolved by the market itself once free pricing arrangements are introduced’ (NZa, 2009: 11).

Caught between creator and guardian logics, the NZa concluded that price liberalization in dental care was wrapped in uncertainty. Therefore, the NZa suggested introducing an experiment. As they said (NZa, 2009: 15):

An experiment will give the opportunity to see how prices, quality and accessibility develop in a free market environment.

In comparing the NZa assessments of 2006 and 2009, we observe that the NZa identified itself as a guardian and a creator of healthcare markets. In their guardian role, they emphasized the importance of identifying and installing (pre)conditions to contain healthcare markets. In their creator role, they emphasized that it needed much observation, interpretation and improvisation to learn how to develop healthcare markets. An experiment would give the NZa the opportunity to combine both its roles. It would give space for developing a dental care market (the market needs to be created), but if things got out of hand, price liberalization could always be turned back (the market needs to be guarded). However, as we reveal later on, these two ways of approaching ‘the market’ fitted better with other conceptions of what ‘an experiment’ is and how to evaluate it.

Getting the experiment through the House of Representatives

Following the NZa’s recommendation, Health Minister Schippers sent the following letter to the Dutch House of Representatives:

I have decided to pursue the introduction of an experiment with free pricing arrangements in Dutch dental care in 2012, provided that the necessary prerequisites of transparency are met. (Schippers, 2011a)

Although the NZa had defined quality transparency as an in-growth condition (see Online Appendix), Schippers maintained in her letter that transparency was a prerequisite for the experimental introduction of free prices. Following her proposal, much of the debate in the House of Representatives centred on whether transparency in content, price and quality was present and whether free prices could thus be introduced (Schippers, 2011c).

Schippers acknowledged that Dutch dental care had not yet established proper quality transparency. However, instead of calling off the experiment because of that, she now followed more closely the line of reasoning that the NZa had forwarded in 2009. To the House of Representatives she explained:

It is exactly the prospect of free pricing arrangements that has triggered the substantial development of quality indicators. (Schippers, 2011c)

To further ease the concerns raised, the Minister stressed that although progress was being made in developing quality indicators, she agreed that it was a top priority in making the dental care market function properly. To demonstrate her seriousness on this point, she proposed pursuing the experiment for three years – instead of the five suggested previously by the NZa. This time reduction, she argued, would put extra pressure on the professional organizations to finish establishing quality indicators (Schippers, 2011b). Moreover, the NZa would monitor the experimental dental care market every three months and the results of these quarterly market scans would be evaluated in the House of Representatives.

If accessibility, affordability and quality of care were harmed, the experiment could always be turned back. (Schippers, 2011c)

According to Schippers (2011c), one should take a possible price rise into account when talking about liberating prices. But any price rise should be linked to a rise in quality as well.

Schippers did not convince all members of the House of Representatives. After all, how can you measure a rise in quality if the quality indicators had not been established? (Member, House of Representatives, personal communication, 17 June 2015). As one representative observed:

This happens more often with experiments. Something is introduced experimentally but it almost never gets turned back, even when it has a negative effect. They do this to push things through. (personal communication, 17 June 2015)

When it came to voting for the experiment, Schippers counted on the support of coalition parties in the House of Representatives. Although the opposition introduced a resolution, in the end, the majority supported the experiment; 93 voted for and 57 voted against (Kuiken and Voortman, 2011) and the free pricing experiment was introduced on 1 January 2012 (Schippers, 2011d).

We want to stress here that Minister Schippers skilfully navigated the experiment through the House of Representatives by drawing from both creator and guardian logics. On the one hand, she emphasized development and adaption (e.g. creating quality indicators on the go). On the other hand she emphasized conditions (the same quality indicators) and limitations (e.g. conditions under which prices were and were not allowed to rise). Most opposition representatives – many were left wing – saw the experiment as an attempt by the liberal Minister to liberate the dental care market without too much trouble (they do this to push things through). Suspicious opposition representatives therefore continued emphasizing the need to guard such markets (what to do when negative effects occur). Nevertheless, due to the support of the coalition parties, the introduction of the experiment was approved.

But what kind of experiment was actually introduced? Moreover, how were its effects going to be uncovered and evaluated? As we reveal in the coming sections, different actors involved in the policy experiment answered these questions differently.

The experiment as an opportunity to learn how to have free prices

Following the positive vote in the House of Representatives, in autumn 2011 the NZa started preparing for the experimental introduction of free prices. These preparations entailed more than just the ‘legal’ liberalization of prices. The NZa also worked on setting some of the conditions deemed necessary for the market to function properly.

To secure transparency in the content of services provided, the NZa reworked the old, technically challenging remuneration system. In that state-controlled system, 400 indicators prescribed what a dentist was maximally allowed to charge for specific treatment. In the new system, patients would have to understand the price of each treatment. Therefore, the 400 indicators were condensed into 150 understandable treatment options (NZa employee, personal communication, 24 March 2015). To secure transparency in treatment prices, dentists were obliged to rework the 150 treatment options into their own price lists which, in turn, needed to be presented in their waiting rooms and on their websites by 1 January 2012; the official starting date of price liberalization.

To develop transparency in quality, the Minister asked dental care organizations to continue working on developing quality indicators. Likewise, the chairs of the professional organizations argued that parliament’s decision to introduce an experiment ‘innately stimulate[d] the industry to work on quality indicators, clinical working guidelines and other initiatives’ (personal communication, 5 March 2015).

However, neither the NZa nor the professional organizations thought that the experiment could proceed without further intervention. One NZa employee said:

It takes time for a market to settle. Providers set prices. Patients come, see that the prices are too high and go somewhere else. That is, if the market works properly. But you’ll see that only after a few months. I mean, most patients don’t go to their dentist more than twice a year. (personal communication, 24 March 2015)

Possible interventions in response to unwanted effects could only be decided in the direct context in which these effects were observed. A certain kind of improvisation was needed:

If parties harm the stakes of consumers, for instance by conducting unwanted behaviour that upsets competition, the NZa should be able to intervene. A monitor should provide the basis for judging which instruments should then be deployed. (NZa, 2009: 105)

This learning-by-doing approach is even more sharply defined by a former representative of one of the professional organizations (personal communication, 5 March 2015):

The experiment would be an opportunity to observe how this all goes, how things develop under close scrutiny, of course, and we’d be able to adjust things if unwanted effects occur. In essence, the implication of such an experiment is that things need adapting and developing. Many things could happen. We’d see how transparency in content, price quality could develop, how quality indicators could develop and compare all that to [what’s happening in] other countries.

The professional organizations clearly saw the experiment as a protective space for reflexive learning about free pricing. This fitted well with the idea that markets need to be created and that required lots of preparation, observation and improvisation. In resonance with this reflexive experimental logic, the NZa would conduct quarterly market scans to be discussed in the House of Representatives and provide the basis for new steps to be taken.

The experiment as a test

There was, however another approach to the experiment, namely, as a test to see if the market, once liberated, could be contained and work properly. This approach draws heavily on the pre-conditionality of transparency in price, content and quality, initially stressed by the NZa and Health Minister. Opposition representatives, as well as patient and consumer associations took this line of reasoning further. These organizations maintained that the experiment could only be initiated when the preconditions had been met. They raised the question of how customers could critically bargain on price and quality if they had insufficient information about quality. Looking back, a representative of ‘Patiëntenfederatie Nederland’ (Dutch Patient Federation) recalls (personal communication, 31 March 2015):

We were aware of the upcoming experiment and were involved in some of its developments. But we thought that it was introduced too soon.

Reasoning from a tester’s perspective, the Dutch Patient Federation questioned the lack of quality transparency in the experimental set-up. They formulated a hypothesis that this lack would inevitably lead to a rise in prices:

According to the Dutch Patient Federation, it isn’t true that prices will fall because of market competition. Prices will only rise because it’s too hard to compare dentists and the quality of their services. (Powned, 2011)

The Dutch Patient Federation and opposition representatives felt that the experiment was not yet ready to be initiated and nor was the dental care sector ready to have free prices (personal communication, 31 March 2015).

To sum up, two conceptualizations of the experiment held sway. In the first, the experiment was interpreted as a time-space for learning how to have free prices in dental care. In this formative approach, ex durante monitoring and improvisation were key. In the second, the experiment was a test to see if free pricing in the dental care sector could be contained. In this summative approach, the installation (or lack) of preconditions, the formulation of a (negative) hypothesis and the ex post revealing of (undesirable) effects stood central. As we discuss in the next subsection, these approaches produced different insights into the experimental dental care market.

Monitoring the experiment: Contested observations

In January 2012, dental care prices were liberated. At that time, the hypothesis that prices were going to rise – as put forward by the Dutch Patient Federation in the prelude to the experiment – had received much media attention (NOS, 2011; Powned, 2011). Consequently, various media investigated developments as soon as pricing was liberated. They compared price lists and reported huge differences between dentists (NOS, 2012a). These media reports put pressure on Minister Schippers and she asked the NZa to conduct two quick scans in January and February 2012 to ‘officially’ monitor initial price developments (NZa, 2012a).

The NZa analysed 183 price lists published on dentists’ websites and compared them to what treatment would have cost in the old system. This was difficult because most of the 400 price indicators of the old system had been conflated into 150 new indictors (in working towards content transparency). In the process, some old indicators were lost, while others were merged, or newly introduced. Consequently, the 150 new indicators could hardly be considered the sum of the old (NZa, 2012a). Nevertheless, the NZa focused on the 36 most comparable products and concluded that dental care had become more expensive by 4 per cent (NZa, 2012a: 13). The NZa pointed to the space for errors in comparing 2011 to 2012 based on two different systems and a limited range of products (NZa, 2012a) and advised waiting for the first quarterly market scan, which would also consider developments in transparency, consumer satisfaction and stakeholders’ roles (NZa, 2012b).

In the first quarterly market scan (NZa, 2012b), the NZa reported positively on the development of quality indicators; these were almost ready for implementation. They also reported positively on transparency developments in price and content. Most clinics had a website and presented the price list in their waiting rooms. However, the NZa also concluded that prices had gone up by 9.6 per cent. Again, the NZa stressed the space for errors due to the conflation of price indicators. In the accompanying policy letter, the NZa therefore emphasized that its findings needed to be interpreted with caution (NZa, 2012b).

The NZa’s market scans became a subject of debate. The professional organizations contested the price rise. The NZa was ‘comparing apples to oranges’ due to the conflation of the indicator system (chair of a professional organization, personal communication, 5 March 2015). The professional organizations did notice that:

For the first time, patients were informed of price and content in waiting rooms and on websites. The experiment furthermore created lots of dynamics. Dentists started giving five-year warranties, something that could never have happened in the regulated system. (Chair of a professional organization, personal communication, 5 March 2015)

The NZa, professional organizations and media were not the only ones monitoring the effects of the experiment. It was also under the scrutiny of other actors. On 1 June 2012, the Dutch Consumer Organisation released a report of a test they had conducted at various dental clinics on the possibility for consumers to undergo just one treatment – placing a crown. Calling on 500 dentists, they discovered that 75 per cent of these dentists were unwilling to accept a new patient for only one treatment. In 44 per cent of the cases, providers were willing, but only if the newcomer became a regular patient (Dutch Consumer Organisation, 2012a). Drawing from an economic theory that prices are controlled by the mechanisms of supply and demand, they concluded that ‘the market can only be successful when enough providers want to compete and when consumers are looking for better or cheaper care’ (Dutch Consumer Organisation, 2012a). Because the first were unwilling and the latter were unable to do so, they concluded that the market experiment failed (Dutch Consumer Organisation, 2012b).

The professional organizations responded by stressing that shopping around for the cheapest provider would highly endanger the quality of care in dentistry.

A mouth has 28 teeth and all of them are related to one another. If you want to make a good long-term plan for them, you need one director. (Chair of a professional organization, personal communication, 29 April 2015)

We stress that, even though the NZa was formally assigned to monitor the experiment, monitoring and particularly also the valuing of the revealed effects was done by multiple actors, most notably professional organizations, consumer organizations and the media. Notwithstanding this plurality of observers, we identified two ways in which the preliminary effects of the experiment were revealed and valued. The first placed great emphasis on the rise in prices and lack of opportunity for consumers to swap dentists in search of better or cheaper care. Actors following this line concluded that introducing free pricing had failed. The second placed the emphasis on the positive effects of price liberalization, such as transparency in price and content, as well as the introduction of five-year warranties on services provided. Actors following this line concluded that free prices triggered several interesting developments worthy of further exploration. What to do with such contested observations, interpretations and judgements?

The formal periodic evaluation of the experiment in the House of Representatives

In the spring of 2012, the Dutch political landscape changed dramatically. The three coalition parties – Liberal Party (VVD), Christian Democrats (CDA) and Dutch Freedom Party (PVV) – failed to agree on an austerity programme to meet the 2013 maximum government deficit of 3 per cent as ruled by the European Union. The coalition fell apart on 23 April 2012 and Schippers became the outgoing Health Minister (NOS, 2012b). This had consequences for the support of the experiment in the House of Representatives, which became evident in the prelude to the first formal periodic evaluation of the experiment. A PVV representative said:

It’s a very simple sum: not enough suppliers and a great demand, so prices go up. In times of crisis that shouldn’t be the case. (Agema, 2012)

The (former) opposition representatives realized that almost a year after they had introduced their previous resolution, a new vote in the House of Representatives could stop the experiment. One representative reflects:

Because the coalition lost the PVV, they no longer had a majority supporting the experiment. That gave us the chance to kill it. (personal communication, 17 June 2015)

Drawing the same conclusion as the Dutch Consumer Organisation – that the market had failed because of quality transparency and the rise in prices – on 5 July 2012, opposition representatives framed a resolution against the experiment:

The experiment with free pricing in dental care has failed. Prices have risen tremendously for patients, there is no freedom of choice and still no transparency in quality. (Kuiken, 2012)

In ‘capital P politics’, specifically in doing opposition work, using the word failure was very successful. The word was already embedded in the hypothesis formulated by the Dutch Patient Federation in October 2011, made explicit by the Dutch Consumer Organisation in June 2012 and repeated by opposition representatives in July 2012. Failure – and the logic of containment and guardianship that it stemmed from – resonated strongly in the House of Representatives. It was used for one purpose, to find support for cancelling the outgoing Minister’s introduction of free pricing arrangements in dental care.

Although Schippers questioned the premature establishment of failure, emphasized learning and called for patience and careful decision-making (Schippers, 2012), the resolution was supported by a close but clear majority: 78 against 72 votes (Kuiken, 2012). As a result, Schippers was politically forced to cancel the experiment after only six months, 2½ years before its planned ending. An opposition representative (personal communication, 17 June 2015) reflected:

It would’ve been neater if we’d given the experiment more time to unfold. But that’s not how things always go in politics. We weren’t there for good governance. Rather, we were there for politics.

Although this citation underlines the impact that capital P political life can have on the evaluation of experimentation processes, we want to stress that this political act during the first formal periodic evaluation of the experiment in the House of Representatives had an effect only because it drew on specific conceptualizations of the experiment and market.

How and what did we learn from this experiment?

We described different logics about markets and the experiments, through which the effects of the experiment were revealed and valued in different ways. We also showed how these logics, together with capital P politics, informed the evaluation process. Notwithstanding the politics and contestations, a lesson was learned from the experiment that has since become ‘taken for granted’. Below, we recapitulate how that lesson emerged.

In the course of the experiment, two lines of thinking reinforced one another and were reproduced and stabilized concurrently: (I) the market was approached as a system to be guarded and (II) the experiment was a test to reveal whether or not the market could be contained. These lines fit well with the first approach towards policy experimentation and evaluation described in our theoretical section. Here, the experiment is considered a time-space in which a predefined policy intervention is tested under controlled circumstances and in which the role of evaluation is geared towards ex post measuring of impact and valuing such measured impact by relating it to predetermined programme objectives (Greenberg and Morris, 2005; Ludwig et al., 2011; Wolpin, 2007).

In the case of the dental care experiment, actors following these tester and guardian logics (a) stressed the lack of preconditions, (b) put forward a sceptical hypothesis that price rises were inevitable, (c) placed much emphasis on measuring price developments in the initial stage of the experiment and (d) deemed such rising prices undesirable. Together, these guardian (I) and tester (II) logics prompted ad hoc measurement of pricing at the start of the experiment and framed how its results should be interpreted ex post; as proof that the dental care market could not be contained (see Muniesa and Callon, 2007).

The reproduction of the tester and guardian logics were advanced by the dramatic change in the political climate during the experiment. Opposition representatives who had interpreted the liberal Minister’s introduction of experimental free pricing as ‘just another way to push things through’, now saw an opportunity to halt that process. Drawing from guardian and tester logics, they stressed measured price rises, connected them to absent preconditions, and called for capital P political intervention. On the way, they managed to convince a majority in the House of Representatives to cancel the experiment.

Meanwhile the lack of transparency in quality – as a control mechanism – was put forward as the main reason for the rise in prices and the failing dental care market. Consequently, quality transparency, treated in 2009 as an in-growth condition by the NZa, became a vital precondition for market liberalization. The failed experimental dental care market became a critical example of that lesson learned (Varkevisser, 2012). Three years later, Minister Schippers pronounced 2015 as ‘the year of transparency’ and developing quality transparency turned into a key objective for the healthcare sector (Schippers, 2015).

As the importance of quality transparency grew from the main lesson learned, the idea that markets need to be created and that experiments should be considered as time-spaces to learn how to do this – in line with the second approach to policy experimentation and evaluation discussed in our theoretical section (see Arkesteijn et al., 2015; Martin and Sanderson, 1999; Regeer et al., 2009) – became overshadowed. The positive effects revealed by the professional organizations – for example, that for the first time patients were being informed about the price and content of dental treatments and that new products and services were being released on the dental care market – were ignored (Schippers, 2012). In addition, the ideas that observed effects were part of an ex durante evaluation process and that undesirable effects could be corrected, were also ignored. The professional organizations therefore concluded that their experiment had been nothing more than a political ball game and their trust in policy development and politics was severely damaged (Representative of a professional organization, personal communication, 5 March 2015).

Figure 1 captures the politics of policy learning and how the two logics concerned created the conditions for one another to be reproduced and stabilized. The dotted line represents the schism between the two lines of thinking about experiments and markets that developed during the progress of the experiment.

Figure 1

The politics of policy learning.

Discussion

In this paper, we fleshed out various market and experimental logics entrenched in a single policy experiment. Based on these insights, we would like to close with an epistemological and a methodological point for further discussion about the role of evaluation in policy experimentation. We discuss them in turn.

Reconstructing the experiment in Dutch dental care enabled us to reveal how the different market and experimental logics were, in concurrence with evaluation and capital P politics, implicated in (a) each other’s reproduction and (b) which insights were taken as starting points for further action. In the process, some concurrent logics informed the lessons learned, while others were forgotten. Other authors have also observed various ways in which the effects of an experiment are valued and have pointed at the relevance of adopting constructivist approaches in the organization and evaluation of policy experiments (Arkesteijn et al., 2015; Martin and Sanderson, 1999). In policy experimentation and its evaluation, multiplicity has become an accepted concept (Martin and Sanderson, 1999; Sabel and Zeitlin, 2012). However, even in these constructivist approaches, the experiment itself is often still considered a singular, productive entity; a time-space that can (a) bring stakeholders with different views together and (b) reveal effects to which different meanings can be ascribed (Arkesteijn et al., 2015; De Wildt-Liesveld et al., 2013; Regeer et al., 2009, 2016; Sabel and Zeitlin, 2012).

Reconstructing the policy experiment in dental care, we actually observed two experiments unfolding at the same time, in two distinct epistemological spheres. In the first, an experiment was used to test if the dental care market could be contained. In the second, an experiment was used to create a protective time-space where stakeholders could reflexively learn how to have free prices. We therefore underline that policy experiments can be just as multiple as the worlds they are supposed to bring together; the latter in line with the experimentalist governance and reflexive evaluation literature (Martin and Sanderson, 1999; Sabel and Zeitlin, 2012). Instead of situated between different worlds and accommodating multiple logics, policy experiments can become fragmented and contested; reproducing and being reproduced by the multiple logics and practices they are supposed to accommodate (Fuchs, 2001; Luhmann, 1996; Van Assche et al., 2014).

For a while in the dental care experiment, two approaches were reproduced simultaneously, but not concurrently. In one approach (market creator), the experiment was a formative time-space for reflexive learning on how to introduce and stabilize certain propositions. In the other approach (market guardian), the experiment was a summative test to see if the introduced proposition functioned properly. Yet after the results of the first round of monitoring were evaluated in and beyond the House of Representatives, one approach could no longer exist beside the other. After all, in the summative testing logic, the monitors had proven that the dental care market could not be contained and the experiment was finished. Those who saw the quarterly monitor as a first formative stage in a continuous trajectory of ex durante evaluation and reflection were unable to counter this ex post and summative judgement.

As we demonstrated, both approaches informed policy learning in distinct ways (see Figure 1). We cannot predict what would have happened if the experiment had continued for the planned three years. Maybe the diversification of products in dental care would have continued, instead of lapsing back to the previous situation. Maybe the development of quality indicators would have been finished. Both developments might have triggered the further stabilization of free prices, feared by left-wing opposition representatives and theorized about in the performativity literature (Mackenzie et al., 2007). Most likely, the professional organizations in dental care would have been less alienated from policy development and politics than is now the case.

Taking into account the different directions of learning that different policy experimental approaches produce, evaluating policy experiments should therefore include reflection on the politics of policy learning. This means being sensitive to the logics and practices through which the experiment is conceptualized, operationalized, implemented, observed and evaluated by the various actors involved; and the exclusionary starting points for further action that these different logics and practices generate. We are therefore sympathetic to scholars who call for a more engaged role of evaluation in policy experiments (Arkesteijn et al., 2015; Martin and Sanderson, 1999; Regeer et al., 2009; Van Mierlo et al., 2010). However, we warn against (a) a priori readings of policy experiments as a protective space for reflexive learning, and point out (b) the various directions of learning that summative and formative approaches may nurture (see Figure 1).

In addition to the principles and insights developed in the reflexive evaluation literature (Van Mierlo et al., 2010; see Arkesteijn et al., 2015; Regeer et al., 2009, 2016; cf. section two of this paper), we propose that evaluators should address four basic questions, in dialogue with the involved actors, including policy makers and politicians, at repetitive stages during the unfolding experiment. These are: (1) what kind of experiment is it? (2) what is the object of experimentation? (3) what role does evaluation have in the experimentation process? and (4) what are we learning and how does that relate to the first three questions? While answering these questions, particular attention should be paid to the logics with which the actors observe, describe and value the experiment, its objective and (preliminary) results. In order to do so, evaluators should be sensitive to the distinctions, observational procedures, conceptual mappings and causal inferences furthered by the participating actors, including themselves.

It is through such reflections on the politics of policy learning – not only on the object of experimentation, but also the logics of experimentation and how these imply one another – that insight can be gained into the how and why of what we learn from policy experiments. We are convinced that answering these questions not only provides insights into paths of learning, but also functions as an important strategy to align the ways in which an experiment is observed and valued by the various actors involved (see Geels and Schot, 2007; Van Mierlo et al., 2010). It provides possibilities for stakeholders to reflect on their own and be aware of others’ approaches to the experiment. Only through such reflection can we better understand, not only what we learn from a policy experiment, but also give words to and come to terms with, what is lost in the process (Butler, 2010; see Arkesteijn et al., 2015; Regeer et al., 2009).