Measuring the impact of consultative citizen participation: reviewing the congruency approaches for assessing the uptake of citizen ideas.

As academic and political interest in citizen participation and democratic innovations is growing, the question of their impact on public policy remains essential to assess their genuine contribution to the normative project of democratization. Impact assessments of consultative participatory mechanisms are commonly conducted with a congruency approach-a desk-based research method which assesses impact based on the textual correspondence between a citizen-created idea and public policy documents. This method, however, lacks reliability and uniformity, and this paper therefore seeks to standardize its application and ways to improve the accuracy of its results by proposing two methodological add-ons. First, a sequential impact matrix that considers the preferences of decision-makers before a consultative participatory mechanism to see the extent to which decision-makers take up citizen ideas that align with or diverge from their own agenda. Second, a mixed method that combines a congruency approach with interviews of actors involved in the follow-up of the participatory process to balance their experiences with the congruency approach's main findings. The variants of the congruency approach are then applied to a deliberative minipublic-the citizen panel 'Brussels-Make Your Mobility'. This analysis shows how these methodological strategies alter the impact assessment's results, and its findings suggest that the use of a sequential impact matrix with a mixed method not only produces an accurate and reliable measurement but also generates valuable insights into the diffuse ways in which minipublics can exert substantial influences on the institutional structures and the political decision-making. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11077-022-09450-w.

Introduction

There has been a resurgence of academic and political interest in citizen participation with the advent of democratic innovations (Elstub & Escobar, 2019; Smith, 2009). Public authorities have increasingly tended to involve ordinary citizens in participatory processes as a way to tackle the legitimacy crisis facing political systems, reduce the gap between elites and citizens, and democratize governance (Warren, 2009). With this proliferation has come the concomitant question of the impact of citizen participation on public policy, which is of paramount importance for the normative project of democratization. To what extent do they exert a genuine influence, or are they instrumentalized by decision-makers (Smith, 2009, pp. 22–24)? There is a broad consensus that citizen participation should have a “genuine impact on policy and be seen to do so” (Rowe & Frewer, 2000, p. 14, see also Abelson et al., 2003, p. 244; Beierle, 2010). Without this, it comprises only the activation of a “thin citizenship” (Bherer et al., 2017, pp. 7–8; Walker, 2014, p. 205), losing its credibility and attractiveness in the eyes of the citizenry (Fung & Wright, 2001; Gundelach et al., 2017; Jacquet, 2017; Pateman, 2012; Wampler, 2008) and generating participatory frustrations that may lead to political apathy and disengagement (Fernández-Martínez et al., 2020; Van Stekelenburg & Klandermans, 2013).

Scholars have dealt with this research agenda via case studies (Alarcón et al., 2017; Barrett et al., 2012; Bua, 2017; Michels & Binnema, 2018; Smith, 2009), larger comparisons (Beierle, 2010; Font et al., 2018; Gastil et al., 2017; Pogrebinschi, 2013) or literature reviews (Abelson & Gauvin, 2006; Geißel & Heß, 2018; Jacquet & van der Does, 2020b; Jager et al., 2020; Michels, 2011; Rowe & Frewer, 2004). However, there has been surprisingly little attention devoted to the actual assessment of impact (Font & Smith, 2019; Jacquet & van der Does, 2020b; Mazeaud & Boas, 2012; Richardson et al., 2019)1. Most works rely on a congruency approach—i.e., a desk-based research method which assesses impact based on the textual correspondence between a citizen-created idea and public policy documents—but they tend to operationalize the impact disparately, looking at different public policy outputs and (in a limited number of cases) applying distinct triangulation strategies. Reviewing the literature conveys the impression that each study developed its methodological framework in isolation from other works. It is also unclear how these researchers have measured the textual correlation, and that lack of transparency not only prevents the transferability and replicability of their studies, but also undermines their capacity for theory-building (Mutz, 2008). As this research agenda is moving toward larger comparisons (Ryan, 2019), the absence of methodological standardization is becoming problematic.

This paper is an effort to standardize the commonly used congruency approach with a case study to see how different calibrations of the congruency approach can produce diverging and more reliable findings. It examines how the operationalization of impact, the moment in policymaking, and the triangulation strategy variously alter the empirical results. I will also propose two methodological add-ons for a more context-sensitive application of the congruency approach, which—as I will argue—can increase the reliability of the findings. The first is the sequential impact matrix (SIM), an assessment framework which considers the preferences of decision-makers before a participatory process and the type of public policy that is taken up afterward. The second suggestion is a mixed method which triangulates the congruency approach with interviews to obtain the contextualized experience of actors involved in the follow-up of the consultative participatory process. The study applies this approach to a typical case of consultative citizen participation: the citizens’ panel “Make Your Brussels Mobility” which took place in the Brussels Capital Region of Belgium in 2017. The results indicate the importance of adopting a more fine-grained congruency approach, distinguishing different sorts of uptakes and carefully selecting—and justifying—the policymaking stage at which the assessment takes place. More importantly, they also demonstrate the necessity of adopting a mixed method to corroborate the general trends identified with the policy document analysis. A mixed method can also reveal new insights about the more diffuse political contributions, such as exerting a considerable influence on the political decision-making or leading to the institutionalization of minipublics in the Brussels Regional Parliament.

The uptake of citizen ideas from consultative participation mechanisms

Consultative participatory mechanisms (CPM) can be understood as collective and formal institutions which involve ordinary citizens and aim to influence political outcomes (Brady, 1999, p. 737, in Ekman & Amnå, 2012, p. 286). This definition includes democratic innovations (Elstub & Escobar, 2019; Smith, 2009) but does not exclude other consultative participatory processes that do not seek to “innovate”.2 CPM can generate diverse important effects on politics and policymaking (see for instance Jacquet & van der Does, 2020a; Michels & Binnema, 2018), but their primary formal function is to provide decision-makers with citizens’ recommendations, proposals and views (Geissel, 2012; Warren, 2009)—which we will refer to under the umbrella term of citizen ideas. CPM’s influence on policymaking thus depends on the discretion of decision-makers (Bua, 2019), meaning that the impact is indirect: citizens’ ideas are transmitted to decision-makers, who subsequently decide whether or not to integrate them into public policy and laws3.

Such indirect impact clouds the impact assessment, as it becomes extremely complex to ascertain whether the impact actually stems from the original influence of a CPM and not from other sources of influence (e.g., lobbies, public debate, or inter-party negotiations). A few surveys have discussed these methodological implications and challenges. Scholars have criticized existing studies’ narrow understanding of the impact and public policy (Abelson & Gauvin, 2006; Blondiaux & Fourniau, 2011; Chess, 2000; Mazeaud & Boas, 2012; Rowe & Frewer, 2004), and the focus on exceptional cases (Spada & Ryan, 2017), while others have specifically reflected on the methodological exercise of impact assessment (Font et al., 2016; Font & Smith, 2019; Jacquet & van der Does, 2020a; Richardson et al., 2019). However, these studies tend to reflect on their own methodology or to briefly discuss various approaches, but no research has yet collated them. This study compares different calibrations of the common congruency approach and applies them to a case of CPM, so as to standardize the methodologies and increase their reliability for future comparisons. Relying on a case study may here seem contradictive, but all comparative studies actually rely on published and grey literature consisting of case studies (Geißel & Heß, 2018; Jager et al., 2020; Michels, 2011; Rowe & Frewer, 2004) or on a large amount of self-conducted case studies (Font et al., 2018).

The congruency approach

The most common methodology used to assess the uptake of citizen ideas is the congruency approach (Jacquet & van der Does, 2020a). In this approach, scholars examine the extent to which citizens ideas textually correspond to public policy documents and laws, and, if both coincide either literally or in intent, they suggest that the CPM has had an impact (Geißel & Heß, 2018; Pogrebinschi, 2013; Pogrebinschi & Ryan, 2018). The congruency approach is popular because it requires less time and human resources than case studies (see for instance Bua, 2017; Johnson, 2015), thereby making larger comparisons possible. This practical advantage nonetheless comes at the cost of reliability: a simple textual correlation serves as a proxy to indicate an impact. The congruency approach therefore investigates a causal relation—i.e., a citizen idea influencing public policy—with an indicator of correlation, without inferring absolute certainty that this is the result of the CPM’s genuine influence and not from other sources of influence (Richardson et al., 2019, p. 271). This paper aims to (1) standardize and (2) reinforce the reliability of the congruency approach. On the former, the paper compares how different calibrations affect the production findings (see sections "The operationalization of public policy" to "The sequential impact matrix") and, for the latter, proposes a mixed method to improve the congruency approach’s context-sensitivity (section "Employing a mixed method").

The operationalization of public policy

While all studies must rely on some degree of conceptual untangling, several scholars have criticized the literature on impact assessment for its simplified conceptualization of public policy and policymaking (Boswell, 2016; Richardson et al., 2019). Extant research tends to form a monolithic conception of public policy and does not make any distinction regarding its content (Mazeaud & Boas, 2012), i.e., what the government chooses to do (Howlett et al., 2018, pp. 150–151). Citizen proposals and political uptakes can nonetheless entail different things, either presenting an abstract principle or implementing specific and concrete policy ideas. When relying on the congruency approach, a clearer and more transparent operationalization is needed to assess the exact content of a citizen idea and how it is taken up in public policy. I propose dividing public policy into different components and different levels of abstraction (see online supplementary file, Section 1. A typology of public policy). We can distinguish three generic forms: policy goals; policy means (Jenkins, 1978; Lasswell, 1958; Walsh, 1994); and the framing of the two former elements (Hoppe, 2011; Michels & Binnema, 2018; Perri, 2018, pp. 276–279). These three policy components also differ depending on the level of abstraction: they can entail broad policy elements at the abstract level; general components at the policy level; or refer to the on-the-ground setting of a public policy (Hall, 1993; Howlett, 2019; Howlett & Cashore, 2009). A differentiated operationalization of public policy allows a more accurate assessment of the textual correspondence of a citizen idea and public policy, as one can explicitly point out when an idea is partially present—i.e., when an uptake remains abstract or shows limited ambition. In the analysis (Sections “The binary approach” and “The plural approach”), I compare a binary approach—by which a citizen proposal either roughly corresponds to public policy or not (e.g., Pogrebinschi & Ryan, 2018)—with a plural approach, which distinguishes full, partial, and absent uptakes (e.g., Font et al., 2018) based on the correspondence of both the components and levels of abstraction between a citizen idea and public policy.

The policymaking moment

Earlier studies tended not to clarify at what point in the policymaking process the assessment was conducted, but analyzes now explicitly mention the phase. They commonly examine a single policymaking moment in order to measure the impact (Jacquet & van der Does, 2020a; Pogrebinschi, 2013), while a few others look at both the agenda-setting and the decision-making stages (Michels & Binnema, 2018; Pogrebinschi & Ryan, 2018) or the implementation stage (Font et al., 2018). Relying on Howlett et al.’s (2009, pp. 12–13) model of a policy cycle, this analysis examines official documents after two key moments: policy formulation and decision-making. As each of these two stages involves different tasks and policymakers—a policy subsystem formulating policies for the former and the government decision-makers validating policies for the latter (Howlett et al., 2009; Pogrebinschi & Ryan, 2018)—the uptake of citizen ideas is likely to differ. I do not consider the impact on the implementation phase because it is a long process which considerably extends the temporal scope of any analysis. It can furthermore escape the control of decision-makers (Boswell, 2016), making it less relevant to the question of uptake. If practically feasible, it does, however, remain a crucial stage in investigating the final “fate” of citizen ideas (Font et al., 2018).

The sequential impact matrix

Most studies do not consider the initial policy agenda of decision-makers, implying that a participatory process takes place in a political vacuum (Vrydagh & Caluwaerts, 2020). In so doing, they overlook the fact that decision-makers have policy preferences before a CPM and that these can affect the uptake of citizen ideas. The sequential approach integrates these initial preferences to investigate the extent to which decision-makers cherry-pick citizen ideas—that is, taking up those that coincide with their own agenda and rejecting or partially integrating those that deviate from it, hinting at the CPMs instrumentalization (Font et al., 2018; Setälä, 2017; Smith, 2009). My analysis relies on Vrydagh and Caluwaert’s (2020, pp. 6–7) sequential approach. They compare citizen proposals with the expressed initial policy preferences of decision-makers to distinguish five types of influences. On the one hand, when a citizen idea aligns with the decision-makers’ agenda, a CPM can either exert a continuous influence or an enriching influence: (1) a continuous influence completely coincides, whereas (2) an enriching one coincides but also supplements it with small changes. On the other hand, when a citizen idea diverges from the agenda, it can take three forms: (3) it has an innovating influence when it recommends something new; (4) a shifting influence when it radically changes the initial policy course; or (5) an inhibiting influence when it convinces decision-makers to abandon the initial policy scheme because the democratic innovation disregarded it.

An important constraint of the sequential approach is the availability of decision- and policymakers’ initial preferences. It may also be difficult to identify the actors responsible for the uptake and, when these actors are not elected officials, it may not be possible to access their public agenda. Fortunately, most CPM are commissioned by elected decision-makers (Vrydagh et al., 2020, pp. 58–59; Warren, 2009) whose initial preferences can be accessed through the electoral programs of their political party, their policy agenda, or interviews in the media. For the few cases without these data, one could always resort to investigating the identity of these actors and interviewing them before the CPM. For the analysis, I combined the sequential approach with a plural approach to uptake, building a sequential impact matrix (SIM). This methodological add-on helps to standardize the congruency approach and reinforce the reliability of its results. Thanks to its visual overview (see online supplementary file, Sections 2 and 3), the SIM transparently displays the political fate of each citizen idea, showing whether it was already being considered by decision-makers before the CPM, what it entails, and how it is taken up in public policy.

Employing a mixed method

Most studies based on congruency do not triangulate their methodology (for exceptions, see Bua, 2017; Font et al., 2018). Impact assessment is often based solely on document analysis, and researchers tend not to compare their results with experience of actors involved in the political follow-up.4 This lack of triangulation undermines the reliability and quality of their results, because we cannot know whether or not the corresponding public policy is the result of a CPM’s genuine influence (Denzin, 2009; Richardson et al., 2019). One way to address this deficiency is to triangulate methodologies: to decrease the deficiencies and biases of the congruency approach (i.e., context-insensitivity) with the strength of another (i.e., interviews of different actors who have a contextualized experience of the follow-up), thereby constituting a mixed method (Flick, 2004). Either method on its own is not decisive, as both methodologies are mutually supportive. The primary function of this approach is the reciprocal corroboration of each method’s findings, thereby generating more reliable results than those produced by each method individually. In taking this approach, one should first conduct a policy document analysis before interviewing actors involved with the citizen ideas’ uptake (Natow, 2020). Table 1 provides a summary overview of all the methodologies applied to the case study in this paper.

Table 1

An overview of the variants of the congruency approach

3. Congruency approach Impact assessment based on the textual congruency between a citizen idea and public policy
3.1 Operationalization of public policy	Binary impact A citizen idea has an impact when it textually corresponds to public policy
	Plural impact A citizen idea has an impact when it textually fully corresponds to public policy, and a limited one when it partially corresponds
3.2 Focus on policymaking	Policy Formulation Focus on the stage where policymakers formulate the first version of the public policy
	Decision-making Focus on the stage where decision-makers adopt the final legal text
3.3. Sequential Impact Matrix	It examines the expressed initial preferences of decision-makers in addition to the plural approach to impact
4. Interviews To interview actors closely involved to the political follow-up of the CPM to triangulate the general findings of the congruency approach

Case description

This paper will apply these different methodological approaches to one case, the Brussels citizen panel “Make Your Mobility Brussels”. A citizen panel belongs to a specific kind of democratic innovations—namely, deliberative minipublics—whose participants are randomly selected and engage in a structured deliberation while receiving information from various experts (Setälä & Smith, 2018). It is a typical case for studying CPM as it did not have the authority to make binding political decisions and its outcome consisted of a series of policy recommendations that were then transmitted to decision-makers. The design of a participatory process does not matter for the methodologies as long as it is consultative and involves a series of citizens ideas that decision-makers and policymakers can take up.

The citizen panel occurred between October and November 2017 (Vrydagh et al., 2020). The organizers used random stratified selection to recruit 40 ordinary inhabitants of the Brussels-Capital Region. The panel was implemented at the policy formulation stage of the new mobility plan “Good Move”, a 10-year plan defining the regional and municipal mobility policies. The public administration Bruxelles Mobilité and an external consortium (Smartteam) had already set the agenda with a series of transversal sections, after consulting stakeholders and civil society organizations in a series of open and closed meetings, as well as the wider population via an online platform on which citizens could submit ideas. The idea of the citizen panel originated in the Brussels Regional Parliament and its president, Charles Picqué (Parti Socialiste, French-speaking socialist party). Its creation and organization was thus relatively isolated from the actors involved in the elaboration of the mobility plan. The regional parliament did not include the Brussels regional government in the organization, except to inform them about the process. Bruxelles Mobilité contributed the panel by presenting the main policy directions that they envisioned for the mobility plan. The panel had a mandate to answer the question, “By 2030, in which kind of neighborhood do you want to live and how to you want to move?”

Over four days in the Brussels Regional Parliament, participants engaged in deliberation under semi-professional facilitation and heard from various experts and witnesses. They formulated their recommendations in a citizen resolution on November 19, 2017, which was then presented to a parliamentarian commission created especially for the political follow-up. Next, this commission converted the citizen resolution into a parliamentarian one so that it could be legally transmitted to the minister for mobility, Pascal Smet (SP.a, Dutch-speaking socialist party). The conversion was ratified by the whole Parliament on May 25, 2018, and the minister for mobility was invited to discuss the resolution on June 25, 2018. Next, Bruxelles Mobilité and Smartteam incorporated the parliamentarian resolution into the mobility plan. The first version was transmitted to the minister for mobility, who had it ratified by the whole regional government in April 2019. A public survey was then organized to let people react to the first version of the plan, and regional elections were held in June 2019, leading to the formation of a new government in July 2019. The new minister for mobility, Elke Van den Brandt (Groen, Dutch-speaking ecologist party), took over the mobility plan and had it definitively adopted by the regional government in March 2020. Due to this change of government, the sequential approaches cannot be applied on the final version of the plan because the preferences of decision-makers ratifying the plan do not precede the citizen panel and the citizen resolution may have thus influenced the political parties’ programs and the new government’s coalition agreement.

Data collection and methodology

The primary data source for this study is the citizen resolution produced by the citizen panel, which consists of 74 ideas that form the unit of analysis. For the public policy that followed the citizen panel, I examined two official documents: the first version of the mobility plan adopted in April 2019 for the policy formulation stage, and the final version of the plan ratified in March 2020 for the decision-making stage. Regarding the initial agenda of decision-makers, I have relied on the 2014 coalition agreement of the regional government and the 2014 regional election program of the political party SP.a (Dutch-speaking Socialist Party) to which the then minister for mobility, Pascal Smet, belongs5. I used two criteria to detect the congruency of a citizen idea with initial preferences and public policy: a citizen proposal could correspond either literally in the text, or more subjectively in the intent. In the case of the latter, it means that the literal formulation of public policy is not exactly the same but it follows and aligns with the citizen idea’s intention. The differentiation of different components and abstraction levels of the public policy however helps standardize this exercise of qualitative distinction (for examples and more information, see Section “The binary approach” and “The plural approach” and the online supplementary file (called OSF hereafter), Section 2. The Sequential Impact Matrix for the policy formulation stage and Section 3. The overview of the Sequential Impact Matrix for the decision-making stage).

Finally, I interviewed actors closely involved with the organization and political follow-up of the citizen panel. There are a variety of different standards regarding the optimal number of interviews; for instance, Font et al. (2016) interviewed at least two actors in order to triangulate the data sources, while Bua (2017) and Michels & Binnema (2018) conducted 75 and 20 interviews, respectively. With a total of seven interviews, I applied a middle-range standard that comparative studies could realistically meet. When selecting interviewees, I made sure to interview at least one key actor from each of the main institutions involved in the organization or the political and institutional follow-up of the citizen panel. Interviewees knew their general roles in the institution would be mentioned in the analysis, which may have led some of them to overstate the importance of certain institutions and actors. Yet, I followed a data triangulation strategy (Mitchell, 1986), interviewing actors from distinct institutions. The seven interviews were semi-structured and lasted between 30 and 90 min each (see Appendix 1 and 2 for an overview of the interviewees and the interview script). I conducted the interviews in person, online, and on the phone between January and March 2021.6 I had conducted the congruency analysis beforehand, enabling me to compare each method’s findings by contrasting interviewees’ perceptions with concrete examples from the congruency analysis and, conversely, balancing the general trend of the policy document analysis with their experiences. The interview format does not, however, provide means to check the political fate of all 74 citizen ideas. I analyzed the interviews with MAXQDA to code segments that relate to the congruency analysis (impact on the first and final version of the plan, different influences and uptakes). As for the new insights revealed by interviews, I have coded these inductively.

Analysis via the congruency approach

The binary approach

The binary approach captures the impact of the citizen panel in a dichotomous way: either a citizen idea corresponds to something in the mobility plan (indicating an impact) or it does not (meaning a lack of influence). For instance, in the Sects. 2.6 and 3.6 of the OSF, the citizen panel asks decision-makers to assign objectives to the public transport company to make it competitive compared with car travel. The final mobility plan mentions that public transport should be competitive compared with cars, albeit without any concrete policy components. A binary approach judges this case to have an uptake because there is a textual correspondence, while a plural approach to uptake would consider it as partial, because it is merely quoted as a framing element which does not entail concrete policy objectives and the means to specify and implement it. Figures 1 and 2 show the results for the first and final versions of the mobility plan, respectively. At the policy formulation stage, 48 citizen ideas (64.9%) were congruent while 26 (35.1%) did not have any uptake. The impact seems to be larger at the decision-making stage, where 55 proposals (74.3%) were congruent with the final plan. Hence, according to a binary approach, the citizen panel had a substantial impact on the policy formulation and an even greater impact on the decision-making.

Fig. 1

The binary approach at the policy formulation stage

Fig. 2

The binary approach at the decision-making stage

The plural approach

The plural approach distinguishes between different uptakes of citizen ideas by looking at the correspondence of public policy components and their levels of abstraction. An uptake is partial when the public policy distorts the citizen idea or when there is a mismatch between the components and levels of abstraction of a citizen idea on the one hand, and the uptake on the other. The uptake is partial when decision-makers refer to a citizen idea without providing concrete policy measures. For instance, the citizen panel wanted to replace parking spots with two trees. This idea was quoted in the first and final version of the mobility plan (see the OSF Sect. 2.5 and 3.5), but the idea is simply mentioned without objectives or means to realize it. Another example of partial uptake is where the mobility plan roughly and incompletely coincides with a citizen idea, which can be captured in the discrepancy either between public policy components and levels of abstraction, or a disjunction within similar types of public policy. On the former, the citizen panel wanted the police to devote more attention to road safety (see OSF Sect. 2.10) but the mobility plan only mentioned the creation of a bike police unit. Whereas the citizen idea was about increasing police attention to ensure road safety for all means of transport, the mobility plan only focused on the creation of a small bike police unit. We thus observe here a mismatch between an ambitious citizen idea and a policy proposal with limited ambition, indicating a partial uptake. Another example was the citizen panel’s suggestion that the Brussels-Capital Region should encourage other regions and the federal authority to build parking around Brussels (see OSF Sect. 2.6), whereas the mobility plan only recommends the construction of parking without referring to the other authorities’ responsibility for building them. The citizen idea and uptake are both included at the policy level, but differ in content as the uptake removed the role of other authorities. The plural approach indicates (see Figs. 3 and 4) that the citizen panel had less impact compared with the binary approach. At the policy formulation stage, 16 citizen ideas (21.6%) were partially taken up, implying that the full impact only concerns 32 citizen ideas (43.2%). As for the final version of the Good Move plan, we see a larger full uptake with 41 proposals (55.4%) but an identical level of partial uptake, with 16 proposals (21.6%). Partial uptakes thus account for a non-negligible proportion of the total uptake.

Fig. 3

The plural approach at the policy formulation stage

Fig. 4

The plural approach at the decision-making stage

The sequential impact matrices

The SIM considers the expressed initial preferences of decision-makers in order to examine the extent to which uptake at the policy formulation stage only applies to citizen ideas that align with the decision-makers’ agenda. It also relies on a plural approach to impact. Regarding examples of aligning influences (OSF Sect. 2.8), the citizen panel wanted to increase the visibility of city trains, an idea that was already present in the coalition agreement of the government and which thus suggests a continuous influence. As for enriching influence, the citizen resolution states that information regarding the transport options must be accessible to all citizens. The SP.a shared a similar preference, as its manifesto invoked the importance of communication for public transport accessibility. Both ideas are similar but the citizen idea is more ambitious, stating access to information as a first-order policy objective. The mobility plan’s adoption of this idea suggests that this citizen idea enriched the decision-makers’ initial preferences. As for diverging influences, we can find an example in the OSF Sect. 2.8: the citizen panel asked for the creation of a central platform to centralize all information on transport options. This idea was never mentioned in the decision-makers’ initial preferences, suggesting that this idea exerted an innovating influence.

Figure 5 shows that this approach generates an insightful picture of the fate of citizen ideas. It firstly gives an indication of where citizen ideas stand regarding decision-makers’ agendas. We can see that most citizen ideas align, with 31 ideas (41.9%) that are continuous and 13 (17.6%) that are enriching. Diverging citizen ideas are less common and only consists of innovating ones—30 (40.5%)—meaning that no citizen idea radically shifted or conflicted with decision-makers’ initial preferences. This finding is interesting as it indicates that the citizen panel’s ideas were largely in line with the decision-makers’ agenda. When considering the type of uptake per type of influence, we can observe that aligning influences received more full uptakes, with 20 continuous proposals taken up (27%) and five enriching ones (6.7%). Diverging proposals were less integrated into the public policy, as only seven innovating (9.5%) proposals obtained full uptake. These findings suggest that decision-makers cherry-picked to some degree among the citizen ideas, taking up more those that align with their agenda and disregarding more those that diverged from it.

Fig. 5

The SIM at the policy formulation stage

The mixed method

I conducted interviews with seven key actors closely involved in the political follow-up of the citizen panel in order to obtain insights into their experiences and perceptions and to see whether they confirmed, rejected or nuanced the general trends identified via the congruency approach.

A binary approach to impact

All interviewees agreed that the citizen panel had an important impact on the mobility plan, although they ascribed different meanings to what that impact was. The prominent meaning related to the alignment of the citizen resolution with the policy agenda. For policymakers, the citizen panel had a significant impact on the mobility plan because it showed that a diverse group of citizens supported the plan. They confirmed the large congruency between the citizen resolution and the first version of the mobility plan, as a policymaker explains: “There was an impact on the elaboration of the Good Move plan. Actually (…), it greatly helped Good Move because the stars were completely aligned at that moment. What we saw in the citizen resolution, I mean, we could not have written it better (…). It was a godsend for the process [of creating] Good Move, because it showed that the project team (…) came [up] with conclusions close, if not very close (…) to citizens’ expectations.”7

The substantial congruency was particularly significant for policymakers due to the context in which they elaborated the mobility plan. When the plan started back in early 2016, the political debate on mobility was salient and polarized. A series of controversial issues and policies on mobility8 generated considerable debate about the state and future of mobility in the region (Hubert et al., 2020), in which the role of cars was central: one side wanted to decrease the importance of the automobile, while another was strongly opposed to this. The media treatment gave a lot of coverage to car supporters, which seemed to amplify the magnitude of their group in the eyes of the decision- and policymakers. The Bruxelles Mobility administration moreover suffered from widespread distrust, with several critical pieces against them in the media and an audit on the entire administration. One member of the consortium explained: “There was really a ‘Bruxelles Mobilité-bashing’ when we started the Good Move plan, and we had to recreate adhesion, recreate the conviction that Bruxelles Mobilité could launch (…) a master plan.” After consulting various stakeholders, policymakers went for a mobility plan that took a clear position against cars despite the sensitive political context. The fact that the citizen resolution—upon which policymakers had little control—was responsive gave them great confidence to stand by their reform project.

The citizen panel’s support allowed the policymakers to show that the mobility plan was supported by a “silent majority” and was not the result of a siloed reflection among mobility experts and car opponents. Despite the tense political context, policymakers interpreted the high level of congruency as indicative of wider public support for their ongoing policy project: “With the citizen panel and its resolution, we indeed got the idea that ‘yes, we are on the right track’, this is something acceptable, even if there are frictions at the political level (…), we can keep [working] in this direction.” The citizen resolution also became a resource for them to legitimize their reform: “So say [the impact] is really important because it gave us additional arguments, and, really, a support in a way; something upon which we could rely on to say ‘It’s not the experts, it’s not Bruxelles Mobilité, it’s not the people who say “I know everything about mobility”, it’s also folks, ordinary people who took position.’ ” Interviews hence corroborated the general trend of the impact identified with the congruency approach.

A plural approach to impact

With respect to the plural approach, the interviews questioned the underlying assumption of partial and absent uptakes. These two outcomes may look as if decision-makers cherry-picked citizen proposals, adjusting them to fit their own agenda or rejecting those that did not, but the interviews yielded an alternative interpretation. While policymakers admitted the selective integration of citizen ideas into the mobility plan, it seems that the rejection of some proposals was due to the legal and political difficulties of actualizing them, rather than to instrumental purposes. One illustrative example concerns the citizen proposal of implementing a toll around the Brussels-Capital Region, but only for non-Brussels citizens. The uptake was coded as partial because the concrete policy means was translated into abstract ideas in the plan, without mentioning who would be subject to it. The partial uptake is here justified by political and legal reasons. Legally, the proposal goes against the European law, as a cabinet member explains: “But yes, only for the non-Brussels citizens, we knew very well that it would never work. But you have to understand this (…), a citizen panel gives you an indication, something which has a raison d’être [purpose] as a whole which can be established, but yeah, the details behind like ‘non-Brussels Citizens won’t pay’, there are considerations that you must know. Their propositions should not be dismissed because there are things in there that will never be accepted by [the European Union] or that are technically impossible to elaborate. But this distinction is quickly made.”

Politically, two reasons were invoked. First, the toll proposal was discredited by many interviewees because it conflicted with two other citizen ideas, which asked decision-makers to put the general interest above particular interests and to reduce the traffic density. Interviewees highlighted the fact that Brussels motorists still represented an important proportion of car users in the region, and that giving them a free pass for the toll would go against the idea of putting the general interest first and reducing traffic density, as a policymaker described: “We did not exactly respect what [the citizen panel] asked because it is not a toll but an electronic km charge. But this is because everyone should pay and because our data shows that (…) the share of Brussels citizens using their cars is also important, especially for short distances. So, we must act on this. So, yeah, we did not exactly implement what they asked, but in the principles and so on, everything remains intact from [the first to the second version] of the plan.” Second, some citizen ideas were politically more sensitive and could potentially jeopardize the whole mobility plan. Of the toll proposal, a cabinet member explained that such proposals pose a high political risk because they can draw all the political attention and compromise the rest of the plan: “A problem like the [toll] is a problem which often serves to bring in new problems or other topics that were already approved (…). Once there is a major problem that crystallizes a new political problem, we let things already approved come back (…) and suddenly, it takes one or two weeks and we end up with not one major problems, but ten problems on the table.”

The interviews thus confirm the selective uptake of citizen ideas. However, whereas this may intuitively hint at a potential instrumentalization or a lack of genuine influence, the interviews offered different interpretations. The toll example reveals that this citizen idea exerted an influence even if was only partially taken up, as decision-makers modified it so that its objective and principles could be realized while respecting the legal constraints and taking the political context into account. Moreover, a lack of uptake may be due to the citizen idea itself, which could be legally or politically difficult to translate into public policy. While further investigation would be required to establish with certainty whether these political and legal arguments are genuine and not a pretext for avoiding an undesirable policy choice, the interviews at least showed that the absence of textual correlation and partial uptakes do not automatically imply a lack of consideration from policymakers.

A sequential approach to impact

The desk-based research suggested that citizen proposals succeeded in bringing new ideas into policymaking. The interviews did not, however, corroborate this finding, as policymakers could not remember learning new ideas from the citizen resolution. One of them explained: “Honestly, content-wise I have discovered nothing. Regarding (…) innovating ideas, no I haven’t discovered anything, by saying, ‘Well, we haven’t thought of that, that could indeed be interesting.’ ” This is not surprising: the policymakers drafting the plan were mobility experts and therefore likely to already be acquainted with the citizen panel’s ideas. Although the citizen panel’s participants received technical information and heard expert views on mobility, it is unlikely that they could equal—in the four-day process—the policymakers’ level of expertise.

Moreover, innovative ideas are sensitive to the data collected, which focused on the expressed initial preferences of decision-makers. Since policymakers do not publish their policy agenda, I could not consider their initial positions. One should thus keep in mind that the share of innovating ideas is likely to be smaller than that of aligning ideas.

Finally, the desk-based research suggested that the citizen panel exerted no shifting or inhibiting influences. Whereas inhibiting influence did not come up in the interviews, shifting influence actually did in a way that cannot be captured through the policy document analysis: the citizen panel helped to convince decision-makers that the mobility plan was supported by a majority of the citizenry, which resulted in its trouble-free adoption via inter-cabinet negotiation and the absence of a fierce political opposition. From that angle, the object of the panel’s influence was not the mobility plan but decision-makers, and that influence started long before the formal decision-making moment. At the point when the parliament had to convert the citizen resolution into a parliamentarian one, many political parties were hesitant about mobility reform, as a public official from the parliament recalls: “Some parties from the majority and the opposition were not happy (…) [with] of all the [citizens] conclusions (…). So, Charles Picqué actually had a lot of difficulties from a political perspective to get the Parliament’s Bureau to accept (…) the transformation of the resolution into a parliamentarian resolution.” A cabinet member made a similar observation: “What surprised me was that, at the start of the process, several French-speaking political parties were still sheltering [against a new mobility policy] (…). Yet, 1 year later, when the first version of the Good Move was going to be passed, I have noticed that, and I think this is not because [of] the citizen panel, well the citizen panel contributed because they saw that “(…)There is in Brussels a net majority in all social classes which really wants a change on the mobility issue”. This has definitely played [a part].”9 The citizen resolution helped to show decision-makers that there was actually no majority in favor of the status quo; rather, a majority of the Brussels citizenry wanted a policy change with respect to mobility. It pushed reluctant decision-makers to realize that there was a silent majority in favor of a change, and it gave the minister and his cabinet the means to convince reluctant decision-makers of the need to support their mobility plan, as the member of the cabinet mentioned: “the minister could expose some political parties (…) by saying, ‘You are wrong, you believe that the Brussels citizens are still attached to this and that, but look at the [citizen resolution]; the city is open for the discussion on mobility, for a change of mobility. There, you are wrong.’ ”

Policymaking moments

The desk-based research indicated that the citizen panel had a significant influence on decision-making, but the interviews revealed that it is unlikely that the citizen resolution exerted a substantial influence on that stage. The changes between the first and the final version were most likely explained by external factors rather than by the use or a reinterpretation of the citizen resolution. Negotiations at the decision-making stage mainly dealt with technical aspects of the plan, and cabinet of the minister for mobility tried to avoid questioning the core principles and values that were already accepted in the first version of the plan. The citizen resolution consisted mainly of general policy ideas with few technical details, which mainly influenced the elaboration of the plan but were not an appropriate source of input for decision-making, as a cabinet member explained: “So, to convince people around the table, and even more, people that work for Bruxelles Mobilité, the STIB and so on, now we are outside the purely political discussion at the level of head of cabinet and so on, but they are not always really impressed by these [the citizens ideas] because behind these propositions, there is still the technicity (…). Here often, if we don’t want something to be realized and that we are already against it, we try to find juridical, technical arguments so that it becomes a huge mountain to climb”10. The interviews confirmed that the plan’s fundamentals did not change between policy formulation and decision-making, suggesting that the citizen panel influenced the first version of the plan rather than the final one. The difference in impact can be explained by the sensitivity of the congruence approach to the examined document: the final plan indeed featured an extended section to present its motives, resulting in a greater congruency of framing components from the citizen resolution.

Overview of results and discussion

Different methodological approaches can be deployed to assess the impact of CPM. This analysis has applied them to a single case to see how they alter the findings and identify ways to increase their reliability (see Table 2 for an overview of the results). The analysis shows that the operationalization of the uptake and the selection of the policymaking moment do produce different results. A binary approach to the uptake amplifies the impact, while a plural approach reveals that many citizen ideas are only partially integrated into public policy. The selection of the policymaking moment also alters the findings, as the congruency approach indicates a larger impact at the decision-making phase. As for reliability, the more precise the measurement tool, the more the analysis integrates the context and produces fine-grained results. Using SIM uncovered an insightful new angle on the fate of citizen ideas: a large majority aligned with decision-makers’ initial preferences, and aligning ideas received considerably more uptake than diverging ones. These results suggest that decision-makers have cherry-picked from among citizen ideas, integrating those that chimed with their own political preferences. The best way to increase the reliability of the congruency approach is to combine it with interviews of actors involved in the citizen panel’s follow-up. This mixed method not only produced the most reliable findings but also revealed ways in which the citizen panel exerted a more diffuse influence on decision-makers and public policy, something desk-based research alone could not capture (see Discussion section, below). In terms of reliability, the interviews contradicted the congruency approach in two aspects. First, interviewees disagreed that the citizen panel had had a significant impact on the final version of the plan. Decision-making in this case centered on technical aspects and decision-makers did not want to amend the core principles and policies of the first version of the plan. Second, the interviewees questioned the innovating potential of citizen proposals, as policymakers explained that they learned nothing new from the citizen resolution.

Table 2

The overview of the methodological approaches

	Policy formulation			Decision-making
	Desk research	%	Interview	Desk research	%	Interview
Binary approach to impact	Impact	64.9	+	Impact	74.3	−
Plural approach to impact	Uptake	43.2	+	Uptake	55.4	−
	Partial uptake	21.6	−	Partial uptake	21.6	−
SIM	Aligning influence & Uptake	36.5	+
	Aligning influence & partial uptake	12.1	+
	Diverging influence & uptake	8.1	−
	Diverging influence & partial uptake	8.1	−

To standardize the congruency approach and increase the reliability of its empirical findings, I recommend employing both the SIM and a mixed method. The SIM’s main added value lies in its effort to make the impact assessment more consistent across cases. Its overview transparently displays the fate of each citizen idea and unveils the qualitative distinction that the researcher must make when capturing the congruency. Its consideration of decision-makers’ initial preferences—provided that these are available and accessible—also augments the context-sensitivity of the congruency approach and can reveal the extent to which decision-makers cherry-pick citizen ideas. It can furthermore uncover the fate of ‘exceptional citizen ideas’ which could inhibit or shift the decision-makers’ agenda but risk being undetectable in the myriad of citizen ideas. Regarding the selected moment of the policy cycle, this analysis cannot identify an appropriate generic moment because it depends on the context of each case. It can only call for careful consideration and justification when selecting the phase and policy documents. As for the mixed approach, it is a necessity if the impact assessment seeks to ensure a minimum degree of reliability. The congruency approach on its own only gives an indication of a potential impact and can thus easily produce a misleading picture. Therefore, the general trends identified via the congruency analysis must always be balanced against interviews of actors involved in the political follow-up. Interviews are extremely valuable because they confirm, nuance and invalidate the results of the congruency approach, and provide a good estimation of whether the congruency stems from the CPM or other factors. They are not, however, the ultimate way to check the causal validity for each citizen idea: it is practically complicated—if not unfeasible—to discuss all citizen ideas resulting from a CPM. Regarding who to interview, I recommend interviewing at least one key actor from each institution involved in the follow-up, so that the triangulation of methods can be combined with a triangulation of data.

Mixed methods should not only be viewed as a validation strategy, as they can add “breadth and depth to [the] analysis” (Fielding et al., 1986, p. 33), and generate valuable insights that are relevant to the broader literature on CPM and minipublics in particular. First, absent and partial uptakes may not be the result of an instrumental or partisan use of citizen proposals by decision-makers, who may have sound reasons for amending or rejecting them. This shows the importance for decision-makers to publicly justify the absence or partial integration of a citizen idea, so that citizens are aware of the consideration that their ideas received and can understand the reasons for any lack of uptake (Bua, 2019, p. 290), thereby avoiding eventual participatory frustration (Fernández-Martínez et al., 2020). Furthermore, although minipublics are supposed to formulate problem-solving and qualitative policy ideas thanks to information their participants receive (Setälä & Smith, 2018), the case indicates that this may not be sufficient to be perceived as a valuable input for the technical discussions or to generate fresh ideas that are unknown to policymakers. This finding is contingent on the minipublics’ length, but a four-days minipublic seems too short to reach the level of expertise of policymakers.

Second, although this case study’s interviews contested the fact that citizen ideas exerted an innovating influence, they also suggested that the citizen resolution contributed to shifting the preferences of several decision-makers. This contradiction highlights a subtle but important distinction between the impact of a citizen panel on public policy and its influence on decision-makers. While a congruency approach assumes that impact can be measured through examining the fate of each citizen idea individually, interviewees instead viewed the citizen resolution as a set of general political directions and principles on mobility. The citizen panel may have been seen as a sort of survey—or a “sophisticated feedback mechanism” as is has been termed by Bächtiger et al. (2014, p. 237)—which gives policymakers an indication of the responsiveness of their policy project to the expectations of a group of informed and diverse citizens. This difference in focus suggests that the citizen resolution exerted an influence on decision-makers that the congruency approach alone failed to grasp. The citizen resolution’s main political directions seem to have influenced decision-makers by showing that a majority of citizens were in favor of reforming mobility, which resulted in an impact on policy- and decision-making: namely the trouble-free adoption of the mobility plan.

This insight has two implications. It first suggests that the explanatory power of Vrydagh and Caluwaerts’ (2020) typology of influence depends on its analytical focus. Not only can their typology give an overview of potential cherry-picking practices, but it can also show how democratic innovations can provoke changes of preferences among decision-makers. These changes are nonetheless not as clear-cut as they imply, either injecting new ideas or causing a preference shift. In line with what Michels & Binnema have coined “strategic impact” (2018, pp. 4–5), citizen resolutions can exert a more diffuse influence that weakens or reinforces existing political preferences, thereby connoting that a CPM creates political winners and losers. The winners are the decision-makers whose political program is backed by the citizen resolution, while the losers are those whose programs are challenged. The case study suggests that the winning side used the citizen panel’s results to convince the losing faction of the need to reform mobility, serving as a trigger for the losing side to shift their preferences. Had the citizen panel formulated a resolution in favor of the status quo, it is unlikely that policy- and decision-makers would have dared to propose such a reforming mobility plan, and nor is it likely that other political parties would have ratified it.

The second implication refines Lafont’s (2019) conception of the political role of deliberative minipublics. She asserts that their added value lies in their capacity to reveal (mis)alignments between public policy, the recommendations from a minipublic, and the actual majority in the public opinion. The case study reveals that, for decision-makers, public opinion is not a given but rather is constructed through what they encounter in the public and political spheres. These encounters can nevertheless fail to channel the citizenry’s preferences accurately to decision-makers, who end up with conflicting perceptions of what the citizenry wants. This results in political struggles to determine which perception of public opinion comes the closest to reality. By displaying a misalignment between the perception of a majority supporting the status quo and the actual silent majority in favor of reform, it appears that the citizen panel allowed one political side to demonstrate that their perception best resembled actual public opinion.

Finally, the citizen panel also had a diffuse and subtle influence which resulted in a structural impact by changing existing policy and institutional practices (Jacquet & van der Does, 2020a; Michels & Binnema, 2018). As the first experience of citizen participation in the Brussels Regional Parliament, it launched a dynamic which eventually resulted in the institutionalization of deliberative commissions. After the citizen panel, several representatives and public officials were satisfied with this experiment and willing to go a step further. Two years later, the regional parliament modified its internal regulations to institutionalize deliberative committees, in the form of minipublics with randomly selected citizens and members of the parliament (Vrydagh et al., 2021). These insights highlight the importance of conducting case studies of CPM in parallel to broader comparative studies, as they contain valuable data to obtain a better understanding of the minipublics’ functions and embeddedness in the political system and can unveil influences and effects that are subtle and diffuse in their shape but significant in their outcomes.

Conclusion

This paper has intended to standardize the impact assessment of consultative participatory mechanisms and propose ways to increase the reliability of the findings. It has compared different variations of the congruency approach and suggested two new methodological add-ons: the sequential impact matrix (SIM) and a mixed method. It then applied these methodologies to a typical case of consultative participatory mechanism: a citizen panel on the mobility in the Brussels-Capital Region. The results show that different applications of the congruency approach alter the findings and can produce a misleading picture of the citizen ideas’ uptake. To standardize the impact assessment I suggest adopting the SIM, which features a transparent overview allowing us to trace the fate of each proposal. It also considers the decision-makers’ initial preferences, revealing the extent to which they only take up citizen ideas that align with their own agendas. To increase the reliability of the findings, I recommend a mixed method to balance the general trends identified via the congruency approach with the experiences of actors involved in the political follow-up. A mixed method is also likely to generate insights as to the more subtle influences that consultative participatory processes can exert on decision-makers and policymaking. The intention of this paper is not to impose an orthodox methodology for impact assessment, but rather to open the discussion on this exercise and inspire other works to develop new assessment techniques that capture the CPM’s genuine political and policy contributions. In parallel to assessing the impact on public policy, we should explore ways to measure the influence on decision-makers, for instance via surveys, interviews, and systematic process-tracing. Future research should also develop tools to increase the coding reliability, such as using a couple of researchers to code and evaluate the textual congruency. Finally, future research should investigate if we can apply a methodological strategy uniformly to all types of CPM in different political contexts.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 69 kb)

Table 3

The overview of the interviewees

Policymaker 1	Online	Member of the public administration Bruxelles Mobilité, closely involved in the integration of the citizen resolution in the first version of the mobility plan
Policymakers 2	On the phone	Member of the public administration Bruxelles Mobilité, closely involved in the integration of the citizen resolution in the first version of the mobility plan Key member of the cabinet of the Minister of mobility of the Brussels Region (2019–…)
Policymaker 3	Online	Key member of the external consortium Smartteam, which assisted the public administration Bruxelles Mobilité and the elaboration of the mobility plan
A cabinet member	Online	Key member of the cabinet of the Minister of mobility (2014-2019), closely involved in the political follow up of the mobility plan
A public official of the Brussels Regional Parliament	Online	Public official closely involved in the initiation, organization, and institutional follow-up of the citizen panel
An organizer of the citizen panel	In person, on the phone	Key member of the organization of the citizen panel
Key member of the Brussels Regional Parliament	In person	Elected member of the Brussels Regional Parliament, played a key role in the initiation, organization and institutional follow-up of the citizen panel