Comparative Policy Analysis and the Science of Conceptual Systems: A Candidate Pathway to a Common Variable.

In comparative policy analysis (CPA), a generally accepted historic problem that transcends time is that of identifying common variables. Coupled with this problem is the unanswered challenge of collaboration and interdisciplinary research. Additionally, there is the problem of the rare use of text-as-data in CPA and the fact it is rarely applied, despite the potential demonstrated in other subfields. CPA is multi-disciplinary in nature, and this article explores and proposes a common variable candidate that is found in almost (if not) all policies, using the science of conceptual systems (SOCS) as a pathway to investigate the structure found in policy as a lynchpin in CPA. Furthermore, the article proposes a new text-as-data approach that is less expensive, which could lead to a more accessible method for collaborative and interdisciplinary policy development. We find that the SOCS is uniquely positioned to serve in an alliance fashion in the larger qualitative comparative analysis that supports CPA. Because policies around the world are failing to reach their goals successfully, this article is expected to open a new path of inquiry in CPA, which could be used to support interdisciplinary research for knowledge of and knowledge in policy analysis.

Introduction

In the 1990s, comparative policy analysis (CPA) and comparative analytical studies, as a modern research tradition, joined comparative politics and comparative public administration, using the comparative method (Geva-May et al. 2020: 368). Peters and Fontaine (2020: 29) contend that the comparative method, as outlined by Lijphart, provides various opportunities for scholars of public policy to enhance their collective understanding of policy and policy processes. Peters et al. (2018: 137) argue that CPA is in fact precisely designed to explain policy outcomes.

Cairney and Heikkila (2014: 383) suggest that, by building on established literatures, new lenses on public policy seek to improve upon rather than compete with or replace existing perspectives. Thus, the posture taken in this article is that there are challenges in the field as well as opportunities, and this article suggests an opportunity to advance comparative public policy (Wong 2018: 963).

Research seems to suggest that a common issue in comparative research is the great difficulties in comparing policies across a variety of situations/contexts, which is referred to as the problems of identifying common variables (Wong 2013, 2016; Haque 1996; Welch and Wong 1998). In identifying future challenges and opportunities of the development of comparative public policy, Wong (2018) recognises the efforts made by scholars calling for collaboration and interdisciplinary research in comparative public policy, which is a problem that has for decades not been fully addressed.

A second issue in comparative public policy, and in particular CPA, is the fact that text-as-data methods have been rarely applied widely, despite their potential, which has been demonstrated in other subfields (Guy Peters and Fontaine 2020: 203). Additionally, according to Guy Peters and Fontaine, “[t]hey have not created new text-as-data approaches as such.”

Peters (2020: 21) argues that there is a clearly a need to utilise other methods and techniques to make comparisons. According to Peters and Fontaine (2020: 14), CPA is in itself multi-disciplinary and as such more akin to multi-methods than any social sciences area. As a candidate to remedy this gap in literature, this article suggests the science of conceptual systems (SOCS) and in particular the use of integrative propositional analysis (IPA) methodology to examine the structure/structural logic of policy models as a way to provide a ‘common variable’ in CPA.

The SOCS is aimed at the pursuit of knowledge and understanding of conceptual systems using rigorous methodologies (Wallis 2016). Conceptual systems are defined as any collection of interrelated concepts found in theories, models, policy models, axioms, laws, strategic plans, and so on, which have a set of interrelated propositions. Generally, all conceptual systems from social sciences to hard/natural sciences have one aspect in common: they have some level of structure, which makes them amenable to an IPA-based analysis or evaluation. We will delve into the IPA in detail below.

Part of the motivation for this study is the critique and necessity for the systems-based approach of the SOCS. This is the problem of linear and simple policymaking (Sabatier 1999), which causes a mismatch between how real-world systems work and how we think of them (Cabrera and Colosi 2008). The simplistic models used to make decisions in complex systems lead to “worse outcomes than the previous status quo” (Beaulieu-B and Dufort 2017: 1) and shortsighted practices (Sterman 2012: 24). As such, implicitly the article also uses the SOCS and IPA as a vehicle to carry out the impact of systems thinking and its benefits in policy analysis.

The contribution of this article is twofold. Firstly, it explores how the structure that is found in each policy model or conceptual system can serve as a common variable when comparing policy. Secondly, the article provides a text-as-data approach with rigour and greater accessibility, which can be easily acquired and applied by policy makers, practitioners, and scholars. The article will specifically illustrate this by explaining how policy as a unit of analysis can be used to integrate theory, policy and research based on a ‘common language’ of structure, which will potentially allow for comparative analysis across systems.

To achieve this, the article is structured in three sections. Firstly, there is a review of comparative public policy and comparative analysis. Secondly, the article provides a background of the SOCS. Thirdly, the article illustrates how and why policies (policy models) are amenable to evaluation based on their structure, which will be shown to generate novel insights into the ability of IPA to improve CPA across different systems.

Comparative Policy Analysis

Lasswell (1971) made a distinction between knowledge of and knowledge in the policy process. However, both are very important for the purpose of this article. With regard to knowledge in the policy process, Radin and Weimar (2018: 8) state that this perspective looks at “how can analysis improve the content of public policy?” This was premised on how the political process affects policy content application and so on. This current article looks at the structure of policy implementation, and in turn examines how the data that underlie the structure of a policy influence the structure of the policy.

Policy analysis is defined as the use of reason and evidence to choose the best policy among a number of alternatives (MacRae and Wilde 1979: 14). Dror (1983: 79) defines policy analysis as a “profession-craft clustering on providing systematic, rational, and science-based help with decision-making”. Brans et al. (2017) argue that what is central to policy analysis has always been the principle that decision-making should be systematic, evidence-based, verifiable and evaluative (transparent and accountable). Geva-May et al. (2020) maintain that evidence-based policymaking also implies, by definition, the search for evidence ‘elsewhere’ for historical, international, disciplinary, or other comparisons of data, facts, and events. Policy analysis also refers to analysis for public policymaking, such as the activities, methodology and tools used to assist and advise in the policymaking context (Parsons 1996; Hogwood and Gunn 1984; Mayer et al. 2004; Dunn 1994; Fishcer et al. 2007). This is also referred to as the interventionist branch of the policy science tree (Enserink et al. 2012). As such, policy analysis, as interventionism, likened to other disciplines, needs to bridge the gap between science and action (Latour 1987).

Reviewing the literature above, two aspects of policy analysis become very evident. Firstly, policy analysis is a means to improve decision-making supported by evidence and secondly, evidence can also be found elsewhere as argued by Geva-May et al. (2020)—it does not always have to involve conventional policy analysis methods. It is fundamental to note here that the approach and methodology suggested in this article will support other existing forms of analysis, allowing for greater understanding of policy outcomes or the development of better policies.

Radin and Weimer (2018: 8) support this idea by arguing that at the most general level, science in many disciplines produces policy-relevant research that can inform policy design. Guy-Peters et al. (2018) also concur with this idea, maintaining that in understanding the various alternatives for comparison, it provides interested researchers with flexibility in explaining policy outcomes, although this requires us to be thorough and to avoid bias. Additionally, Radin and Weimer (2018: 2) note that researchers in many fields contribute to knowledge that is potentially relevant to public policy. Guy-Peters and Fontaine (2020: 14) agree that because of the multidisciplinary nature of CPA, it is more akin to multi-methods than any other social sciences areas. It is clear that policy analysis, as an interdisciplinary field, always leaves the door open for alternative knowledge that would improve evidence supporting decision-making. In this article, we explore how the structure found in policies can add insight to our comparative studies.

Drawing on the reviewed literature for this current article, the perspective of ‘other comparisons of data’ or alternative knowledge will be taken, by using the emerging SOCS as a pathway to suggest the structure of policies as a common variable for CPA and by extension provide an alternative text-as-data method to policy analysis.

It is important to note that policy analysis methods are used to help analysts to design and assess policy alternatives systematically (Radin and Weimer 2018: 8). Building on the idea that we can use other evidence to aid analysis, Guy-Peters et al. (2018) suggest that in comparative work, when focusing on the nature of policy itself, it is amenable to either quantitative or qualitative methods.

The next section conceptualises a public policy as a conceptual system and explains why and how it is amenable to the SOCS. Van de Ven (2007: 278) holds that investigating and analysing the internal logics of policies can broadly be seen as a form of design science, policy science, or evaluation research.

Public Policy/Models/Theory as Conceptual Systems

Public policies can be viewed as policy design and the output of analysis (Kingdon 1997), which means that the documents representing the content of policies can be viewed as shared understanding. These generally include wordy or text-based design artefacts, such as legislation, guidelines, pronouncements, court rulings, programs, and constitutions (Ingram and Schneider 1997). These policy designs (their content) can be seen as abstract representations of physical world systems (Schwaninger 2015, p. 572), which is intentional in approximating physical systems by building an artificial system (Simon 1969)—this artificial system, can also be found in our conceptual systems.

Policy design as conceptual systems can be conceptualised in the following manner: A system can be seen as a set of elements or parts with interactions among the components of the pattern/structure (Meadows 2008). Policy designs as conceptual systems have a structural logic existing of elements (variables/concepts/boxes) and the patterns (causal relations/arrows in a diagram) in which the elements of the policy will occur (Mohr 1987). Now, as suggested by Schneider and Ingram (1988), because we can diagram a sentence linking together parts of speech, it is also possible to diagram the structural logic of a policy. A key assumption of this article is that more useful/effective policies will be more structured.

Wallis (2020b), drawing on Warfield (2003: 515), defines a system as “any portion of the material universe which we choose to separate in thought from the rest of the universe for the purpose of considering and discussing the various changes which may occur within it under various conditions”. He further makes the distinction between the ‘material’ universe as being separate from the ‘conceptual’ universe and emphasises the useful relationship between the two. The SOCS can thus be seen as an extension of systems thinking to describing, investigating, and understanding policy designs as conceptual systems (again extending systems thinking to policy analysis at the conceptual level).

This distinction and maybe more importantly the relationship between the two universes allow for a more holistic view to policies, thus not only studying the material universe “policy environment” systematically, but also making sure that we study our conceptual universe systematically. This is premised on Ashby’s law of Requisite Variety, building on the idea that the control mechanism (policy) must have greater or equal complexity with regard to the system it intends to control, or in this case the environment it intends to address (Ashby 1957). This premise suggests benefits of looking at policy models as conceptual systems.

One key benefit of viewing public policy models as conceptual systems is that, by building a policy map, it enables investigations into the likelihood of negative policy interactions (Siddiki 2018). Moreover, using the systemic mapping approach to facilitate more readily policies that are built on interdisciplinary theory as the study of policy design as conceptual systems allows us to investigate the policy (and perhaps why it failed) in its entirety, instead of a reductionist approach—as outlined in the introduction.

It was already briefly suggested earlier that the structure of our policies or our conceptual systems has an impact on the practical application or implementation of our policies. As such, drawing from the emerging SOCS, and remembering that policy analysis knowledge can be drawn from various types of evidence or scientific evidence, we now turn to the new science to explore how it can be a common variable in CPA.

The Importance of Causality for Structure

Causality in this sense can be understood as the relationship between the concepts in ae conceptual system. Additionally, it can be argued that these causal relationships create the structure by connecting different concepts. According to Sloman and Hagmayer (2006: 408), who draw on Bayer’s nets theory, “studies of learning, attributes, explanation, reasoning, judgement and decision making suggest that people are highly sensitive to causal structure.” What is important here is the understanding that causality is essential, and Wallis (2016) suggests that we can improve the structure of policy by improving the causal relationships between concepts.

With regard to structure, we look at the relationships between the concepts (causal logic); even though in our map’s causal logic (concepts without any causal relation) forms part of the structure. However, the more relationships there are (arrows between boxes/circles) the more structured and the more useful diagram/conceptual system is. In this science, we look at usefulness. Wallis (2020c) makes a good argument for this, drawing on Saltelli and Funtowicz (2014) for whom “all models are wrong, but some are useful”. This is evident because some policies/theories fail, and others are useful for their purpose. According to Wallis (2020c), the structure of policies represents a kind of usefulness. This has been seen in over 30 years of science investigating structure.

The Science of Conceptual Systems: A Brief History

Briefly, Cabrera (2006: 3) argues that concepts exist in a system made out of other concepts which has interconnected patterns and is a conceptual ecosystem. These concepts are bound by causal connections, which form propositions that are examples of “a declarative sentence expressing a relationship among some terms” (Van de Ven 2007: 117) and that create a set of statements understandable to others by making predictions about empirical events (Baridam 2002: 7). At the elemental level, policies like theories have concepts and causal relations with underlying data that create propositions as understandable statements, which implies that they are commutable and public (Baridam 2002). Like theories, we use policies to make predictions about empirical events, such as implementing COVID-19 regulations and measures (problems and solutions on paper) in the hope of stopping infections and eventually having a virus free country (empirical event), such as New Zealand and others.

According to Wallis (2016), the SOCS is aimed at the pursuit of knowledge and understanding of conceptual systems whilst using rigorous methodologies. At least three streams of research on structure suggest that structured knowledge is useful for changing the world positively and reaching desired goals. These streams are as follows:

In the field of education/human development, “Systematicity” is used to evaluate the structure of maps and evidence. This suggests that beginners create simplistic low structured maps compared with the more structured and complex maps of experts (Novak 2010).

In the field of political psychology, the measure of Integrative Complexity has been used for the past 30-years (Suedfeld et al. 1992; Wong et al. 2011).

The third stream of research on structure involves studies of formal theories and policy models within and between multiple disciplines (Wallis et al. 2016).

The third stream, which is of great significance for this current article is IPA.

The IPA Method

According to Wallis (2016), IPA is primarily used to analyse conceptual systems from text on paper to determine their structure (Wallis 2016). The IPA methodology uses the policy document’s text itself as data (Wallis 2016: 585). This process includes the following six steps: (1) Identify propositions within one or more conceptual systems (models, etc.). (2) Diagram those propositions with one box for each concept and arrows indicating directions of causal effects. (3) Find linkages between causal concepts and resultant concepts between all propositions. (4) Identify the total number of concepts (to find the Complexity). (5) Identify concatenated concepts. (6) Divide the number of concatenated concepts by the total number of concepts in the model (to find the Systemicity).

For a very brief and abstract example, consider Fig. 1. The figure has three variables/concepts (A, B, C), therefore, the Complexity is C = 3. There is one concatenated concept (C). So, the Systemicity is C = 0.33 (the result of one concatenated concept divided by three total concepts).

Fig. 1

Abstract example of a model for demonstrating IPA

Concepts (relating to variables) are enumerated to show the Complexity or explanatory breadth of the conceptual system. The causal interconnectedness of those concepts is evaluated for their Systemicity (structure, or explanatory depth). Systemicity is measured on a scale of zero to one with one being the highest (Wallis and Valentinov 2017: 109). Using IPA, Complexity, on its own, is seen as a weak indicator of success for a conceptual system, building on the idea of the SOCS. The main concern for this measurement of structure is that those theories, policy models and general conceptual systems with a higher level of structure are more useful for practical application and implementation.

It can be argued that the external validity of the methodology has been established. Firstly, the findings suggest that theories in the natural sciences have high levels of structure (Systemicity) and are proved to be effective and useful in application, such as Ohm’s Law (Wallis 2016, b; Wallis 2010a, b). In contrast, theories in the social sciences have a Systemicity generally of less than 0,25, including theories of conflict, psychology, and very important for this article, policies (Wallis 2010a, b, 2011; 2013; Shackleford 2014; Parmentola et al. 2018; Wallis et al. 2016; de Wee 2020; de Wee & Asmah-Andoh, in press). Correlating with these findings, Light (2016) found that policies in the United States of America only succeed 20% of the time. This is parallel to the 0, 25 Systemicity generally found in social sciences, which essentially translates to a 75–80% chance that most of our policies could fail.

The Conceptual System: A Basis for Comparison

Briefly, in SOCS, using IPA, we quantify and diagram the structural logic of the policy, and find links between the measure (percentage providing a predictor) of the policy structure and its usefulness in the real world. As a brief example, one could consider Figs. 2 and 3 that are maps of different theories, one from physical science and the other from social science. These propositions are mapped out to indicate how we can use the structure found in any policy as the basis of comparison.

Fig. 2

The structural logic of Ohm's law as a practical map (Wright and Wallis 2019, p. 152)

Fig. 3

The structural logic of the democratic peace proposition as a practical map.

Source: Authors own compilation

Georg Ohm developed the propositions: An increase in resistance and an increase in voltage would result in an increase of current, which will cause a decrease in resistance. An increase in current causes an increase in voltage that would result in an increase in resistance, which will cause a decrease in current.

The democratic peace proposition has many possible empirical and theoretical forms. One of these holds that “the more democracies there are in a region or the international system, the more peaceful the region or international system will be” (Reiter 2012)

The two brief examples were created using IPA steps 1–3 (to diagram) and the person reading these propositions from any country who reads the English language can see the causal relations between the concepts. This is the benefit and perhaps the most important contribution of the SOCS and IPA to CPA. Firstly, the examples indicate how IPA takes a “common denominator” approach by presenting theories graphically in an easily understandable format of concepts (in boxes/circles) and causal arrows (indicating the direction and causality) (Wallis 2020a, b, c). The key contribution here is that with IPA we have an objective and rigorous method we can use to evaluate, analyse, and develop policies based on their structural logic, which was first suggested by Schneider and Ingram (1988); however, without a clear measure. Furthermore, it also provides a “common language” for interdisciplinary work, which is important for policies if they are to be successful, remembering of course that no problem in policy is ever addressed using only ‘one’ policy. As Siddiki (2018) found, policies that are used to address a problem often led to inter policy conflict that causes problems with “policy coherence”, which May et al. (2006) refer to in terms of how well policies with similar objectives fit or go together.

Additionally, IPA and the SOCS can be of great benefit for policy design and CPA. Applying the IPA’s steps 4–6 to Figs. 2 and 3, their level of structure differs significantly. Ohm’s law has a structure/Systemicity of 1 (100% usefulness in application) and the democratic peace proposition, has a structure/Systemicity of 0. In the real world, Ohms law has been very successful, and the democratic peace theory has numerous shortcomings.

For years, it has been accepted that policy designs are “copied, borrowed or pinched” from similar policies in other locales (Schneider and Ingram 1988: 62) which makes policy design less a matter of invention than of selection (Simon 1981) involving large stores of information and making comparisons. However, this led to suboptimal situations where policy layering and patching is done haphazardly (Van der Heijden 2011) leading to a palimpsest-like mixture of incoherent or inconsistent policy (Howlett and Rayner 2007; Carter 2012). Again, perhaps using IPA and viewing policies as conceptual systems allows one to create coherent policies more easily, which are argued to be more effective in application (Siddiki 2018).

Based on the case made earlier, this stream of research seems to provide policy analysis with policy-relevant research that can inform policy analysis (Geva-May et al. 2020; Guy-Peters et al. 2018; Radin and Weimer 2018). Hence, it provides this current article the basis for suggesting the evaluation of the structure of policy as a common variable to be used in CPA, additionally because this “on paper” analysis provides a new text-as-data approach. Although other aspects of policymaking and analysis, such as empirical data and the implementation process, are important, we suggest focussing on the structure of policy in comparative analysis. This article suggests the different but allied perspective that this stream of research provides, which will be explained later.

Extending on the above, pragmatically, inter-rater evaluation can be done, where scores of the analysis can be compared and the consensus of the raters measured through agreement or concordance (Bless et al. 2016: 226). Prior studies indicate how a repeated study using the same data would have stable findings over repeated observations, which confirms its external validity, or generalisability (de Wee 2020: 135).

In the followings section, we turn to IPA and the application of the method on conceptual systems including policy. The section not only demonstrates the methodology but also the use of the method on policy in previous research. Additionally, it indicates how less “financially demanding” the method is than the computational and statistical models. Based on using textual analysis (IPA) on policy, the article suggests that this is an alternative or new pathway to compare, analyse and evaluate policies. The next section discusses text-as-data and the SOCS compatibility with it.

A New Text-as-Data Approach: The Basis for Collaboration and Interdisciplinary Policy Development

Current research (Fisher et al. 2013; Leifeld 2013; Guy-Peters and Fontaine 2020; Leifeld and Haunss 2012) indicates the prevalence of qualitative text analysis methods in policy analysis, including the discourse networks that rely on text analysis to measure discourse coalitions quantitatively through network analysis. Guy-Peters and Fontaine (2020: 204) argue that text-as-data-methods for CPA can include new theories and methods relevant to public policy and policy analysis.

Guy-Peters and Fontaine (2020) identify the different directions text-as-data application is currently developing. Firstly, there is causality, with Egami et al.’s (2018) framework of estimating causal effects in sequential experiments. Owing to limited space, this process will not be explained in full (see Egami et al. 2018). Secondly, there is computer science research on word embedding and on artificial neutral networks, which is what they call ‘deep learning’ (LeCun, Bengio & Hinton 2015). From the findings of Guy-Peters and Fontaine (2020), it becomes apparent that the use of statistical and computational analysis is a challenge in the discipline, especially because there are no globally best methods for retrieving certain information from text. Secondly, with manual approaches, this is practically impossible because of the prohibitive costs (Guy-Peters and Fontaine 2020: 213).

This article suggests a key benefit of viewing the IPA method as part of the SOCS when doing CPA as a new “text-as-data” technique. IPA objectively evaluates structure, and it provides a way for comparison. Additionally, the text-as-data method outlined in the most current literature tends to have two key problems: firstly, it is difficult to use and secondly, it is expensive and practically impossible. To the field of CPA, IPA provides a method that can be used easily and with rigor, which is not mentally taxing or too expensive. To the text-as-data method, this article provides accessibility. The IPA method is more accessible and can be applied with ease by scholars and practitioners from various countries and various disciplines to improve the policies they design. This method also provides a basis from which interdisciplinary work on policy can branch out.

As such, when analysing policy, one can do a within-case analysis and thereafter a cross-case comparison, to generalise findings further, whilst considering the context. However, we will explore this further in the next section.

Collaboration and Interdisciplinary Perspectives Based on Policy Structure

In fact, the IPA methodology and the resources to assist in the application are readily available online at https://projectfast.org/, where there are additional tools to enhance the analysis, which can introduce more collaboration. Collaboration can be achieved, through analysing a policy individually and outcomes. Or you can build or analyse the policy together using the six-steps of IPA on the free online social network analysis software KUMU (www.kumu.io). This platform allows participants in policy analysis or policy making to organise complex information (concepts and causal relations of a policy based on IPA), to diagram graphically, to build and to analyse policies based on their structure. This platform is open for multiple participants globally who are given permission to participate. Potentially, all participants can have the same policy or empirical data/research and can build a policy or map on this platform; however, this article will present more on this topic in the section on future research areas and implications.

This article argues that when exploring integration studies and literature around interdisciplinary research or problem solving, it becomes apparent that we can use an interdisciplinary approach, which can clarify the observer’s standpoint, define and orient the observer to a problem. Moreover, this approach can be used to map the full social and decision-making context and apply multiple methods to generate, evaluate and implement solutions (Clark 2002, 2011).

Ideas, concepts, and methods around interdisciplinary tools of the policy sciences to problem solving have been available for decades (Lasswell 1971; Brunner 1996); however, they have not been effectively used it seems. It has been recognised that the current complex problems cannot be resolved with just theories and policies but require contextual interdisciplinary understanding (Clark et al. 2011). One of the interdisciplinary questions to policy problems is “what is the common problem that underlies the increasingly apparent disconnect between real world problems and the knowledge and skills currently offered by policy makers?” For this article, the answer lies in the fact that we have policies that are developed but do not reflect our context enough, most of which are often imported from somewhere else and do not address the complexity of the environment.

For example, Wright and Wallis (2019) wrote a paper on poverty and the various explanations for poverty in the United States. These various “theories” on why there is poverty were then integrated using IPA, which led to a more coherent and interdisciplinary explanation as to why poverty exists. The benefit of the study suggests that by using IPA, one can synthesise various theoretical/data perspectives and create an interdisciplinary explanation to guide policy. A second and a more recent example is the study on responses to COVID-19 by Fink and Wallis (2020a, b, c), where they looked at the models used to combat the disease. Two countries (Germany and New Zealand) handled the crisis the best, as their policies were built on a structured scientific model compared to that of the US and the UK, where the leadership generally rejected scientific results and responded with single values, which caused the US to have the largest number of infections. Here the simple single value approach coupled with a general denying of scientific data led to the US struggling in combating the virus. Again, this feeds back to the problem of having models that are not equal or at least the same as the complexity of the virus.

These two examples are used to indicate how interdisciplinary research can be used to improve CPA and policy development in general. Due to space, a complete case study is not possible. However, evidence seems to suggest that complex and structured plans/policies supported by good interdisciplinary data could lead to better action and decisions by our leaders in attempting to improve the human condition. Thus, the contribution of this article is to indicate that there is an accessible and rather easier method one can use to improve our society.

Echoing the words of Le´le´ and Norgaard (2005), this article suggests we identify the structure or lack thereof, of policies as a common variable. As demonstrated by Wallis (2019), “based on the structure of policies we can pull together what is known about a problem from academic and practical experience” (Bammer 2013:6) and produce a more accurate and complete understanding for more effective action (Newell 2001: 22). Theoretically this is sound, but pragmatically how can this be done in CPA?

A Few Practical Points for Collaboration and Interdisciplinary Work

The text-as-data approach of IPA, which is part of the larger SOCS, can be advanced as an answer to the call made by Wong (2018) for more interdisciplinary research.

Two policies can be comparatively analysed in two different (or multiple) states, for example, a drug policy in South Africa and another in the United States of America. The structure can be measured and compared. This allows us to trace how a design evolves over time.

Using IPA, you can design policies, map them out and see where there are concepts/variables (represented by circles/blocks) on your map that might have no arrows leading to any other box. Moreover, you can apply “gap analysis” by filling the gap with concepts (supported by data) and creating links with the other concepts.

Addressing “unknown unknowns”, Wallis (2020a) argues that practically speaking, something that is unknowable for one person, or in one policy environment, system, or country, can be known by another. Thus, when we look at the systemic structure of policies comparatively, we can identify and determine when something is “missing” in our understanding (based on the diagram) and use different policies and policy empirical data to supplement our own map.

Finally, drawing from the discussion in this section, the idea of using IPA as a text-as-data methodology is supported by the assertion of Guy-Peters and Fontaine (2020: 203) that like other inductive analysis procedures, IPA also makes it possible to discover new phenomena, concepts, and relations from the latent dimension of a text. This notion of new concepts and relationships is supported by Geva-may et al. (2020:368) who state that the comparative perspective at micro- or macro-levels is paramount to effective and efficient policy development. Moreover, this novel idea based on the SOCS is one that can potentially advance effectiveness and efficiency. Using IPA in concert with other conventional CPA methods can improve policy analysis.

Next, the article places IPA as a potential method to be used in qualitative comparative analysis (QCA).

Qualitative Comparative Analysis and the CPA

QCA approaches focus on not only the independent variable and its necessity or sufficiency for an outcome of the dependent variable; it also focuses on how an outcome can be achieved by various configurations or combinations of independent variables (Schneider and Wagemann 2012; Guy-Peters and Fontaine 2020: 43). Guy-Peters and Fontaine (2020) continue that based on this idea, QCA is similar to most-similar and most-different systems design, as they are also based on the concepts of necessity and sufficiency. QCA can produce empirically well-grounded, context-sensitive evidence about policy instruments (Pattyn et al. 2019), such as IPA when looking at policy models as conceptual systems. An advantage of QCA for public policy analysis is that it enhances analysists’ ability to capture the outcomes of decision-making, implementation and evaluation accurately.

Guy-Peters and Fontaine (2020: 7) argue the following: “If we take political science, economics, and sociology as the “heartland” of CPA, then existing methodological discrepancies among scholars may not be too extreme”. However, these are not the only disciplines contributing to CPA. It is argued that many other new disciplines bring complementary approaches, hence new insights on causality and causation. Here they refer to various disciplines, such as anthropology, law and history with their techniques, which “…are other sources of evidence (that) contribute to the development of CPA” (Guy-Peters and Fontaine 2020: 7). It is clear that CPA draws on various disciplines, and as such, it is sufficient to argue that with the insight of the emerging SOCS, IPA can also be used when analysing policy.

The suggestion is that this methodology of evaluating the structure of the policy against the implemented results can sit comfortably with the QCA method of which the “primary goal is to discover empirical relationships that can help theoretical proposals” (Anckar 2020: 43). Of course, one has to remember that QCA is premised on the idea that an outcome (a certain value on the dependent variable) can be reached by several combinations of conditions (i.e., independent variables) (Anckar 2020: 43). The contribution of this article is to propose a new dimension to the present methods, specifically focusing on analysing and evaluating policy documents (policy models or policy-like documents), based on their level of structure, which can be done across policy fields.

From the above, it is clear that for CPA and for the advancement of policy development itself, there is a need for the triangulation of methods and theory that can provide evidence to enhance effective decision-making. Thus, this article agrees with Guy-Peters and Fontaine (2020: 14, emphasis added) that future studies must ‘think about triangulation and the use of multi-methods as a means of gaining a more complete picture of the policy issue under scrutiny”. Next, the article suggests potential future research areas in CPA.

Implications and Future Research

Although this is a preliminary exploration, conclusions can be drawn and taken from this article, as it is a starting point to a conversation about finding a common variable in CPA and introducing a new text-as-data approach.

The conceptual principles drawn and the introduction of a new potential field in CPA, using structure as a common variable, can be a starting point for future studies aiming to test the usefulness of applying IPA in CPA empirically. Scholars can analyse and evaluate policies from different environments, bearing in mind not to sample only a dependent variable based on successful (implementation) cases, which will take away the idea of an ‘inferential felony’ (Geddes 2003). IPA can be used in QCA, comparatively analysing policy outcomes with other analysis methods to test if the hypothesis of IPA holds, and potentially falsify it, which will advance the literature about IPA.

Two key deductions can be made. Firstly, comparing a policy based on its structure (coherence) can improve both the implementation success and understanding of a policy environment. Secondly, based on the ‘common denominator’ of IPA (applicable to all theories/policy models) we now have a ‘common language’ we can use for CPA when designing policies for policy relevant research, which in turn provides a good platform for interdisciplinary research.

Most studies of policy analysis compare concepts/elements (in policy) to the real world. However, such a reflective hermeneutic approach can only study selected chunks of policy situations comparatively. Using the text-as-data, or textual hermeneutic approach, analysists can use IPA to evaluate the policy, and identify potential gaps in the policy (where there are concepts with no causal relation to other concepts or where there is just a fuzzy implied causality). These gaps between concepts can be identified as potential reasons, which can pinpoint where analysis can be done. As such, future studies that aim to do comparative analysis can use the policy document (and its internal logic structure) itself as a lynchpin to determine or pinpoint potential areas that could have led to policy failures or success. A broader number of variables for comparative studies can be identified, which could require more interdisciplinary research to make for a more nuanced understanding of policy outcomes, which has for decades not been fully answered (Wong 2018). Thus, the article has interdisciplinary implications for scholars from various disciplines producing policy relevant evidence.

Because of limited time, space and resources, there are various aspects of CPA that were not covered in this attempt to introduce IPA to this relevant branch of research. However, because this is a preliminary exploration, future studies should cover more of the field. Additionally, the conversation through academic articles can be advanced to position the SOCS and IPA better in CPA.

Concluding Remarks and Limitations

Premised on the fact that it draws on the long history of the SOCS, attempts to answer longstanding calls for a common variable in CPA and suggests a new text-as-data method for CPA, this article is somewhat exploratory in nature. Moreover, this article is preliminary, in that it wants to locate the SOCS as a possible pathway to compare, analyse and evaluate policies based on their structure to improve both policy development and how we compare across systems. Depending on more extrapolation, this article presents CPA as a start to a new area for comparison.

The article explored avenues that could potentially contribute an answer to the calls for “a common variable” method that facilitates interdisciplinary and collaborative research and a new accessible text-as-data tool. In its exploration of the literature, the article established that the SOCS and the measurement of structure is a candidate for a common variable, as all policies in the world (dare we say) are written on paper. Additionally, the article provides an existing and developing method (IPA) as a tool to analyse policies and suggests the practical benefits of this tool to CPA (see future research areas and implications).

The use of the structure of the policy in CPA as a lynchpin in analysis or policymaking can bring various variables, concepts, and ideas into play. This approach enables policymakers, academics, practitioners, and other interested participants to find and manage potential “unknown unknowns” about a certain field of policy, such as health and the economy, because of the interdisciplinary, integrative and collaborative characteristic of the SOCS and IPA. Moreover, this could potentially improve our policy feedback, and policy learning in cross system comparative analyses.

What is important to take away here, which is in line with the findings of MacRae and Wilde (1979), Dror (1983), Brans et al., (2017) as well as Geva-May et al. (2020) is the fact that based on the structure of a policy as a variable, one can develop systemic and evidence-based policy. This is because IPA measures how structured a policy is and how well it reflects the real world. Furthermore, it is evidence based, as it identifies where possible shortcomings can be, where we can add information, collaborate and have interdisciplinary knowledge. Additionally, it is verifiable since interrater reliability can be used across systems, and it is evaluateable because it suggests measurable and actionable propositions in policies, which can be measured and as such keep government accountable. Moreover, using policy feedback, the evaluation of these propositions can improve policy through policy lessons that will advance its sustainability/longevity.

This article does not come without limitations. Firstly, IPA as a method does not consider the ‘off-the-page’ aspects of policy analysis such as data, the mood of analysts or if the implementation followed the policy to the letter. However, locating IPA within the larger QCA methods, the fact that CPA draws on various disciplines and the fact that the IPA provides CPA the means to use the policy structure as the lynchpin for policy analysis all mean that these conventional and existing methods can be used to fill that void. They will be building a stronger and more reliable alliance (to identify the gaps in policy structures and make sure they are measurable, as this will make evaluation and transparency easy).

This article also does not empirically test or compare policies; it is theoretical and grounded in a review of literature. Moreover, as proposed in the section above, an empirical approach would make findings more concrete and verify the strength of the ideas presented here.

Lastly, there is the problem of language, as it is clear that not all countries will have their policies written in English, which could serve as a limitation. However, the measuring of structure could be done using IPA in different languages. Additionally, most countries in the world use English in their academic institutions and publications, and thus translations could be done by language experts. This could help advance the IPA methodology and CPA.

In conclusion, echoing the words of Fuchs and Hofkirchner (2005), we have an ethical responsibility to develop new knowledge and new understandings to deal with issues. This exploration has opened a new pathway to improve how we develop policy and our ability to compare policies across systems, and essentially by extension improve the human condition.