The Measurement of Social Capital in America: A Reassessment.

One of the more successful approaches to the measurement of social capital across US counties relies on a two-step algorithm procedure. In the first step, ten variables accounting for the per capita number of various types of voluntary organizations are averaged to generate an Aggregate Index. In the second step, the Aggregate Index and three other factors are used to extract an overall Social Capital Index. Here, we propose several methodological improvements to this already solid methodology. We replace the Aggregate Index calculated as a simple average with a measure generated with principal component analysis, and subsequently with a formative partial least squares dimension-reduction procedure. We explore variations of these procedures, according to the rent-seeking nature of the organizations that make up our groupings. We illustrate our methodology by using US county data. We find that, even when holding the normative concept and the data constant we generate alternative metrics with different characteristics. This result has far-reaching implications for both the theory of social capital and the public policies that rely on the evidence surrounding social capital. There appears to be an inherent arbitrariness to measuring complex social phenomena using a reductionist analytical framework. At the same time, there are limits to evidence-based policy interventions. These limits need to be mitigated with a balanced approach relying on both analytical tools and qualitative evaluations.

Introduction

Social capital is a complex construct with tremendous implications for economic and social development (Putnam et al., 1994; Putnam, 2001b). There is a vast theoretical and methodological literature written on this subject. The measurement of social capital is another vast area of research. There are many approaches, some relying on aggregate indices, others on complex questionnaires and surveys (Engbers et al., 2017; Inkeles, 2001; Lin & Erickson, 2008; Paldam, 2000; Putnam, 2001a; Serra, 2001; van Oorschot et al., 2006).

In this paper, we propose a methodological improvement to the commonly used two-step procedure to estimating the social capital index, popularized by Rupasingha et al., (2000, 2006). According to the original method, the social capital index represents the First Principal Component extracted from four factors. In the first step, the first factor is aggregated by averaging ten variables representing ten different types of voluntary organizations. In the second step, the aggregated first factor thus calculated, together with three other factors, are used to derive the social capital index through Principal Component Analysis (PCA henceforth).

Our methodological improvement prescribes replacing the aggregation of the first factor with a metric obtained through PCA and/or a Partial Least Squares dimension-reduction procedure (PLS henceforth). The overall social capital index is then extracted through PCA and/or the same PLS dimension-reduction procedure used in the previous step. In addition, we allow for variations in the formative content of the social capital index, according to the rent-seeking nature of the ten groups of voluntary organizations. We end up with six alternative measures, some of them with slight variations in their formative content.

We find these measures to be statistically different form each other when analyzed form the perspective of their distribution characteristics, including spatial distribution. This result raises several interesting questions. First, we note that variations in data and differences in normative content are not the only necessary conditions for the derivation of metrics with different characteristics. Even when using the same data and the same normative content of the construct we end up with different results simply because we use different algorithms for extracting the index. This finding has far-reaching implications, not only for the study of social capital, but for every conceivable field of study relying on formative constructs and indices.

We contend that statistical differences might result in theoretical and practical implications for economics, and social sciences in general. If social capital is used both as a predictor and an outcome variable, we wonder about the differences in econometric results when using measures with different characteristics. We suspect that one reason for why various aspect surrounding social capital theory are still elusive can be traced back to this measurement issue. By alternating the metrics estimated here in the models used in the literature, we would most likely see variations in terms of explanatory power, significance level, and even direction of correlations.

We also wonder about the practical implications of using these measures from the perspective of public policies. Social capital represents a category that informs policy interventions in public health, transportation, education, housing, and many others. Different patterns of social capital distribution would suggest a different mix of target, mediation, and segmentation approaches. It would also recommend different levels of intervention: individual, community, or multilevel.

The main contribution of our paper consists in a methodological innovation and the interpretation of its implications. We propose alternative social capital metrics while holding constant the normative concept, the data, and the variables. We find these metrics to have different distribution characteristics. Hence, we can clearly show there are structural limits to the measurement of social capital when using analytical frameworks that are inherently reductionist. We believe our findings should settle the dispute among economists and practitioners with respect to the elusiveness of evidence. This elusiveness is not caused only by disputes around the theoretical content of the construct; nor is it due only to using vastly different datasets. When dealing with very complex social phenomena, any reductionist analytical framework will induce an irreducible arbitrariness in the measurement.

Based on our finding, we are also able to show there are limits to the use of evidence-based policy decisions. We clearly believe that effective policy interventions must be based on sound empirical evidence. At the same time, we caution against relying exclusively on analytical tools to inform policymaking. As always, a balanced approach is probably what works best. The analytical framework needs to be complemented by a thoughtful evaluation of qualitative aspects that give the social phenomena the aforementioned complexity.

Our paper is organized as follows. Next section overviews the theoretical background in extenso. Section three introduces the data and the methodology. Section four presents the results. Section five offers a detailed discussion and interpretation of results, and section six concludes.

Theoretical Background

In his groundbreaking work, Putnam (2001b), Putnam et al. (1994) contends that individuals' proclivity for association, bonding and trusting each other above and beyond family ties has important economic consequences.

The concept of social capital has gained significant currency in the last decades, yet it remains elusive and hard to define. It does not represent a directly observable variable that can be easily quantified and verified by third independent parties. There is an emerging consensus among scholars that social capital “is not one but many” (Paldam, 2000). It is indeed a latent variable with at least several distinct, mutually interacting dimensions: the propensity to form voluntary association, the intensity of bonding in social networks, the level of generalized trust, and the nature of social norms surrounding network interaction and trust (Bjørnskov, 2006).

A main idea in Putnam's worldview is that social capital appears stationary through time. Because it is rooted in cultural norms and traditions, it changes only slowly, on a historical timescale. From the perspective of contemporary political economy, social capital can be treated mostly as exogenous and primary to institutional development. In other words, trust is a determinant of institutional quality and not the other way around.

Many other scholars pursue this line of argumentation and contend that social norms and generalized trust stem, not from rational calculations and economic-style cost–benefit analysis, but from learned moral habits, handed down from generation to generation. Various nations have attained different levels of economic development mainly because they set out on the road to economic growth and modernization with different stocks of social capital (Fukuyama, 1995). Generalized trust represents at the same time the most important economic resource at our disposal and the moral basis of our society. When articulated in this way, it only seems fitting that nations that are morally superior are also the ones with the highest level of economic development (Uslaner, 2002).

Using a different perspective, Olson (1971) famously contends that economic growth is destabilizing and larger groups face more severe free-rider problems. From this perspective, social capital is more likely to manifest itself in the form of special interests who become entrenched and use public resources to their own private benefits (Fukuyama & Davis, 2014).

In the same vein, Bourdieu (1973) views social capital as a glorified instance of mutual back-scratching and self-dealing. Social capital is a positive sum game for the members of an interest groups at the expense of the larger community. Social capital is thus predatory in nature and perpetuates economic inequality and privilege.

Other scholars contend that social capital impose a leveling down effect on people’s individual potential, by constraining their aspirations and imposing the equivalent of a lowest common denominator (Harper, 2001; Ledeneva, 1998; Portes, 1998).

Using a sample of 38 countries Knack (1999) finds little evidence that the proclivity to associate leads to superior economic growth and explains this result in favor of Olson's organized special interests and against Putnam's social capital as a positive force. Similar results are obtained by Coates and Heckelman (2003).

There is, however, a wide consensus around the intangible nature of social capital. Unlike financial capital, social capital cannot be measured, cannot be stored, or saved, and has no definite locus. It represents a social and economic resource in a potential form, at both individual and communal levels (Au, 2019; Claridge, 2018). Social capital can be conceived as a relational asset that pays off in terms of information, influence, status, and reinforcement. The main difference between communal and individual levels of social capital has to do with the specific way in which resources are being mobilized (Lin, 1999, 2002).

Individuals invest in trust, relationships, networking, and others, but they do not have control over many aspects of the collective dynamic. Social norms (Coleman, 1988; Feld, 1981), networks (Erickson, 2009), and culture, including the level of social trust represent aspects associated with communal social capital, stemming from the investment in individual social capital. Communal social capital and its radius of trust are thus emerging from individual actions. Individuals are oftentimes motivated by narrow interests, such as upward social mobility, social alliances, and networking; but they also care about the greater good (Burt, 2000; Lin, 2001; Lin & Dumin, 1986).

A related, yet distinct set of concepts are represented by bonding and bridging social capital. Bonding as the embodiment of communal social capital refers to the strength of ties within a homogenous community. Bridging tends to be more inclusive because it establishes ties at the confluence between communities, stemming from contacts among individual social actors (Burt, 2000; Erickson, 1996; Frank & Yasumoto, 1998; Lin, 2001). In other words, bonding tends to foster closed networks and preserves existing resources, whereas bridging leads to open networks and the acquisition of new resources.

Because of all the above considerations, social capital displays the characteristics of both a private and public good. Investment is predominantly done at an individual level, and the return has an important individual component. Yet, another important and significant component accrues at the communal level and benefits the greater good (Alguezaui & Filieri, 2010; Bhandari & Yasunobu, 2009; Coleman, 1988; Dasgupta, 1999; Fukuyama, 2001; Kostova & Roth, 2003; Newton, 2001).

Last but not least, because it is generated by the patterns of community interaction, social capital has an important spatial dimension. Geography and demographics matter. Communities are influenced by and at the same time determine the patterns of social mobility and networks. (Deville et al., 2016; Diemer & Regan, 2022; Glückler et al., 2017).

As noted by many scholars, there is a dissonance between the rich conceptual framework and the operationalization of social capital (Markowska-Przybyla, 2012). The various dimensions and angles to the measurement of social capital give rise to an endogeneity problem, whereby one cannot clearly delineate cause from effect (Markowska-Przybyla, 2012; van Oorschot et al., 2006). In fact, one can safely assume there are countless feedback relationships among the countless concepts and constructs linked to social capital. In the words of Putnam (2001a, p. 137), who is often cited, all these dimensions are “as tangled as well-tossed spaghetti (Putnam, 2001b).

Most empirical studies employ the structure of the social capital categorization used by the Social Capital Benchmark Study (Rogers & Gardner 2015a), developed by a panel of scholars, and covering a wide range of conceptualizations. It has four categories: (1) social trust; (2) formal membership and group participation; (3) altruism; and (4) informal interaction among individuals, as reported by Saguaro Seminar in 2001 (Rogers & Gardner 2015b).

Many studies use variations of this structure: sometimes there is an additional dimension to account for trust (Engbers et al., 2017); or four dimensions are combined and compressed into three—networks, trust, and civism (van Oorschot et al., 2006). Yet, other studies, use a layered dimension structure—behavior, attitudes, populations—to which they add an extra dimension layer to account for the cultural context; for example: trust, integrity, truth, nurturing, support, and uplift, in the case of New Zealand (Spellerberg, 2001).

One of the more compelling and consistent social capital measurements in the United States has been put forth by the Department of Agricultural Economics, Sociology, and Education, at Penn State University (Rupasingha et al., 2006).1 This approach was used in a series of papers with significant academic impact. It has the advantage of addressing the measurement problem in the context of a nation that is culturally homogeneous. On the flip side, it cannot be extended to international studies without adjusting for significant variations in the cultural context.

Rupasingha and Goetz (2007) use an initial set of ten variables to measure the predilection for formal network, informal sociability, and civic engagement. A similar approach is used by Rupasingha et al. (2000) to determine that social capital brings an additive quality to economic growth. In a related paper, Rupasingha et al. (2006) use the same measurement approach in order to investigate the relationship among education, inequality and the production of social capital. Because of presumed endogeneity, education and inequality are in fact treated as additional dimensions to social capital. Last but not least, building on the same methodology, Goetz and Rupasingha (2006) find that the presence of Wal Mart stores has the tendency of depressing the stock of social capital in local communities, measured at a county-level.

We believe the methodology developed by Anil Rupasingha, Stephan J. Goetz, and David Freshwater (RGF henceforth) to be among the most pertinent to date. It has several important qualities: it covers the dimensions that are common to almost all definitions of social capital endorsed in economics and sociology; it captures aspects that are both at the individual and community level; and it captures characteristics of both bonding and bridging social capital. At the same time, it is among the most parsimonious to date. The steps involved in its estimation are among the least onerous.

Other measurement methodologies based on massive surveys and exhaustive census data are unwieldy and hard to use effectively; moreover, they cover so many aspects that it is hard to tell in the end what they are measuring, unless we define social capital as an embedded, overarching quality common to all and every type of social and economic networked interaction. In our opinion, the ambition to develop an all-encompassing definition of social capital -one that tries to account for everything conceivably related to this notion—might result in metrics that lack discriminant validity.

We set out to refine and improve an already solid social capital index by addressing two minor methodological aspects. We question the wisdom of using a simple average of ten variables to derive the first factor used to calculate the overall social capital index; and, we contend the extraction of the First Principal Component might not be the best choice given the correlations among the four factors considered in the original methodology: simple, per capita aggregate of ten variables measuring membership in various organizations, census response data, the number of tax-exempt not-for-profit organizations, and voter turnout in presidential elections.

As it is well known, mean estimates tend to be sensitive to outliers and are not appropriate when dealing with skewed distributions. In the original RGF estimation of the aggregate mean, there is no explanation for why the average is equally weighted. We infer the authors consider by default all types of organization membership to be equally important in capturing networks and sociability. While there is a certain dose of arbitrariness in the default solution to aggregating the data, we have no philosophical objection if there is no better alternative. After all, when in doubt about the weighing of each of the ten variables, equal weights are as good as any other weighting scheme. There are, however, theoretical considerations suggesting not all organizations contribute in the same way to the aggregation of social capital. At the very least, one should consider the insights of Olsen (1971) and Knack (1999) with respect to the rent-seeking nature of certain organizations. We deal with this aspect in the methodology section by allowing our index to vary with the nature and type of the organizations.

Nonetheless, we contend one can improve the measurement process by replacing the aggregate mean of the ten variables with an index obtained using a dimension reduction approach. This approach would replace all ten variables with only one that captures the maximum common variance possible from the original variables. We propose two solutions: the First Principal Component, derived with PCA; and a metric extracted following a formative, PLS dimension-reduction procedure, implemented in WARP-PLS.

Principal Component Analysis represents a dimension-reduction technique used in econometrics, but also in machine learning. Complex phenomena have various aspects and dimensions, oftentimes correlated among themselves. Because it is difficult to handle too many variables at once and make sense of too many inter-related facets of the same phenomena, scholars prefer to compress, or reduce the numbers of variables from many to only a few uber-variables called factors. PCA generates a hierarchy of factors that best capture the common variability in the data (Jolliffe, 2005; Rencher, 2002).

Partial Least Squares is a dimension-reduction procedure that is trying to achieve the same aim as PCA but employs a different algorithm. It uses a linear regression model to project both observable and predicted variables into a new space, unlike PCA, which is estimating orthogonal hyperplanes of maximum variance. Because the variables are structured in two matrices corresponding to the two sets of latent variables, PLS is also known as projection to latent structures. PLS is considered to be somewhat more sophisticated than PCA because it is capable of capturing non-linear relationships among the latent constructs (Wold et al., 1984).

We use these replacements to generate alternative estimations to RGF’s overall Social Capital Index (SK Index henceforth). When using PCA to derive the first factor, we also use the same approach to extract the social capital index. When using the formative, non-linear PLS dimension-reduction procedure to extract the first factor, we also use the same procedure to extract the social capital index.

Materials and Methods

In most of their studies, RGF use the variables provided by the Department of Agricultural Economics, Sociology, and Education (DAESE henceforth) at Penn State University. The data pertains exclusively to the United States and is available at the county-level for the year 2014.2 There are ten groups (variables) measuring membership in various organization: (1) Number of establishments in religious organizations (NAICS 813110) (2) Number of establishments in civic and social associations (NAICS 813410) (3) Number of establishments in business associations (NAICS 813910) (4) Number of establishments in political organizations (NAICS 813940) (5) Number of establishments in professional organizations (NAICS 813920) (6) Number of establishments in labor organization (NAICS 813930) (7) Number of establishments in bowling center (NAICS 713950) (8) Number of establishments in fitness and recreational sports centers (NAICS 713940) (9) Number of establishments in golf courses and country clubs (NAICS 713910) (10) Number of establishments in sports teams and clubs (NAICS 711211). All ten variables are adjusted on a per capita basis times 10,000.

RGF aggregate these ten variables into a simple, equally weighted average. This weighted average represents one of the four factors used to estimate the social capital index (SK Index). The other three factors are voter turnout, the proportion of census respondents, and the total number of non-profit organizations, all adjusted on a capita basis. SK Index represents a formative outcome estimated with Principal Component Analysis. The approach used to calculate SK Index is illustrated in Fig. 1.

Fig. 1

The original approach used by RGF to estimate the Social Capital Index (SK Index). The Aggregate Index is calculated as a simple weighted average of ten variables representing ten types of organizations. SK Index is extracted as the First Principal Component using four factors

As mentioned earlier, the theoretical constructs related to social capital are silent with respect to the relative importance of each type of organization. Since there is no a priori theoretical reason to weigh the Aggregate Index, a simple average is as good as any other measure. In principle, one has no reason to doubt this approach if the distribution of the data allows it.

The simple average is a good description of central tendency provided the distribution is not too skewed and there are relatively few high impact outliers. Unfortunately, this is not the case here. Take for example the first observation in our data set corresponding to Autauga County, Alabama (FIPS 1001). The simple average is 1.37457, yet nine out of ten variables are below the average (Fig. 2). The average is clearly driven by the first variable (the number of religious organizations) that has a value of 9.585820. The skewness of the distribution corresponding to this first observation is 2.57 and the kurtosis equals 7.8048.

Fig. 2

The distribution of ten different types of organizations for Autauga County, Alabama in 2014 (adjusted on a per capita basis times 10,000). Each dot corresponds to each of the ten types of organizations. The red line represents the simple average

The vast majority of observations display this pattern of skewness: the simple average is overwhelmingly driven by outliers. We do not present skewness and kurtosis results for each observation. Nonetheless, we calculate instead skewness and kurtosis for each observation and plot them against the 3141 counties in our dataset (Fig. 3).

Fig. 3

Plot of skewness and kurtosis for the ten-variable distribution and for the 3141 US counties in the data set

Anscombe-Glynn, D’Agostino and other parametric tests for statistical significance cannot handle a distribution with only ten data points. However, the most conservative stance on statistical significance posits that a skewness larger than two and a kurtosis larger than five are deemed abnormal. In our dataset, there are 2786 observations with a skewness larger than two and 2955 observations with a kurtosis larger than five. There is an almost complete overlap between the observations with a skewness larger than two, and a kurtosis larger than five. Clearly, the simple average, as estimated by RGF does not represent a reliable measure of central tendency for most of our data. An Aggregate Index calculated as a simple average is therefore suspicious to say the least.

We proceed with replacing the method used by RGF with two alternative solutions, using PCA and PLS. Regardless of which methodology we use, the initial step requires a formative estimation of the first factor. Our first solution extracts the First Principal Component out of the ten variables discussed earlier to obtain a Composite Index (COMPIndex henceforth) as a replacement for RGF’s Aggregate Index. Our COMPIndex is a formative measure because we define it as a metric that captures the largest portion of the variance common to the ten dimensions under analysis. In the same vein, our second solution employs a PLS dimension-reduction procedure implemented in WARP-PLS to extract a PLSIndex, also from the ten original variables discussed earlier.

Social capital has a significant spatial dimension as well, as revealed by a small but compelling line of research (Forrest & Kearns, 2001; Glaeser & Redlick, 2008; Ignjatović & Tomanović, 2011; Middleton et al., 2005; Rutten et al., 2010). We also perform a spatial analysis to compare the degree of spatial autocorrelation captured by the different variants of our social capital metrics. To this end, two standard measures of spatial dependence are employed, namely the Moran's I and the Local Indicators for Spatial Association.

Moran's I (Moran, 1950) is a global indicator of spatial dependence, computed as follows:where xi and xj represent the social capital index in counties i and j respectively, is the average value of the index, and wij are spatial weights capturing the interdependence between county j and county i. We use a first-order queen contiguity spatial weights matrix, i.e. wij = 1 if counties i and j are neighbors, and wij = 0 otherwise. A permutation test is applied to validate the statistical significance of the results (Anselin & Rey, 1991).

Given the existence of spatial heterogeneity, the degree of autocorrelation estimated by global indicators can vary significantly among the territorial units in a geographic space, therefore we need indicators that measure local spatial autocorrelation as well. The Local Indicators for Spatial Association (LISA) provide local statistics that gauge spatial dependence between each individual county and its geographic neighbors, revealing local clusters of similar values (Anselin, 1995). They allow for a better understanding of the spatial variations in the interactions between individual counties. LISA statistics are computed as:

Results

Because the First Principal Component and the formative PLS aggregate are notoriously difficult to interpret, we limit the discussion of the results to noting that distributions have different means, standard deviations, skewness, and kurtosis. None of them are normal, including RGF’s original Aggregate Index. Summary statistics are presented in Table 1. In addition to the information presented in Table 1, we found that COMPIndex captures 82.75% of the common variance of the ten variables. The corresponding eigenvalue is 28.71.

Table 1

Side-by-side comparison of RGF’s Aggregate Index, COMPIndex, and PLSIndex

	FGR’s original Aggregate Index	COMPIndex	PLSIndex
Mean	1.38	0.00	− 2.5470e−06
Median	1.26	− 0.92	− 0.23
Minimum	0.00	− 9.73	− 1.63
Maximum	6.89	37.93	12.52
Standard deviation	0.70	5.35	1.00
Coefficient_of variation	0.51	7.72e+14	3.9263e+05
Skewness	1.83	1.66	2.74
Kurtosis	6.82	5.26	16.30

RGF’s Aggregate index is calculated as a simple average of ten variables, COMPIndex is calculated as the First Principal Component of ten variables, and PLSIndex is calculated using a non-linear, dimension-reduction procedure in WARP-PLS

A visual illustration of the differences in the distribution characteristics is provided in Fig. 4. A cursory inspection reveals the shapes of the distributions to be different, as suggested by the parameters of the distributions.

Fig. 4

Density plot comparison of RGF’s Aggregate Index, COMPIndex, and PLSIndex. RGF’s Aggregate index is calculated as a simple average of ten variables. COMPIndex is calculated as the First Principal Component of ten variables. PLSIndex is calculated using a non-linear PLS dimension-reduction procedure in WARP-PLS

We also vary the number of variables used in extracting the first factor, based on (Knack, 1999) who distinguishes between Olsen-type and Putnam-type organizations. Olsen-type organizations are rent-seeking. Their members collude to reallocate resources in their favor and acquire the bargaining power required to extract various benefits. Professional associations are good examples of Olsen-type organizations. Their members have strong incentives to use collective bargaining, lobbying, and various pressure tactics in order to bolster pricing power, obtain government subsidies and grants, or favorable regulation. In our dataset the 3rd, 4th, 5th, and 6th types of associations represent Olsen-type groups. It is not hard to see that business, political, professional, and labor organizations aim to promote and foster the interests of their members in various ways.

Putnam-type organizations are not rent seeking. They foster cooperation, social bonding, and social interaction for their own sake, for the sake of advancing social causes, or to provide social goods. Civic associations boast objectives that range from promoting equal opportunities for the LGBT community, to those who extend aid to homeless people, or to the victims of war. Bowling centers—of Putnam fame—aim to informally bring together neighborhoods or communities for leisure and socializing. Religious organizations promote social bonding and represent a source of spiritual comfort among those who share common beliefs and embrace similar worldviews.

We estimate two additional COMPIndex, and two additional PLSIndex variations following (Knack, 1999) insight into Olsen-type and Putnam-type organizations. We call them COMPIndex-OLSEN and COMPIndex-PUTNAM. We show summary statistics in Table 2. Although there is no straight-forward economic interpretation of the results, we note again differences in the characteristics of their distributions. These differences were nevertheless expected because we varied the formative content of the constructs.

Table 2

Summary statistics for alternative formative estimations of the OLSEN and PUTNAM-type first factors

	COMPIndex-Olsen (var. 3, 4, 5, 6)	COMPIndex-Putnam (var. 1, 2, 7, 8, 9, 10)	PLSIndex-OLSEN (var. 3, 4, 5, 6)	PLSIndex-PUTNAM (var. 1, 2, 7, 8, 9, 10)
First Principal Component—proportion of total variance captured	0.7534	0.8738	–	–
Eigenvalue	1.51	28.56	–	–
Mean	− 3.52e−16	6.3e−15	3.88e−05	− 2.87e−06
Median	0.33	− 0.91	− 0.22	− 0.21
Minimum	− 15.22	− 9.65	− 0.70	− 1.55
Maximum	0.99	37.81	14.78	15.79
Standard deviation	1.23	5.34	0.999	0.999
Coefficient of variation	3.48e+15	8.48e+14	25,746.01	3.49e+05
Skewness	− 4.72	1.65	6.67	3.34
Kurtosis	35.25	5.23	70.01	27.80

COMPIndex-OLSEN and COMPIndex-Putnam are estimated as the First Principal Component, whereas PLSIndex-OLSEN and PLSIndex-PUTNAM are estimated using a non-linear, PLS dimension-reduction procedure

None of these alternative calculations are normally distributed, including RGF’s original Aggregate Index. The results of the normality tests are significant, but they are not reported here. All pairwise tests performed using these distributions show significant differences among each and every pair. Because we have concerns related to the validity of t-tests, we only report Wilcoxon signed rank test results (Table 3).

Table 3

Wilcoxon signed rank test results for pairs of alternative estimators of the first SK factor

	COMPIndex (all 10 groups)	COMPIndex-OLSEN (groups 3, 4, 5, 6)	COMPIndex-PUTNAM (groups 1, 2, 7, 8, 9, 10)	PLSIndex (all 10 groups)	PLSIndex-OLSEN (groups 3, 4, 5, 6)	PLSIndex-PUTNAM (groups 1, 2, 7, 8, 9, 10)
FGR’s Aggregate Index	Z = 126.59 p < 0.001	Z = 264.76 p < 0.001	Z = 126.60 p < 0.001	Z = 276 p < 0.001	Z = 263.56 p < 0.001	Z = 274.80 p < 0.001
COMPIndex (All 10 variables)		Z = − 42.40 p < 0.001	Z = − 82.23 p < 0.001	Z = − 36.67 p < 0.001	Z = − 36.47 p < 0.001	Z = − 37.48 p < 0.001
COMPIndex-Olsen (Variables 3, 4, 5, 6)			Z = − 42.17 p < 0.001	Z = 70.66 p < 0.001	Z = 83.82 p < 0.001	Z = 66.49 p < 0.001
COMPIndex-Putnam (Variables (1, 2, 7, 8, 9, 10)				Z = − 36.37 p < 0.001	Z = − 33.70 p < 0.001	Z = − 37.18 p < 0.001
PLSIndex					Z = − 19.52 p < 0.001	Z = − 67.06 p < 0.001
PLSIndex-OLSEN (Variables 3, 4, 5, 6)						Z = − 2.85 p = 0.002

In the second step, our overall social capital index calculation follows FGR’s method, but replaces the Aggregate Index with the alternative metrics discussed above. The variation in methodology is illustrated in Figs. 5 and 6.

Fig. 5

Alternative estimation of the overall Social Capital Index. COMPIndex is calculated as the First Principal Component of ten variables representing ten types of organizations. SK COMPIndex is extracted as the First Principal Component using four factors

Fig. 6

Alternative estimation of the overall Social Capital Index. PLSIndex is calculated using a formative, PLS dimension-reduction procedure, using ten variables representing ten types of organizations. SK PLSIndex is extracted using a formative, PLS dimension-reduction procedure applied to four factors

The overall social capital index, our SK COMPIndex, is extracted as the First Principal Component of four factors, and our SK PLSIndex is extracted following a formative non-linear PLS procedure implemented in WARP-PLS, using the same four factors. If we consider the variations in the calculation of the first factor following (Knack, 1999), we end up with six alternative calculations of the overall social capital index. A side-by-side comparison of summary statistics is presented in Table 4.

Table 4

Summary statistics for alternative formative estimations of the Social Capital Index extracted from four variables

	RGF’s original SK index	SK-COMPIndex	SK-COMPIndex-OLSEN	SK-COMPIndex-PUTNAM	SK-PLSIndex	SK-PLSIndex-OLSEN	SK-PLSIndex-PUTNAM
First Principal Component—proportion of total variance captured	0.3	0.31	0.32	0.31	−	−	−
Eigenvalue	1.2	1.25	1.26	1.25	−	−	−
Mean	− 3.15e−06	− 2.12e−14	3.17e−14	2.13e−14	1.59e−06	− 4.77e−06	2.55e−06
Median	− 0.23	0.04	0.15	0.04	0.00	− 0.05	0.06
Minimum	− 3.18	− 19.06	− 8.08	− 19.10	− 3.47	− 3.03	− 3.90
Maximum	21.81	4.75	12.93	4.66	14.94	13.33	7.49
Standard deviation	1.26	1.12	1.12	1.12	1.00	1.00	1.00
Coefficient of variation	4.00e+05	5.28e+13	3.54e+13	5.25e+13	6.28e+05	2.09e+05	3.93e+05
Skewness	2.88	− 3.07	− 0.19	− 3.11	1.8102	2.1205	− 0.11
Kurtosis	31.62	37.88	10.68	38.263	21.558	18.469	1.6696

Just as before, the results show none of the alternative estimations of the overall social capital index to be normally distributed (the individual test results are not presented here). Just as before, all pairwise tests performed on the resulting distributions show significant differences among each pair. Because we have concerns related to the validity of t-tests, we only report Wilcoxon signed rank test results in Table 5. There is one notable exception: The Wilcoxon test statistic is not significant in the case of the pair SK COMPIndex and SK-COMPIndex-OLSEN. We cannot reject the null hypothesis that the two population mean ranks (locations) are similar.

Table 5

Wilcoxon signed rank test results for pairs of alternative SK estimation methods

	SK-COMPIndex	SK-COMPIndex-OLSEN	SK-COMPIndex-PUTNAM	SK-PLSIndex	SK-PLSIndex-OLSEN	SK-PLSIndex-PUTNAM
RGF’s original SK index	Z = − 18.11 p < 0.001	Z = − 36.88 p < 0.001	Z = − 18.25 p < 0.001	Z = − 29.93 p < 0.001	Z = − 25.17 p < 0.001	Z = − 29.49 p < 0.001
SK-COMPIndex		Z = 1.15 p = 0.125	Z = − 62.30 p < 0.001	Z = 14.79 p < 0.001	Z = 24.45 p < 0.001	Z = 7.73 p < 0.001
SK-COMPIndex-OLSEN			Z = − 37.38 p < 0.001	Z = 58.53 p < 0.001	Z = 73.55 p =  < 0.001	Z = 40.53 p =  < 0.001
SK-COMPIndex-PUTNAM				Z = 15.04 p < 0.001	Z = 24.66 p < 0.001	Z = 7.92 p < 0.001
SK-PLSIndex					Z = 70.00 p < 0.001	Z = 21.27 p < 0.001
SK-PLSIndex-OLSEN						Z = 28.52 p < 0.001

Our alternative estimations of the overall Social Capital Index differ from a spatial perspective as well. Both the global (Table 6) and the local (“Appendix”) measures of spatial dependence reveal significant dissimilarity in the results pertaining to the social capital metrics and their variations.

Table 6

Global spatial autocorrelation statistics for alternative SK estimation methods

	Moran's I	z-value	Pseudo p-value
RGF’s original SK index	0.501	48.829	0.0010
SK-COMPIndex	0.517	47.578	0.0010
SK-COMPIndex-OLSEN	0.522	47.927	0.0010
SK-COMPIndex-PUTNAM	0.516	47.524	0.0010
SK-PLSIndex	0.346	32.768	0.0010
SK-PLSIndex-OLSEN	0.338	31.665	0.0010
SK-PLSIndex-PUTNAM	0.318	29.753	0.0010

The Moran's I statistics in Table 6 show high and statistically significant global spatial autocorrelation for all variants of social capital index, with the biggest values for Olsen variant of the SK index based on PCA. RGF’s original SK index also displays a high value for the Moran's I, while the PLS based indices are modest. It seems that PCA performs better than PLS in terms of capturing spatial dependence in terms of social capital indicators. What drives this high degree of spatial interaction among US counties in social capital influence? The main factor of influence is most likely population mobility, favored by geographical proximity, leading to direct social interactions and the creation of social networks (Deville et al., 2016; Diemer & Regan, 2022; Glückler et al., 2017).

The Local Indicators for Spatial Association (LISA) presented in “Appendix” also suggest significant dissimilarity between the alternative computation methods used for the social capital index. The maps in “Appendix” show large geographical clusters of similar values for the social capital index, either big ones (colored in red) or small ones (in blue). Differences occur not only between the main alternative computation methods, but also between their sub-variants (Olsen vs. Putnam). For instance, the SK COMPIndex-OLSEN reveals large clusters of low social capital for all of Alaska and an important part of the Southern US, while the SK COMPIndex-PUTMAN shows high social capital for most of Alaska, the South and the Midwest.

Discussion and Implications

Up to this point, our main contribution consists of reconsidering RGF’s estimation of the first factor entering the final calculation of the overall Social Capital Index by using two alternative approaches, each one with three variations. COMPIndex is extracted as the First Principal Component of the ten original variables considered by FGR. PLSIndex is extracted following a formative, PLS dimension-reduction procedure implemented in WARP-PLS. In addition, we also obtain: COMPIndex-OLSEN, COMPIndex-PUTNAM, PLSIndex-OLSEN, and PLSIndex-PUTNAM by varying the number of groups that enter the calculation of the first factor. In the second step, the first factor is used together with three additional factors in order to extract the overall social capital index. We end up with two main measures, each one with three variations: SK COMPIndex, SK COMPIndex-OLSEN, SK COMPIndex-Putnam, SK PLSIndex, SK PLSIndex-OLSEN, and SK PLSIndex-PUTNAM.

We pair and compare our metrics using non-parametric Wilcoxon signed rank tests and conclude they are quite different. We also use spatial analysis to complement our non-parametric tests and, as expected, we find that the dissimilarities among the visual maps associated with alternative social capital calculations are very striking. Although each metric is constructed using precisely the same data, the results are different, depending on the characteristics of the formative procedure. Clearly, the differences in results are statistically significant, but how significant are these differences from an economic perspective?

It is reasonable to expect the social capital index derived from OLSEN-type organizations might be different from that derived from PUTNAM-type organizations. They purport to measure things that are conceptually different: rent-seeking organizations generate a different type of social capital than non-rent-seeking ones. And both might differ from the metrics that capture both rent-seeking and non-rent-seeking organizations. It only makes sense to expect differences in outcomes when the formative content of the construct changes.

But what about differences among metrics that have the same formative construct, but are estimated using different algorithms? How different are the indices extracted with PCA from those extracted with PLS? How different is SK COMPIndex from SK PLSIndex? We know they are statistically different, and there are good reasons why one might prefer one to the other, depending on circumstances.

The goal of PCA is to reduce the number of dimensions and to eliminate redundancy in the data. PCA makes a lot of sense when dealing with many variables and/or when there is a high correlation present among the variables. Neither situation is present in our data, however, so at first glance, the use of PCA is not warranted (Table 7).

Table 7

Correlation matrix of four factors used by DAESE at Penn State to estimate a Social Capital Index (SK) as the First Principal Component

	1. Aggregate	2 Voter turnout	3. Census resp.	4.Non-profit org
1_Aggregate	1
2_Voter turnout	0.1173	1
3_Census resp	0.0153	0.1251	1
4_Non-profit org	− 0.1611	0.0541	0.1156	1

The data shown here is for the year 2014

On the other hand, our social capital index is the outcome of a formative approach, hence the lack of numerous, highly correlated dimensions should not be a relevant concern. Some authors consider that PLS estimators are superior to those extracted from PCA, because they boast superior class separation and, hence, better discriminant ability. While PCA requires numerous dimensions, PLS can generate comparable results with fewer of them. This is not to say PCA is always outclassed: it appears PLS has a nagging tendency to find groupings where there is no definite class structure (Chen, 2016; Kemsley, 1996).

It is for these reasons that we use both methods. We consider there are no compelling ex-ante statistical arguments consistently ranking one procedure above the other. We prefer to employ both procedures and compare the results. In the end, the significance of our results should be driven by differences in their theoretical and practical implications.

Social capital metrics can be used in several ways. They can be used as predictors and/or outcomes in econometric models to advance economic and social theory, that is, to help us understand the relationships among an entire range of variables and constructs. As already noted, Rupasingha and Goetz (2007) use the social capital index as a predictor of poverty rates across the United States. Rupasingha et al. (2000) investigate the effect of social capital on economic growth. Rupasingha et al. (2006) treat social capital and other economic variables as endogenous, hence social capital is modelled, not only as a predictor, but as an outcome as well. In the same vein, Goetz and Rupasingha (2006) find social capital to be the outcome of a changing economic and social dynamic.

To better understand the implications of different metrics, we should be able to test their predictive power side-by-side. This represents a momentous task, going far beyond the purpose and the scope of this current paper. There are tens if not hundreds of economic studies using social capital measures as either explanatory, or outcome variables, or both. A truly meaningful research project would have to identify the most consequential papers and substitute the original measures with the ones proposed here. The natural starting point would be the papers cited here, that is, the original studies conducted by RGF. Our social capital metrics estimation follows a very similar, yet improved, two-step procedure. If the results of testing our metrics using the same data employed by RGF yields different results, one would have produced evidence these methodological variations generate—not only statistically different metrics—but also different economic implications.

We speculate the relationship among social capital, income, poverty, economic growth, and education—all explored in the studies mentioned earlier—would vary as a function of the social capital measure employed. At best, we would probably see minor changes in the explanation power of the models, the significance of coefficients, the strength of the relationships, and the magnitude of effect sizes. At worst, we would obtain radically different outcomes, including but not limited to the direction of the relationship, the significance of coefficients, and the fit of the overall model.

The theoretical literature on social capital is indeed pointing out the ambiguity and inconsistency of the evidence. There is no full agreement with respect to the true nature of the relationship between social capital and economic growth, as already discussed in the first part of the paper (Bourdieu, 1973; Fukuyama, 1995; Markowska-Przybyla, 2012; Portes, 1998; Putnam, 2001a, 2001b; Uslaner, 2002). But there is also a great deal of haziness in understanding the relationship among social capital, education, mobility patterns, and health status, among many others. No doubt, this is the due to the complex nature of the construct (Fukuyama, 2001; Paldam, 2000); but it is also due to differences on how the concept is measured (Dasgupta, 2005; Stone, 2001). Our research shows—above and beyond the extant evidence—the extreme sensitivity of the outcome to the measurement method even when there is agreement with respect to the theoretical construct and its operationalization.

Another consideration pertains to the usefulness of social capital measures in the context of public policy decisions. Social capital clearly represents a policy resource (Montgomery, 2001).

The relevant question is whether the statistical differences observed among the alternative measures proposed by our research would translate into consequential differences from the practical point of view of policymakers. In other words, would they lead to different decisions?

The answer is complex and requires a good understanding of the policy intervention framework. Social capital is either a target, mediator, or the segmenting dimension of a public policy intervention. Some policies aim at increasing and strengthening social capital; others rely on existing ties and patterns of community cohesion as transmission belts to achieve their goals; yet other policies focus on sub-groups or categories to induce beneficial and transformative change. Interventions can occur at the individual level, at the community level, or both (Moore & Kawachi, 2017). Moreover, most interventions represent in fact multiple combinations of the above typology. For example, KidsFirst, an early childhood intervention program initiated to promote healthy development for children in Canada was deemed both a target- and channel-type intervention, at individual and community level (Villalonga-Olives et al., 2018).

There are innumerable aspects of public policy interventions that follow the same framework, including, but not limited to public housing, government accountability, public health, transportation, fiscal and financial policy, immigration policy, and many others (Karlan, 2005; Knack, 1999; Lang & Hornburg, 1998).

The relationship between public health and social capital appears significant, although the finer mechanisms are not always well understood (Ferlander, 2007). It has been recognized that the discrepancy in responses to public health interventions across various communities in the US and elsewhere is due to different patterns of social interaction. Nowhere was the importance of understanding and measuring social capital more noticeable than during the recent COVID-19 pandemic (Alfano, 2022; Wong & Kohler, 2020). Different regional distributions of social capital and rates of urbanization clearly accounted for variations in vaccination uptake across many countries (Qiao et al., 2022).

Just like in public health, public transportation policies are consequential because there is a link between social capital and physical mobility (Currie & Stanley, 2008; Urry, 2002)]. Public transport can render a city “livable,” and thus impacts and is itself impacted by the type of community (Vuchic, 2017). Social capital is at the same time an indicator of community needs and community resources (Stone, 2001). Different patterns of networks and community resources when combined with different mobility paradigms yield various levels of social well-being (Preston & Rajé, 2007). The salient differences in public transportation among American, European, and Asian cities are correlated with a combination of social capital and public policy interventions (Axhausen, 2006; Currie & Stanley, 2008; Pucher, 2002).

The same goes for housing policy and urban planning. The success of these interventions is what makes cities livable and addresses economic and social inequalities. These policies rely on understanding the patterns of networking, civic engagement, and community resources. The accurate gauging of social capital patterns can make or break communities (Lang & Hornburg, 1998; Rohe & Stewart, 1996; Spence, 1993; Witte, 1996).

Scholars have duly noted the elusive nature of the evidence linking social capital to public policy interventions (Shiell et al., 2020). We contend this is just a reflection of a similar type of elusiveness that pervades the evidence in the theoretical literature on social capital, as discussed earlier. We have shown here that there is an inherent arbitrariness in social capital measurement even when maintaining a consistent and rigorous approach to concepts, data, and variables.

If we refer to the maps in the “Appendix” showing the spatial distribution of social capital across US counties, we notice striking differences in social capital patterns. The American policymaker trying to design a policy intervention—be it in public health, transportation, housing, urbanism, or education—is faced with a tough dilemma. One must choose the social capital distribution pattern on which to base the intervention. Settling for the one generated by the SK PLSIndex would have different implications than the one generated by the SK COMPIndex. One might need a certain mix of target, channel, and segmentation, at the individual or community level for the former, and a significantly different mix for the latter. Most likely, the response to implementing these policies would also vary greatly, as documented in the extant literature on public policy interventions.

This dilemma appears irreducible, in the sense that there are structural limits to how accurately one can measure a complex construct, such as social capital. We contend that the only way forward is to combine quantitative with qualitative analysis as a basis for policy decision-making. There are clear limits even to the most sophisticated and savvy analytical framework that can only be mitigated with a thoughtful, common-sense approach. Evidence-based public policy decisions need to strike a healthy balance between all aspects of the matter at hand. A purely analytical framework, devoid of contextual and pragmatic assessment can become as arbitrary as a game of chance.

Concluding Remarks

We set out to improve the measurement of social capital following a two-step algorithm procedure commonly used in the econometric literature. In the first step, based on ten variables, we use Principal Component Analysis and/or a formative, Partial Least Squares dimension-reduction procedure to extract a first factor. This factor, together with three other factors are used to extract the overall social capital index, also using PCA and/or PLS procedures. In addition, we propose variations in the structure of the formative first factor, according to the rent-seeking nature of the ten variables. We produce six alternative metrics. We find these metrics to be statistically different in their distribution characteristics, including spatial distribution.

The most unsettling aspect is that differences in our results do not stem from using different datasets and/or variables. Even when controlling for data and formative content, we end up with significant discrepancies due to variations in the procedure employed to extract the social capital index. Due to obvious constraints, we cannot test the econometric predictive power of our results. This task would require an exhaustive replication of many successful studies conducted to date. We argue, however, that the result of many empirical studies would become even more muddled because of using our alternative social capital metrics. The significance levels of coefficients, the strength and direction or correlations, the overall explanatory power of econometric models would change when alternating the indices discussed above.

It is also unclear the extent to which various measurement algorithms would lead to different public policy decisions. We contend, that different metrics of social capital would determine different public policy interventions strategies, in which the mix of targets, channels, and segmentation variables might be markedly different. Some metrics might recommend mostly individual-level interventions. Other metrics might suggest interventions at the community level. This might explain why, after more than two years of global pandemic, scholars and practitioners still do not fully understand the discrepancies in the response to public health measures across various countries and continents.

Two summarize, the significance of this paper is twofold. First, we show that there is a structural arbitrariness in measuring complex constructs, such as social capital. This happens despite following a rigorous and consistent methodology, controlling for data and variables. Beyond a certain level, the accurate gauging of the multi-faceted dynamic of social capital using a reductionist analytical framework is simply not possible. One must contend with an irreducible level of elusiveness.

Second, based on our findings, we argue there are limits to evidence-based public policy interventions. For the same reasons the theory cannot be entirely clear on the finer mechanisms linking social capital to other variables, public policy cannot rely entirely on the result of quantitative measurements. The solution is most likely represented by an approach in which the limits of the analytical framework are balanced by qualitative knowledge, common-sense, experience, and empathy.