Validity of polygenic risk scores: are we measuring what we think we are?

Polygenic risk scores (PRSs) have become the standard for quantifying genetic liability in the prediction of disease risks. PRSs are generally constructed as weighted sum scores of risk alleles using effect sizes from genome-wide association studies as their weights. The construction of PRSs is being improved with more appropriate selection of independent single-nucleotide polymorphisms (SNPs) and optimized estimation of their weights but is rarely reflected upon from a theoretical perspective, focusing on the validity of the risk score. Borrowing from psychometrics, this paper discusses the validity of PRSs and introduces the three main types of validity that are considered in the evaluation of tests and measurements: construct, content, and criterion validity. This introduction is followed by a discussion of three topics that challenge the validity of PRS, namely, their claimed independence of clinical risk factors, the consequences of relaxing SNP inclusion thresholds and the selection of SNP weights. This discussion of the validity of PRS reminds us that we need to keep questioning if weighted sums of risk alleles are measuring what we think they are in the various scenarios in which PRSs are used and that we need to keep exploring alternative modeling strategies that might better reflect the underlying biological pathways.

Introduction

Polygenic risk scores (PRSs) aim to quantify the genetic liability of common diseases and traits, the collective of genetic factors that contribute to their development (1). PRSs are typically calculated as a weighted sum of the risk alleles of single-nucleotide polymorphisms (SNPs) and investigated for their potential to improve the prediction of common diseases in clinical care to guide preventive and therapeutic interventions (2).

While the concept of polygenic inheritance is centuries old and long lacked data to prove its merit, the calculation of risk scores developed from an empirical tradition with little attention for its theoretical foundation (3). In the early days of genome-wide association studies (GWASs), researchers considered their few newly identified SNPs as separate variables in the prediction of disease risks (4,5), and PRSs were a practical solution to include larger numbers of variants in the regression analyses (6). Some early studies calculated unweighted risk scores that summed the number of risk alleles, assuming a similar impact on disease risk for all SNPs, but these were rapidly replaced by weighted scores that acknowledge that some SNPs have stronger effect than others (6). In recent years, the construction of PRSs is being improved from a computational perspective, with proposals for a more liberal selection of independent SNPs and a more refined estimation of their weights (7,8).

Even though PRSs typically explain only a small proportion of the genotypic variance (2,9,10), the validity of the PRS as a measurement of polygenic predisposition remains largely undiscussed (3). It may be that researchers are aware and accept that the validity is imperfect as many more genetic associations remain to be identified (2), all models have limitations, and they can be useful even when imperfect. It may also be that researchers feel no reason to question the validity of the weighted sums of risk alleles as (1) the risk distributions are what Fisher predicted for a large number of variants with weak effects (11,12), (2) there seems to be no evidence for strong gene–gene interaction (12) and (3) these PRSs generally hold their discriminative ability in external validation samples (see e.g. (13–15)).

Yet, several methodological concerns about the validity of the PRS have been raised (3,16–19). It is argued that the functional variation underlying SNP associations may not be captured by risk alleles (16) and that the additive model may not adequately capture the polygenic liability (3). Two studies that compared the additive model with alternatives showed that not only the weighted sum of risk alleles was compatible with empirical data but that several other mathematical models fitted as well when effects at single loci are small (17,18). And it is questioned whether the recent trend of including millions of SNPs challenges the validity of the definition of risk alleles as most their effect sizes are too small to produce ‘observable’ changes in the calculation of risks, even by the thousands (19).

In this paper, I discuss the theoretical concepts of validity that are used for evaluating tests and assessments in the social sciences. I introduce the main types of validity and illustrate their relevance for the construction of PRSs by reflecting on several topics in PRS research: the observation of PRSs as independent effects, the consequences of relaxing SNP inclusion thresholds and the accuracy of GWAS weights.

Types of validity

The validity of a test or measurement indicates whether it measures what it intends to measure (20,21). Assessing the validity of a measurement is challenging when what needs to be measured is difficult to describe and cannot be directly observed. When we cannot directly evaluate the validity of a measurement, which is the case for PRSs, then we need to rely on various sorts of indirect evidence to assure that PRSs measure what we want the scores to measure. But what do we want PRSs to assess? And how well are they doing?

In psychometrics, three types of validity are commonly distinguished: construct, content and criterion validity (Fig. 1) (20). Construct validity informs about the intention of the measurement, about what is the construct that needs to be measured. It relates the construction of the PRS, as a weighted sum of risk alleles, to the underlying theoretical views about genetic liability. It questions whether the genetic liability of a disease is quantified by an additive model that combines the risk alleles into a single score. Construct validity is examined by correlating the PRS of a disease with assessments that measure the genetic predisposition of related (convergent validity) or unrelated diseases (discriminant validity).

Figure 1

Three types of validity applied to the measurement of polygenic risk scores. Legend: * In the context of the specific application of the measurement.

Content validity relates the PRS to what ought to be measured to assess the construct of genetic liability in a certain context. Content considers the measurement of PRS in the practical context of how the score will be applied. For example, when a PRS is used to predict disease in the absence of non-genetic risk factors, then the score needs to capture all polygenic variance. If the score is included in a prediction model with clinical risk factors that mediate part of the polygenic risk, then the score needs to capture the polygenic risk that is not captured by the intermediate phenotypes.

Criterion validity examines to what extent PRSs correlate with other measurements that they are expected to be related to, either at the same time (concurrent validity) or in the future (predictive validity). Most evidence on the validity of PRS is about this predictive validity, and for most common diseases, this predictive validity is modest, except when the PRS includes one or more SNPs that have a strong impact on disease risk. The modest predictive validity may limit the PRS’s potential for clinical utility, but it may be informative enough for establishing criterion validity.

Finally, the widespread adoption and uniform application of PRS suggest that the score certainly does have face validity, the fourth type of validity that is often distinguished: the score seems valid on its appearance (22). Strong face validity in the context of limited construct, criterion and content validity is not enough. Whether the construction of PRSs, as weighted sums of risk alleles, is valid depends on the purpose of assessing the polygenic contribution to disease. In the next sections, I will illustrate why a PRS may be valid for the prediction of risk when used on its own but not when it is combined with clinical risk factors, and why a PRS that has its weights from GWASs or that include millions of SNPs may not be valid in clinical applications to inform people about their genetic risk of disease.

Independent effects

In recent years, researchers often report that PRSs predict the risk of common diseases independently from clinical risk factors. Independent effects have been reported for PRSs in breast cancer (23), coronary heart disease (24,25) and coronary artery disease (26,27) and were based on a formal mediation analysis (25), a statistically significant effect size for PRS after combining with clinical risk factors (26), and the absence of correlations or interactions between PRS and clinical risk scores (23,24). The observation of independent effects suggets that the PRSs were not associated with the clinical risk factors, but none of the studies showed these associations. Associations would however be expected as earlier studies on PRSs that included smaller numbers of selected SNPs did report associations with clinical risk factors (28,29), since PRSs are investigated for these intermediate phenotypes themselves, such as for obesity (30) and hypertension (31).

Independent effects may be expected between PRSs and behavioral and environmental risk factors, such as diet and lifestyle. These independent effects have been observed for lifestyle in stroke (32), coronary heart disease (33,34), adiposity (35,36), diabetes (37) and breast cancer (38), and for stressful life events in depression (39). However, there is less evidence of an independent effect of PRSs when they are combined with clinical risk factors, early symptoms or early-life predictors that represent the outcome. Examples include baseline glucose level to predict type 2 diabetes later in life (40), social impairments to predict psychosis (41), childhood obesity to predict adult obesity (42) and education at younger ages to predict highest educational attainment (43). SNPs that play a role in the pathways that lead to disease through these clinical risk factors or early-life stages may be associated with disease risk through these intermediate variables (44). These clinical risk factors may be measured with various levels of accuracy, which will affect how well they are able to mediate a SNP disease relationship, but to a priori expect that they are independent is incorrect. Developing prediction models that combine genetic and non-genetic risk factors is straightforward, but when the construction of the PRS needs to consider the possible mediating role of various non-genetic risk factors, then more consideration is needed to find out whether and how each of the variables mediates the association between SNPs and disease risk, and how the SNPs are best combined into risk scores.

If we assume that SNPs are biologically related to intermediate clinical risk factors, then we must rule out that the observation of independent effects is an artifact introduced by the method of calculating PRSs. To this end, we must verify whether individual SNPs are associated with the intermediate phenotypes, and find out, if they are, how polygenic predisposition is best quantified in a way that allows the associated SNPs to predispose their intermediate risk factors. Evidently, when none of the SNPs is associated with the clinical risk factors, then a single PRS might suffice.

Figure 2 presents a simplified scenario of how SNPs and PRS relate to coronary artery disease (CAD) with blood pressure and cholesterol as intermediate clinical risk factors. The graph is analogue to the directed acyclic graphs that are used in epidemiological research to express the direct causal relations between study variables (45–47), with the difference that, for prediction, the relationships do not need to be causal. Drawing ‘causal’ graphs helps identifying which variables should be considered in the construction of risk models that combine genetic and clinical risk factors and how they need to be modeled. Figure 2A shows that among the SNPs that predispose to CAD are SNPs for blood pressure and cholesterol. From mediation analyses, we know that the effects of these SNPs on CAD risk decrease when their clinical risk factors are included in the model. We expect that only the SNPs for which no intermediate factors are included will impact CAD risk. ‘Residual’ effects may be observed based on measurement error in the clinical variables and in the genetic data if SNPs do not accurately capture the underlying risk-increasing associations (16).

Figure 2

Independent effects between single-nucleotide polymorphisms, polygenic risk scores and clinical risk factors. Legend: PRS, polygenic risk score; SNP, single-nucleotide polymorphism; CAD, coronary artery disease. For illustration purposes, other possible associations between variables are omitted.

When SNPs from various known and unknown pathways are combined into a single PRS, we should expect that the PRS may not be associated with each of the clinical risk factors (Fig. 2B), or the effect may be attenuated (33). The PRS then presents as an independent risk factor, while some of the SNPs in the score may still predispose the clinical risk factors. The effects of the SNPs that predispose blood pressure and cholesterol would be reduced when the SNPs were entered as separate variables in the analysis, but they now remain part of the PRS. Their effects, at least conceptually, are now counted twice: through the clinical risk factor and in the score.

When a PRS is constructed for inclusion in a prediction model with clinical risk factors, the score should not measure the polygenic contribution (as illustrated in Fig. 1) but the part of the polygenic contribution that is not captured by clinical risk factors. PRS needs to quantify the ‘residual’ polygenic contribution. For this, we need alternative methods that capture the effects of SNPs in ways that allow relevant clinical predictors to mediate when their predisposing genes are included in the score. We may need methods that divide the PRS into multiple scores that each are optimized so clinical risk factors can act their predisposing role in pathways (Fig. 2C) (48–50).

Vassy and colleagues investigated pathway specific PRSs in type 2 diabetes (51). They predicted type 2 diabetes using a total PRS consisting of 62 SNPs, as well as using separate, non-overlapping PRSs based on SNPs associated with insulin resistance (10 SNPs) and beta-cell function (20 SNPs). Table 1 shows that the odds ratio of the beta-cell function PRSs were higher than those of the insulin resistance PRSs, which is explained by the fact that these scores included SNPs that had the highest ORs of all SNPs considered in the total PRS. In three different populations, the odds ratios of the overall and beta-cell PRSs remained unchanged after adjustment for clinical variables, but the ORs of all insulin resistance PRSs reduced. This underscores that clinical risk factors may mediate the association between PRS and the risk of disease when the construction or PRS allows for this mediating role.

Table 1

A comparison of overall and pathway-specific polygenic risk scores in type 2 diabetes

	Model 1	Model 2	Model 3	Model 4
	PRSt	PRSβ	PRSir	PRSβ	PRSir
Framingham offspring study (n = 3471)
Demographic model	1.08(1.06, 1.10)	1.11(1.08, 1.15)	1.04(1.00, 1.10)	1.11(1.08, 1.15)	1.05(1.00, 1.10)
Clinical model	1.06(1.04, 1.08)	1.10(1.06, 1.14)	0.98(0.93, 1.04)	1.10(1.06, 1.14)	0.99(0.93, 1.04)
CARDIA study, whites (n = 1650)
Demographic model	1.08(1.04, 1.12)	1.09(1.02, 1.16)	1.06(0.96, 1.17)	1.09(1.02, 1.16)	1.06(0.96, 1.17)
Clinical model	1.06(1.02, 1.10)	1.09(1.02, 1.17)	1.01(0.91, 1.12)	1.09(1.02, 1.17)	1.01(0.91, 1.11)
CARDIA study, blacks (n = 820)
Demographic model	1.05(1.01–1.09)	1.06(0.98, 1.14)	1.09(1.00, 1.20)	1.06(0.98, 1.14)	1.10(1.00, 1.20)
Clinical model	1.05(1.00–1.09)	1.06(0.99, 1.15)	1.05(0.96, 1.15)	1.07(0.99, 1.15)	1.05(0.96, 1.16)

Data are obtained from (51). Values are odds ratios with 95% confidence intervals. Models 1–3 have one PRS in the model; model 4 includes both PRSβ and PRSir. PRS, polygenic risk score; PRSt, PRS total; PRSβ, PRS beta-cell function; PRSir, PRS insulin resistance; CARDIA study, Coronary Artery Risk Development in Young Adults study. Demographic models are adjusted for age and sex, and clinical models are additionally adjusted for parental history of diabetes, body mass index, systolic blood pressure, fasting plasma glucose, high-density lipoprotein and fasting triglycerides. Reprinted with permission from Jason L. Vassy, Marie-France Hivert, Bianca Porneala, Marco Dauriz, Jose C. Florez, Josée Dupuis, David S. Siscovickm Myriam Fornage, Laura J. Rasmussen-Torvik, Claude Bouchard and James B. Meigs: Polygenic Type 2 Diabetes Prediction at the Limit of Common Variant Detection, Diabetes 2014 Jun; 63 (6): 2172–2182: https://doi.org/10.2337/db13-1663. Copyright 2014 by the American Diabetes Association.

Unfortunately, Vassy et al. did not report the c-statistic as a measure of the discriminative ability for the prediction models that included either one or both pathway-specific PRSs. We do not know if and how considering multiple pathway-specific PRSs changed the c-statistic as in contrast to adding a single PRS to clinical risk factors. The c-statistic may be higher when pathways that are more heritable have PRSs with higher effect sizes that are then not reduced by variants with weaker effects from other pathways. This is illustrated by the data of Vassy et al., which showed that the beta-cell function PRS consistently had higher odds ratio than the total PRS (Table 1). It is also possible that separate PRSs lead to a lower c-statistic when part of the genetic effect is removed after adjustment for clinical risk factors. And it might be that the two approaches yield the same improvement in c-statistic when the contribution of the SNPs was minimal to begin with.

Finally, the extent to which clinical risk factors can mediate the association between SNPs and disease not only depends on how the PRS is constructed but also on the assessment of the clinical risk factors. Adequate assessment is a challenge when biomarkers that fluctuate over time are not measured timely and frequently enough to capture that variation. Such variations may occur based on daily rhythms or be induced by that week’s diet and other relevant lifestyle factors. The development of combined risk models should therefore not only focus on how to optimally assess the genetic contribution but also how to optimally measure the clinical risk factors.

GWAS weights

PRSs are typically constructed using weights from large GWASs. These effect sizes are preferred to using weights obtained from the study in which the PRS is investigated for the robustness of the estimates. Yet, taking weights from GWAS assumes that SNPs have the same impact on disease risk in all populations of the same ethnicity. This is unrealistic. First, GWAS estimates are pooled across multiple studies that differ in study design and study population and that may even differ in the diagnostic criteria and assessment of the disease of interest. The GWAS weight for each SNP may reflect none of the effect sizes of the individual studies. Second, the effect sizes may be overestimated because of winner’s curse and biases (52–54). Even when the identified GWAS hits are true positives, their effect sizes may be attenuated in other populations.

The overestimation of GWAS effect sizes is illustrated in the 23andMe’s white paper about its new PRS for type 2 diabetes (55). 23andMe compared its SNP weights with the weights of the genome-wide significant SNPs in the GWAS of Scott et al. (56). The sample size of the GWAS was about 160 000 individuals and of the 23andMe study about 940 000. Figure 3 shows that all effect sizes were in the same direction, but that most effect sizes of the GWAS were more extreme than the estimates from 23andMe. If the GWAS estimates were used to predict type 2 diabetes risk in the 23andMe population, poorer calibration of the PRS should be expected, especially at the tails of the risk distribution.

Figure 3

Per allele effect sizes for single-nucleotide polymorphisms in type 2 diabetes. Legend: Picture provided by 23andMe, reproduced from (55). The dots represent the genome-wide significant polymorphisms in the study of Scott et al. (56).

The choice of weights is relevant for the content validity of the PRS. If the PRS is intended as a measure of polygenic variance (Fig. 1), then it can be argued that the PRS needs to be designed such that it reflects the variation in the population that is studied. The weights of the SNPs, the point estimates of their effect sizes, are at the core of the PRS calculation, and their adequate estimation is important. Weights are typically obtained from GWAS, but this may not be the obvious choice when study populations are large enough to be used for estimating their own weights. These cohorts do not need to be used to identify SNPs, but they could be used for re-estimating or adjusting the GWAS effect sizes. We need more insight in the variation of effect sizes within populations of the same ethnicity if we want to understand the generalizability of PRSs (53,54).

Relaxing SNP inclusion thresholds

A recent trend in the construction of PRSs is to use millions of SNPs in the prediction of risks (27,57,58). These PRSs go beyond only including genome-wide significant SNPs from GWASs and beyond setting lower thresholds (higher P-values) than genome-wide significance for the selection of SNPs (7). These PRSs are constructed using methods like LDpred that optimize the weights for all SNPs using their GWAS weights, linkage disequilibrium and an estimate of the proportion of SNPs that are expected to have non-zero weights (8). The often millions of SNPs that are assumed to have zero weights are generally kept in the PRS calculation even when their weights are small and would have been zero if the weights would be quantified using, say, only three or four digits after the decimal point.

Most of these millions of SNPs have such negligible impact on risk that their inclusion in the score does not affect the predictions: excluding all SNPs except, say, the genome-wide significant SNPS or the top 1000 SNPs with the largest weights will unlikely change individual predictions (27,59–61). When Khera and colleagues constructed 30 PRSs using up to 7 million SNPs for each of five common diseases (27), most PRSs had lower c-statistics than the PRSs based on genome-wide significant variants only (19). Considering millions of SNPs that have negligible impact on disease risk might be a non-issue computationally, but it is not a non-issue if the scores need to reflect the underlying theoretical views on the genetic predisposition of the disease of interest.

Construct and content validity are about semantics and labeling. When the weights of ‘risk’ alleles are as low as 0.0000001 (and lower), do we consider these SNPs to be associated with disease risk, and should they be included in the PRS? When a PRS is constructed based on 2 million of SNPs of which, say, 100 are able to change predicted risks before the decimal, will we tell a patient that their risk is calculated using 2 million or 100 SNPs?

When researchers relax the SNP inclusion thresholds to include millions of SNPs, they often aim for and select the PRS with the highest proportion of explained variance or the highest c-statistic, even when the differences in these metrics are minimal and adding millions of SNPs is unlikely to change predicted risks for individuals (24,27,58,62). These minimally higher proportions of explained variance or minimal improvements of the c-statistic do not evidently translate into better health or more healthcare benefits. There may be no benefit for relaxing threshold beyond GWA significance (63), and there may be no good reason to go beyond the genome-wide significant SNPs for PRSs that are to be used in healthcare. If it is deemed valid to include millions of SNPs in PRSs, then we need to challenge ourselves to specify what is basis for this validity judgment.

Conclusion

PRSs do not ‘exist’ in the same way blood pressure and cholesterol level exist. The latter may be measured inaccurately, but blood flow has a pressure, and blood may contain more or less cholesterol. PRS is constructed, a pragmatic solution introduced when the number of SNPs became too large to be considered as separate variables in a regression analysis. PRS might be valid as an algorithm for predicting risk when used alone or in combination with variables we expect to be independent, such as age, sex and behavioral risk factors. Yet, when modeled together with clinical risk factors that are associated with its SNPs as intermediate phenotypes, the construction of PRS should be such that these risk factors can act as intermediate phenotypes, capturing the effects of the SNPs that predispose them.

PRSs do not ‘exist’ in the same way clinical risk models do not exist either. The validity of clinical risk models needs to be demonstrated, and the choices in the model development need to be justified (64). Clinical risk models need to be developed and externally validated in relevant settings so that they predict what they intend to predict in the population where the risk model is intended to be used. They need to be compared with other risk models that calculate the same risks using different algorithms. The demonstration of validity should be no less rigorous for PRS.

This paper aimed to reflect on the validity of PRSs by introducing the types of validity that are deemed important in the design of measurements, tests or questionnaires (20). PRSs have strong face validity; they intuitively seem to make sense, but this apparent face validity is not enough. More comparative research is needed to investigate the construct, content and criterion validity of PRS; to explore alternative ways of quantifying polygenic risk; and to rigorously compare new and current methods (3). A critical reflection of what needs to be measured by PRSs, from a theoretical perspective to assure their construct validity and from a practical perspective to assure their content validity, will help evaluating whether the PRSs that are constructed are the ones that were intended.

Models are simplifications of reality. They can be useful even when they are wrong. The same holds for PRSs, but we need to keep questioning if what we assess is what we think we do and to seek for alternative modeling strategies that might better reflect the underlying biological pathways. The construction of PRSs needs to acknowledge the biological reality, not create a new one.