Validating the QCOVID risk prediction algorithm for risk of mortality from COVID-19 in the adult population in Wales, UK.

Introduction: COVID-19 risk prediction algorithms can be used to identify at-risk individuals from short-term serious adverse COVID-19 outcomes such as hospitalisation and death. It is important to validate these algorithms in different and diverse populations to help guide risk management decisions and target vaccination and treatment programs to the most vulnerable individuals in society.
Objectives: To validate externally the QCOVID risk prediction algorithm that predicts mortality outcomes from COVID-19 in the adult population of Wales, UK.
Methods: We conducted a retrospective cohort study using routinely collected individual-level data held in the Secure Anonymised Information Linkage (SAIL) Databank. The cohort included individuals aged between 19 and 100 years, living in Wales on 24th January 2020, registered with a SAIL-providing general practice, and followed-up to death or study end (28th July 2020). Demographic, primary and secondary healthcare, and dispensing data were used to derive all the predictor variables used to develop the published QCOVID algorithm. Mortality data were used to define time to confirmed or suspected COVID-19 death. Performance metrics, including R2 values (explained variation), Brier scores, and measures of discrimination and calibration were calculated for two periods (24th January-30th April 2020 and 1st May-28th July 2020) to assess algorithm performance.
Results: 1,956,760 individuals were included. 1,192 (0.06%) and 610 (0.03%) COVID-19 deaths occurred in the first and second time periods, respectively. The algorithms fitted the Welsh data and population well, explaining 68.8% (95% CI: 66.9-70.4) of the variation in time to death, Harrell's C statistic: 0.929 (95% CI: 0.921-0.937) and D statistic: 3.036 (95% CI: 2.913-3.159) for males in the first period. Similar results were found for females and in the second time period for both sexes. Conclusions: The QCOVID algorithm developed in England can be used for public health risk management for the adult Welsh population.

Introduction

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection was first identified in Wuhan, China [1]. On the 24th January 2020, the UK recorded its first case of SARS-CoV-2 and as of 22nd August 2021, there have been 6,492,906 confirmed cases with 131,640 COVID-19-related deaths in the UK [2, 3]. Research has shown that increased age, being male, certain minority ethnic groups, and having pre-existing conditions such as diabetes, cardiovascular disease, and obesity are associated with serious adverse COVID-19 outcomes, including hospitalisation and death [4-9].

To protect the most vulnerable, and to minimise the burden on the National Health Service (NHS) and its staff, it is important to identify those at greatest risk of serious adverse COVID-19 outcomes [10, 11]. COVID-19 risk prediction algorithms can be used to identify and prioritise at-risk individuals for targeting vaccination and treatments as well as to inform risk management decisions and policy as the pandemic evolves [12].

The New and Emerging Respiratory Virus Threats Advisory Group (NERVTAG)’s effort to develop a population risk assessment framework led to the development and validation of the QCOVID tool, a population-based prediction algorithm to predict the risk of being admitted to hospital or dying from COVID-19 across an adult population [3, 13, 14]. The algorithm was initially developed and validated on a cohort of six million primary care patients from 1,205 English practices contributing to the QResearch database, which allows linkage at the individual-level to general practitioner (GP) primary care data, death records, hospital admissions data and COVID-19 test results. Predictive demographic, clinical, and pharmaceutical variables (Box 1) were based on the clinical vulnerability group criteria used to identify those advised to shield at the start of the pandemic, and risk factors associated with adverse outcomes for respiratory diseases [15, 16].

Demographic

Age in years on 24th January 2020

Biological sex at birth

Townsend Deprivation Score

Ethnicity

What is your housing category - care home, homeless or neither?

Lifestyle

Body Mass Index

Conditions on current shielding patient list

Have you had chemotherapy in the last 12 months?

Have you had radiotherapy in the last 6 months?

Have you had a bone marrow or stem cell transplant in the last 6 months?

Have you had a solid organ transplant (lung, liver, stomach, pancreas, spleen, heart or thymus)?

Do you have sickle cell disease or severe combined immune deficiency syndromes?

Do you have cystic fibrosis, bronchiectasis or alveolitis?

Have you a cancer of the blood or bone marrow such as leukaemia, myelodysplastic syndromes, lymphoma or myeloma and are at any stage of treatment?

Do you have lung or oral cancer?

Do you have congenital heart disease or have you had surgery for it in the past?

Conditions moderately associated with increased risk of complications as per current NHS guidance

Do you have a learning disability or Down’s Syndrome?

Chronic Kidney Disease (CKD) stage

Do you have asthma?

Do you have diabetes?

Do you have Parkinson’s disease?

Do you have cerebral palsy?

Do you have epilepsy?

Do you have rheumatoid arthritis or Systemic lupus erythematosus?

Do you have dementia?

Do you have chronic obstructive pulmonary disease (COPD)?

Do you have motor neurone disease, multiple sclerosis, myasthenia, or Huntington’s chorea?

Do you have coronary heart disease?

Do you have heart failure?

Other medical conditions that investigators hypothesized to confer elevated risk

Do you have peripheral vascular disease?

Do you have severe mental illness?

Have you had a prior fracture of hip, wrist, spine or humerus?

Do you have atrial fibrillation?

Do you have cirrhosis of the liver?

Do you have pulmonary hypertension or pulmonary fibrosis?

Have you had a thrombosis or pulmonary embolus?

Have you had a stroke or transient ischaemic attack?

Concurrent medications

Have you been prescribed immunosuppressants four or more times in the previous 6 months?

Have you been prescribed anti-leukotriene or long acting beta2-agonists (LABA) four or more times in the previous 6 months?

Have you been prescribed oral prednisolone containing preparations prescribed four or more times in the previous 6 months?

Replication of results in diverse populations is an important component of scientific research and is especially important for validation of prediction algorithms generated using routine data where the results may be used to plan clinical management of individual patients. It was decided to replicate and compare the performance of the algorithm in each of the four nations in the UK to ensure validity and contribute to the application of the algorithm in managing responses to the outbreak. A recent published study validated the QCOVID predictive algorithm in estimating the risk of mortality from COVID-19 in 35 million adult residents of England by the Office for National Statistics using linked Census 2011 data [17]. The aim of our study was to externally validate the QCOVID risk prediction algorithm to estimate mortality outcomes from COVID-19 in adults in Wales, UK. This paper replicates the English validation study and follows the RECORD and TRIPOD reporting guidelines [18, 19].

Methods

Study design and data sources

This study used routinely collected anonymised health and demographic data held in the Secure Anonymised Information Linkage (SAIL) Databank to create a retrospective population-based individual-level linked e-cohort. The SAIL Databank is a Trusted Research Environment (TRE), which hosts linkable anonymised individual and household-level health, demographic, administrative and environmental data for the population of Wales [20, 21].

Following the emergence of the SARS-CoV-2 infection and the subsequent COVID-19 pandemic, two population-level cohorts (known as C16 and C20) were created to support rapid analysis, provide evidence in understanding the evolving pandemic, and evaluate national interventions attempting to reduce the spread of infection [22]. The C20 contains all individuals alive and living in Wales from 1st January 2020 and followed up until death, emigration/break in Welsh residency, or cohort end date (currently 30th June 2021). This cohort is updated on a monthly basis to extend the available follow-up time. The C16 acts as a contextual comparative cohort and contains all individuals alive and living in Wales on 1st January 2016 and followed up until death, emigration/break in Welsh residency, or 31st December 2019.

For this study, we used the C20 to create a cohort of all individuals aged 19–100 years, living in Wales and registered with a SAIL providing general practice on 24th January 2020. The 24th January 2020 was chosen as the cohort entry as this is the date of the first confirmed COVID-19 case in the UK. Individuals were followed up until death or study end date (28th July 2020), with the study divided into two time periods, 24th January 2020–30th April 2020 and 1st May 2020–28thJuly 2020, to match the English validation study [17]. Individuals who had died prior to 1st May 2020 were excluded from the second time period analysis.

Predictor variables

To validate the QCOVID algorithm, the C20 cohort was linked to the Welsh Longitudinal General Practice (WLGP), Patient Episode Database for Wales (PEDW), Wales Dispensing DataSet (WDDS), and Office for National Statistics (ONS) Census 2011 (CENW) data [23] to derive the pre-existing conditions and demographic characteristics that were used to develop the QCOVID algorithm (Box 1).

The C20 cohort was used to define age, sex, and Townsend score. Townsend score is a measure of deprivation, based on the area of residence, and a higher score implies a higher level of deprivation. The CENW is linked to derive ethnicity (i.e. Bangladeshi, Black African, Black Caribbean, Chinese, Indian, Pakistani, Mixed, Other, and White) [24]. The ethnicity variable had a category corresponding to ‘not recorded/unknown’. This category was used whenever the corresponding value was missing.

The majority of pre-existing conditions were identified in the WLGP primary care data source using Read codes version 2 (CTV2). Where no timeframe was stated, a lookback period from 1st January 1998 to 24th January 2020 was used. For body mass index (BMI), the latest BMI measurement within 5 years to 24th January 2020 was used. BMI records outside this time period as well as BMIs <15 and >47 were set to missing. If an individual had multiple BMI records on the latest date, the highest BMI was included. Predicted values using all QCOVID predictor variables with age interactions from linear regression models, were used to impute any missing BMI values. Recorded BMI is dependent on the condition of interest and healthcare utilisation activity of the individual, therefore, it is possible to have individuals with no BMI recorded when using routinely collected healthcare data. For diabetes, if the latest health record had defined an individual with both type 1 and type 2 diabetes, type 2 took precedence [3]. For the housing covariate, if the latest record defined an individual being homeless and living in a care home, then living in a care home took precedence. For the learning disabilities covariate, if the latest record identified an individual as having learning disabilities and Down’s syndrome, then Down’s syndrome was prioritised.

Office of Population Censuses and Surveys (OPCS) Classification of Interventions and Procedures version 4 (OPCS-4) coded conditions in the inpatient (PEDW) data were used to identify chemotherapy status, Chronic Kidney Disease (CKD) stages, congenital heart disease surgery, bone marrow or stem cell transplant, radiotherapy, and solid organ transplant.

DMD (Dictionary of Medical Devices) coded prescriptions in the WDDS were used to identify individuals who had been dispensed immunosuppressants, anti-leukotriene or long acting beta2-agonists (LABA), or oral prednisolone at least four or more times within 6-months prior to 24th January 2020.

Outcome of interest – death involving COVID-19

We utilised a combination of data held in ONS Annual District Death Extract (ADDE) and Annual District Death Daily (ADDD), Welsh Demographic Service Dataset (WDSD) and Consolidated Death Data Source (CDDS) to identify all deaths, inclusive of in-hospital and out of hospital deaths, of Welsh residents. Deaths involving COVID-19 (confirmed or suspected) were identified using the tenth revision of the International Classification of Diseases (ICD-10) codes U07.1 or U07.2, or from text fields containing the causes of death within the data sources. Time to death from COVID-19 was calculated separately in the first period (24th January 2020–30th April 2020) and the second period (1st May 2020–28th July 2020).

Algorithm validation

The QCOVID risk equations (version 1) reported in the original study were fitted for males and females separately [3, 14]. The original paper utilised the Fine-Gray sub-distribution hazard model which is commonly used to estimate incidence of outcomes where competing risks exist. It relates covariates to the cumulative incidence function (CIF) of the outcome of interest [25, 26]. The following modifications for the Welsh adult population were required due to data issues. At the time of analysis, Systemic Anti-Cancer Therapy (SACT) data were not available, therefore, anyone receiving chemotherapy within 12-months of 24th January 2020 was assigned the chemotherapy group B (middle severity group) coefficients from the original study [27]. Due to low cohort numbers and subsequent outcome numbers for some ethnic groups, we collapsed ethnic groups to ensure ethnic minority populations or groups were not excluded from our study. Black Caribbean individuals were assigned Black African coefficients, Chinese individuals were assigned the coefficients for the Other ethnic group, and, all White ethnic groups were assigned the White British coefficients.

Performance metrics, including measures of discrimination and calibration, were calculated to validate the predicted risk of death from COVID-19 using the QCOVID algorithm at 97 days for the first period and 88 days for the second period [28-30]. We calculated R2 values, D statistic, Harrell’s C statistic and Brier scores with corresponding 95% confidence intervals for the total cohort by sex and over the two time periods. The performance measurements were also calculated by age bands, ethnicity and Townsend deprivation quintiles. The R2 values refer to the proportion of variation in survival time explained by the model while the Brier score measures predictive accuracy. The D statistic and Harrell’s C statistic are discrimination measures that quantify the separation in survival between patients with different levels of predicted risks, and the extent to which people with higher risk scores have earlier events, respectively. To measure calibration, we compared the mean observed and predicted risks within each twentieths of predicted risk (20 groups) for the two time periods. Observed risks were derived in each of the 20 groups using non-parametric estimates of the cumulative incidences.

Results

Overall, there were 1,956,760 individuals aged 19-100 years included in the final analysis for Wales. Of these, 967,975 (49.5%) were male with a mean age of 50.8 (SD 18.7) and the majority of individuals were from White ethnic backgrounds (1,741,527, 89.0%) (Table 1). In comparison with the English validation cohort and original cohort (Supplementary Table 1), these distributions of demographic characteristics were similar except for ethnicity with a lower proportion of individuals from ethnic minority backgrounds in Wales, but also a higher proportion (6.5%) of individuals missing this information (Table 1). The Welsh cohort had similar prevalence of pre-existing conditions when compared to the English validation cohort and original cohort. However, the proportion of people with higher BMI, CKD, respiratory cancer, venous thromboembolism (VTE), coronary heart disease (CHD) and osteoporotic fractures was slightly higher in the Welsh data and slightly lower for immunosuppressant use, dementia, or a serious mental illness compared to the English validation cohort. The proportions of people with missing BMI values, pulmonary hypertension and VTE were slightly higher in the Welsh data compared to original cohort.

Table 1: Demographic and clinical characteristics for the total cohort and those who died with COVID-19 in the two time periods

	Overall cohort			COVID-19 deaths in first period (24th Jan–30th Apr 2020)			COVID-19 deaths in second period (1st May–28th Jul 2020)
	N	%		N	%		N	%
Overall	1,956,760			1192			610
Sex
Male	967,975	49.47		674	56.54		299	49.02
Female	988,785	50.53		518	43.46		311	50.98
Age, years	50.8	18.7		79.4	11.8		81.0	11.1
Age group, years
19-29	318,681	16.29		*			*
30-39	313,802	16.04		*			*
40-49	304,363	15.55		16	1.34		*
50-59	353,539	18.07		61	5.12		28	4.59
60-69	291,042	14.87		132	11.07		49	8.03
70-79	240,840	12.31		305	25.59		136	22.30
80-89	111,631	5.70		429	35.99		250	40.98
≥90	22,862	1.17		242	20.30		138	22.62
Ethnicity
Bangladeshi	7,011	0.36		*			*
Black^	8,312	0.42		*			*
Indian	8,885	0.45		*			*
Mixed	27,582	1.41		*			*
Other^	27,786	1.42		*			*
Pakistani	7,688	0.39		*			0	0.00
White	1,741,527	89.00		1113	93.37		579	94.92
Not recorded	127,969	6.54		52	4.36		19	3.11
Townsend deprivation quintile
1 (most affluent)	335,459	17.14		156	13.09		98	16.07
2	413,486	21.13		221	18.54		129	21.15
3	559,024	28.57		369	30.96		179	29.34
4	453,474	23.17		304	25.50		141	23.11
5 (most deprived)	195,317	9.98		142	11.91		63	10.33
Accommodation
Neither homeless nor care home	1,940,224	99.15		987	82.80		476	78.03
Care home or nursing home	16,536	0.85		205	17.20		134	21.97
Body-mass index, kg/m2
<18.5	21,944	1.12		53	4.45		33	5.41
18.5 to <25	316,569	16.18		277	23.34		161	26.39
25 to <30	375,501	19.19		300	25.17		154	25.25
≥30	403,871	20.64		294	24.66		114	18.69
Not recorded	838,875	42.87		268	22.48		148	24.26
Chronic kidney disease
No Chronic Kidney disease	1,874,451	95.79		869	72.90		412	67.54
Stage 3	72,669	3.71		252	21.14		165	27.05
Stage 4	3,928	0.20		30	2.52		20	3.28
Stage 5	5,712	0.29		41	3.44		13	2.13
Learning disability
No learning disability	1,928,040	98.53		1163	97.57		587	96.23
Learning disability	28,486	1.46		29	2.43		23	3.77
Down Syndrome	234	0.01		0	0.00		0	0.00
Chemotherapy								0.00
No chemotherapy in past 12-months	1,949,761	99.64		1167	97.90		597	97.87
Chemotherapy in past 12-months	6,999	0.36		25	2.10		13	2.13
Cancer and immunosuppression
Blood cancer	10,547	0.54		38	3.19		14	2.30
Respiratory cancer	5,691	0.29		20	1.68		10	1.64
Radiotherapy in past 6-months	1,827	0.09		*			*
Bone marrow transplant in past 6-months	56	0.00		0	0		0	0.00
Solid organ transplant	806	0.04		*			*
Prescribed immunosuppressant medication by GP	2,884	0.15		*			*
Prescribed leukotriene or LABA	38,658	1.98		59	4.95		42	6.89
Prescribed regular prednisolone	15,819	0.81		61	5.12		28	4.59
Other comorbidities
Diabetes	161,227	8.24		359	30.12		178	29.18
COPD	66,937	3.42		209	17.53		100	16.39
Asthma	290,490	14.85		186	15.60		109	17.87
Rare pulmonary diseases	9,471	0.48		26	2.18		12	1.97
Pulmonary hypertension or pulmonary fibrosis	3,741	0.19		17	1.43		14	2.30
Coronary heart disease	89,686	4.58		239	20.05		137	22.46
Stroke	55,336	2.83		233	19.55		121	19.84
Atrial fibrillation	62,712	3.20		253	21.22		140	22.95
Congestive cardiac failure	30,937	1.58		151	12.67		99	16.23
Venous thromboembolism	43,708	2.23		111	9.31		54	8.85
Peripheral vascular disease	18,639	0.95		77	6.46		36	5.90
Congenital heart disease	17,071	0.87		30	2.52		12	1.97
Dementia	18,840	0.96		304	25.50		160	26.23
Parkinson’s disease	5,717	0.29		40	3.36		32	5.25
Epilepsy	26,112	1.33		31	2.60		19	3.11
Rare neurological conditions	5,789	0.30		*			*
Cerebral palsy	1,318	0.07		0	0.00		0	0.00
Severe mental illness	282,709	14.45		209	17.53		109	17.87
Osteoporotic fracture	73,679	3.77		154	12.92		96	15.74
Rheumatoid arthritis or SLE	22,485	1.15		35	2.94		16	2.62
Cirrhosis of the liver	7,210	0.37		17	1.43		*
Sickle cell disease	1,094	0.06		0	0		0	0.00

Data are n (%) or mean (SD). * represents values which have been suppressed due to small numbers <10. ^represents collapsing of categories to suppress small numbers.

In total, there were 1,192 (0.06%) COVID-19 deaths during the first period and 610 (0.03%) in the second period, which was similar to the English validation (0.08% and 0.04%, respectively) [16]. In general, individuals who died from COVID-19 during the first period were more likely to be male (674, 56.5%), aged 70 years and older (976, 81.9%), with diabetes, CKD, obesity, and cardio-pulmonary diseases being the pre-existing conditions with the highest proportions of death (Table 1). Individuals who died from COVID-19 during the second period had similar characteristics to the first period, however, with a slight change to the sex ratio (56.5% of deaths in first period were in males compared to 51.0% deaths in the second period were in females).

The performance metrics calculated to validate the predicted risk of death from COVID-19 using the QCOVID algorithm are presented in Table 2 [3, 14]. The metrics have been provided for both sexes and time periods. In the first time-period for males, the algorithm explained 68.8% (95% CI: 66.9–70.4) of the variation in time to death, the Harrell’s C statistic was 0.929 (95% CI: 0.921–0.937), the D statistic was 3.036 (95% CI: 2.913–3.159) and Brier score was 0.0007. Similar results were found for females and in the second time period. Similar results were also found in the English validation, the D statistics was 3.761 (3.732–3.789), Harrell’s C statistic was 0.935 (95% CI: 0.933–0.937) and Brier score was 0.0013 in males in the first period, with similar results found in females and in the second time period [17]. Performance metrics by age band, ethnicity and Townsend deprivation quintile can be found in the Appendices (Supplementary Tables 2–5).

Table 2: Performance of the risk models to predict risk of COVID-19 death by sex and time period for total cohort

	First period (24th January 2020–30th April 2020)			Second period (1st May 2020–28th July 2020)
	COVID-19 death in females	COVID-19 deaths in males		COVID-19 death in females	COVID-19 deaths in males
R-squared statistic	0.691 (0.671–0.710)	0.688 (0.669–0.704)		0.721 (0.698–0.742)	0.711 (0.686–0.733)
D statistic	3.062 (2.922–3.202)	3.036 (2.913–3.159)		3.293 (3.113–3.472)	3.207 (3.024–3.390)
Harrell’s C statistic	0.930 (0.920–0.940)	0.929 (0.921–0.937)		0.950 (0.942–0.959)	0.933 (0.921–0.945)
Brier score	0.0005	0.0007		0.0003	0.0003

Data are estimated (95% CI).

The Harrell’s C statistic varied across the age bands and time periods (Figures 1, 2), with acceptable discrimination (>0.7) in both time periods for males and females, and across age groups. The oldest group (90+ years old) yielded poorer discrimination for both males and females as well as the youngest male group in the first time period. In the second time period, it was not possible to plot the Harrell’s C statistic for the youngest age groups for females (19-39 and 40-44 years) or for 19-39 years in males due to low numbers. Whilst the Harrell’s C statistic was slightly lower in Wales compared to England across sex and age groups, the pattern of reduced discrimination for certain age groups was similar.

Figure 1: The concordance index by sex and age group in the first time period (24

Figure 2: The concordance index by sex and age group in the second time period (1

The calibration plots in Figure 3 showed that the predicted and observed risks of COVID-19 related death were similar for both males and females in the first time period, demonstrating the QCOVID equations were well calibrated. However, there was slight under-prediction in the highest risk category for COVID-19 death which was also demonstrated in the English validation and original cohorts [3, 17]. Predicted and observed risks of COVID-19 related death in the second time period can be found in Supplementary Figure 1.

Figure 3: Predicted and observed risk of COVID-19-related death in the first time period (24

Figure 4 demonstrates that the sensitivity at different absolute risk thresholds for COVID-19-related deaths was higher for females in the top 13 centiles compared to males in the first period and was higher in females than males across the second period. 60.2% and 65.4% of deaths occurred in those in the top 5% for predicted absolute risk of death from COVID-19 in the first time period for males and females respectively; 64.9% and 72.0% of deaths occurred in those in the top 5% for predicted absolute risk of death from COVID-19 in the second time period for males and females, respectively (Supplementary Table 6).

Figure 4: Sensitivity for COVID-19-related death in the first (24

Discussion

The results from this validation of the QCOVID risk prediction algorithm show that the models fit the Welsh population data well and yielded similar results, but with less precision (predictably, given the smaller population size) compared to the English validation and original study. This study used individual-level linked data on the adult population of Wales, registered with a SAIL providing general practice, which is independent of the original and validation study populations [22]. Use of SAIL Databank allowed linkage across primary and secondary health care data with mortality outcome data to allow replication of the original and English validation studies and inclusive of all predictor variables [3, 17].

The risk models from the original QCOVID and English validation paper were based on GP data largely from England [3, 17]. Age standardised death rates in Wales pre-pandemic were about 6% higher than in England [31]. Some differences in prediction accuracy are expected and this is consistent with the higher observed to predicted mortality numbers at the higher end of risk in Figure 3 [32]. The predicted and observed risks of COVID-19-related death were similar across most of the predicted risk distribution, demonstrating the models were well calibrated, (60.2-72.0% of deaths occurred in the top 5% for predicted absolute risk of death), apart from the highest 20th of risk where the risk of death was higher in Wales, as shown in Figure 3. This is similar to the English validation study, which demonstrated 65.9-77.2% of death occurred in individuals in the top 5% for predicted absolute risk of death [17].

The overall Harrell’s C statistic was >0.9 for males and females for both time periods, demonstrating good overall discrimination of the models. Lower and more varied Harrell’s C statistics across the age bands are likely due to a smaller population and more deaths occurring in the first period during the first peak of the UK pandemic [33].

Despite the predictive model performance metrics indicating that the algorithm performed well on the Welsh data, there are a number of limitations. The Welsh cohort was restricted to individuals registered to a SAIL providing general practice, therefore, results are based on 80% population coverage (330/412, of all general practices in Wales). This restriction was necessary due to the amount of predictor variables that required primary care GP data. Whilst we were able to calculate all predictor variables required, 42.8% of our cohort did not have a BMI recorded in the previous five years, therefore, missing observations were imputed. Also, this study was designed to replicate the English validation study and therefore focussed on COVID-19-related deaths, COVID-19-related hospital admissions will be presented in a subsequent paper. Additionally, as highlighted in the English validation study, testing for COVID-19 was limited in the early stages of the pandemic and therefore some of the early deaths might not be recorded as being COVID-19-related. As this study period covers the start of the pandemic, outcomes relate to the COVID-19 Wild type triggered wave and does not include subsequent Alpha and Delta variant waves. Finally, it was not possible to calculate performance metrics for some age groups and ethnic groups. Due to low numbers of some ethnic groups and consequent death we collapsed some ethnic groups to ensure privacy protection whilst including them in our study. We combined Black African and Black Caribbean groups, and Chinese and Other groups. This analysis was carried out on a smaller and less ethnically diverse population compared to the original studies [3, 17].

Conclusion

This validation of the QCOVID algorithm indicates that the risk prediction models are applicable on a population independent of the original study, which has not been reported before. Our validation is based on Welsh primary care registered patients, for whom the QCOVID algorithm was not modelled on, whereas the original study was based on English primary care registered patients. The Welsh validation offers evidence that the QCOVID algorithm can be used for public health risk management and also could be applied to other populations. This study covered the first wave of the pandemic in Wales/the UK; however, with the emergence of new variants of concern, subsequent new waves of infection and changes in presentation in symptoms of SARS-CoV-2 it is important to adapt these algorithms over longer periods and assess their predictive ability in the context of the evolving pandemic. Further work will include applying an updated algorithm to assess the predictive risk of COVID-19 death and hospitalisation over a longer period of time. We will also assess the impact of the national vaccination program to see how changes in immunity level have impacted adverse COVID-19 outcomes.