Development and validation of a simple risk score for diagnosing COVID-19 in the emergency room.

As the COVID-19 pandemic continues to escalate and place pressure on hospital system resources, a proper screening and risk stratification score is essential. We aimed to develop a risk score to identify patients with increased risk of COVID-19, allowing proper identification and allocation of limited resources. A retrospective study was conducted of 338 patients who were admitted to the hospital from the emergency room to regular floors and tested for COVID-19 at an acute care hospital in the Metropolitan Washington D.C. area. The dataset was split into development and validation sets with a ratio of 6:4. Demographics, presenting symptoms, sick contact, triage vital signs, initial laboratory and chest X-ray results were analysed to develop a prediction model for COVID-19 diagnosis. Multivariable logistic regression was performed in a stepwise fashion to develop a prediction model, and a scoring system was created based on the coefficients of the final model. Among 338 patients admitted to the hospital from the emergency room, 136 (40.2%) patients tested positive for COVID-19 and 202 (59.8%) patients tested negative. Sick contact with suspected or confirmed COVID-19 case (3 points), nursing facility residence (3 points), constitutional symptom (1 point), respiratory symptom (1 point), gastrointestinal symptom (1 point), obesity (1 point), hypoxia at triage (1 point) and leucocytosis (-1 point) were included in the prediction score. A risk score for COVID-19 diagnosis achieved area under the receiver operating characteristic curve of 0.87 (95% confidence interval (CI) 0.82-0.92) in the development dataset and 0.85 (95% CI 0.78-0.92) in the validation dataset. A risk prediction score for COVID-19 can be used as a supplemental tool to assist clinical decision to triage, test and quarantine patients admitted to the hospital from the emergency room.

Introduction

In December of 2019, an outbreak of a novel coronavirus disease was reported in the Hubei Province of China. Caused by the emerging virus Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), the disease has quickly spread across the world. On 30 January 2020, the World Health Organization (WHO) declared the outbreak a Public Health Emergency of International Concern, and on 11 March 2020, declared it a pandemic [1]. As of 11 October 2020, there were over 37.0 million confirmed cases worldwide, with 7.7 million being in the USA [2].

The pandemic has caused significant adverse impacts throughout the USA and the world. Since the early days of the COVID-19 pandemic in the USA, hospital systems have found themselves overwhelmed and with limitations in the capacity to triage, diagnose and treat patients afflicted by COVID-19. As the pandemic continues unabated and as it spreads through the USA, it is necessary to improve hospital screening and stratification of at-risk populations to enable timely and appropriate quarantine, treatment and use of limited resources.

At present, no validated risk score or stratification system is readily available to aid the clinical decision-making process of hospital-based staff in determining when testing for COVID-19 is appropriate [3]. The availability of testing for COVID-19 continues to be an ongoing limitation throughout the USA. A system of clinical risk stratification can help to identify patients that present a higher risk and warrant COVID-19 testing in a resource-limited setting.

In this retrospective study, we reviewed the records of patients presenting to an emergency department in an acute care hospital in the Metropolitan Washington D.C. area who were tested for SARS-CoV-2 and admitted. We reviewed the clinical characteristics, radiographic findings and laboratory findings between those who tested positive and negative, then developed a simple bedside risk prediction scoring system.

Methods

Cohort design and subjects

A retrospective review was performed for patients tested for COVID-19 and admitted to MedStar Southern Maryland Hospital, a 262-bed acute care hospital located in a suburb of Washington D.C. between 1 April 2020 and 30 April 2020. During this time, the hospital and surrounding region experienced a surge of COVID-19 admissions, but universal testing for COVID-19 was not performed for hospitalised patients. Patients were included in the study if they presented to the emergency room and were admitted to the hospital with laboratory-confirmed COVID-19 (cases) or tested negative for COVID-19 within 24 h of hospital admission (controls). All COVID-19 diagnosis was made by nasopharyngeal swab and reverse transcription polymerase chain reaction for SARS-CoV-2. Patients who were admitted from the emergency room directly to the intensive care unit (ICU) were excluded from the study. These patients were excluded as a majority of them were not in a condition to describe their symptoms at the time of the presentation, and the severity of their illnesses often necessitated COVID-19 testing during empiric work up.

Data collection

For all eligible patients, records from initial hospital encounter were reviewed. Demographics (age, sex, race and smoking status), past medical history (diabetes, hypertension, chronic obstructive pulmonary disease (COPD), asthma, coronary artery disease (CAD), congestive heart failure (CHF), atrial fibrillation, chronic kidney disease (CKD) and end-stage renal disease (ESRD)), sick contact with suspected or confirmed COVID-19 case, presenting symptoms (fever, chills, myalgia, cough, shortness of breath, nausea, vomiting or diarrhoea), triage vital signs (temperature, heart rate, systolic and diastolic blood pressure, respiratory rate and oxygen saturation), initial basic laboratory test results (complete blood count and creatinine) and chest X-ray results were collected.

Development of prediction model

The dataset was randomly split into a development cohort and a validation cohort with a 6:4 ratio. Baseline characteristics were compared between cases and controls within each cohort. Categorical variables were compared using the χ2 test and continuous variables were compared using the Student t test. Univariable logistic regressions were performed in a development cohort to identify potential predictors of COVID-19 status. A multivariable logistic regression model was built in a stepwise fashion by a parsimonious approach. Variables associated with COVID-19 status in univariable analysis (P-value < 0.1) entered the model one by one in the order of their strength of association with COVID-19 diagnosis. Variables retained in the model if the addition of the variable improved the fit of the model (P-value < 0.05). Then we developed risk score models using the coefficients from the final logistic regression model. A total of three risk score models were created. The first model rounded the coefficient from the regression model and used them as a score for each variable. The second and third models simplified the coefficients to make it easy and practical to use. The risk scores were validated in the testing cohort. The area under the receiver operating characteristic curve (AUROC) was calculated for each risk score. The analyses were performed using STATA version 15.1 (STATA Corp., Texas, USA).

Ethical consideration

This study was approved by the Institutional Review Board (IRB) of the MedStar Health Research Institute with a waiver of individual consents. (IRB ID: MOD00004296)

Results

Study population

A total of 656 patients were admitted to the hospital during the study period. Of them, 79 patients who were admitted to the ICU directly from the emergency room were excluded from the study. Among 577 patients admitted to the medical floor, 338 patients received testing for COVID-19 and were included in this study. Of those included, 136 (40.2%) patients tested positive for SARS-CoV-2, and 202 (59.8%) patients tested negative. The demographic characteristics of the patients were described in Table 1. In the entire cohort, the median age was 65 years old (interquartile range (IQR) 54–76 years old), 52.7% were males, 82.8% were African Americans, 10.1% were Hispanics and 5.6% were Caucasians, 30.5% were current or former smokers, 14.2% were from skilled nursing facilities, 42.9% were obese, 64.5% had hypertension, 38.5% had diabetes, 21.0% had CKD, 16.0% had CHF and 12.7% had COPD.

Table 1.

Baseline characteristics of the patients admitted to the hospital in a development and a validation cohort

	Development cohort (n = 203)			Validation cohort (n = 135)
	Non-COVID (n = 116)	COVID (n = 87)	P-value	Non-COVID (n = 86)	COVID (n = 49)	P-value	Total (n = 338)
Median age (IQR), years	68.5 (55.5–77.5)	62 (50–75)	0.100	65 (53–76)	61 (55–75)	0.636	65 (54–76)
Sex, male	65 (56.0%)	41 (47.1%)	0.209	49 (57.0%)	23 (46.9%)	0.261	178 (52.7%)
Nursing facility residence	15 (12.9%)	19 (21.8%)	0.093	5 (5.8%)	9 (18.4%)	0.021	48 (14.2%)
Race
African American	97 (83.6%)	70 (80.5%)	0.022	71 (82.6%)	42 (85.7%)	0.071	280 (82.8%)
Hispanic	3 (2.6%)	11 (12.6%)		1 (1.2%)	4 (8.2%)		34 (10.1%)
Caucasian	14 (12.1%)	5 (5.7%)		12 (14.0%)	3 (6.1%)		19 (5.6%)
Other	2 (1.7%)	1 (1.1%)		2 (2.3%)	0 (0.0%)		5 (1.5%)
Smoking	39 (33.6%)	16 (18.4%)	0.016	34 (39.5%)	14 (28.6%)	0.201	103 (30.5%)
Obesity	36 (31.0%)	49 (56.3%)	<0.001	33 (38.4%)	27 (55.1%)	0.060	145 (42.9%)
Diabetes	51 (44.0%)	31 (35.6%)	0.231	27 (31.4%)	21 (42.9%)	0.181	130 (38.5%)
Hypertension	82 (70.7%)	53 (60.9%)	0.144	51 (59.3%)	32 (65.3%)	0.491	218 (64.5%)
COPD	19 (16.4%)	9 (10.3%)	0.217	11 (12.8%)	4 (8.2%)	0.411	43 (12.7%)
Asthma	5 (4.3%)	8 (9.2%)	0.159	2 (2.3%)	6 (12.2%)	0.019	21 (6.2%)
CAD	10 (8.6%)	7 (8.0%)	0.884	9 (10.5%)	7 (14.3%)	0.509	33 (9.8%)
CHF	18 (15.5%)	11 (12.6%)	0.563	18 (20.9%)	7 (14.3%)	0.339	54 (16.0%)
CKD	23 (19.8%)	15 (17.2%)	0.640	21 (24.4%)	12 (24.5%)	0.993	71 (21.0%)
ESRD	13 (11.2%)	10 (11.5%)	0.949	8 (9.3%)	5 (10.2%)	0.864	36 (10.7%)
Atrial fibrillation	11 (9.5%)	4 (4.6%)	0.188	4 (4.7%)	7 (14.3%)	0.049	26 (7.7%)
History of stroke	14 (12.1%)	6 (6.9%)	0.221	8 (9.3%)	7 (14.3%)	0.376	35 (10.4%)

aData are presented as median (IQR) for a continuous variable, and n (%) for categorical variables.

The dataset was split into development and validation datasets. In total, 203 patients were assigned to the development cohort, and among them, 87 (42.9%) patients were tested positive for COVID-19. In total, 135 patients were assigned to the validation cohort, and among them, 49 (36.3%) were tested positive. Baseline demographics and clinical characteristics of each cohort are summarised in Table 1 and Table 2. Nursing facility residence were more common in COVID-19 admissions as compared to non-COVID-19 admissions in both development cohort (P-value = 0.093) and validation cohort (P-value = 0.021). Diabetes, hypertension and COPD were more common in non-COVID-19 admissions as compared to COVID-19 admissions in both cohorts, but these differences were not statistically significant (P-value > 0.05).

Table 2.

Clinical presentation and initial work-up results in the emergency room

	Development cohort (n = 203)			Validation cohort (n = 135)
	Non-COVID (n = 116)	COVID (n = 117)	P-value	Non-COVID (n = 86)	COVID (n = 49)	P-value	Total (n = 338)
Sick contact	4 (3.4%)	25 (28.7%)	<0.001	2 (2.3%)	16 (32.7%)	<0.001	47 (13.9%)
Triage vital signs
Fever	12 (10.3%)	25 (28.7%)	<0.001	9 (10.5%)	11 (22.5%)	0.059	109 (32.3%)
Tachycardia	53 (45.7%)	47 (54.0%)	0.240	35 (40.7%)	25 (51.0%)	0.246	160 (47.3%)
Tachypnoea	42 (36.2%)	38 (43.7%)	0.281	31 (36.1%)	22 (44.9%)	0.311	133 (39.4%)
Hypotension	16 (13.8%)	17 (19.5%)	0.272	8 (9.3%)	5 (10.2%)	0.864	46 (13.6%)
Hypoxia	33 (28.4%)	55 (63.2%)	<0.001	30 (34.9%)	29 (59.2%)	0.006	147 (43.5%)
Presenting symptoms
Respiratory symptoms	60 (51.7%)	74 (85.1%)	<0.001	49 (57.0%)	38 (77.6%)	0.016	221 (65.4%)
Cough	31 (26.7%)	61 (70.1%)	<0.001	29 (33.7%)	32 (65.3%)	<0.001	153 (45.3%)
Shortness of breath	54 (46.6%)	59 (67.8%)	0.003	42 (48.8%)	34 (69.4%)	0.021	189 (55.9%)
Constitutional symptoms	28 (24.1%)	55 (63.2%)	<0.001	27 (31.4%)	24 (49.0%)	0.043	134 (39.6%)
Fever	23 (19.8%)	47 (54.0%)	<0.001	15 (17.4%)	24 (49.0%)	<0.001	109 (32.3%)
Chills	10 (8.6%)	21 (24.1%)	0.002	10 (11.6%)	9 (18.4%)	0.279	50 (14.8%)
Myalgia	7 (6.0%)	13 (14.9%)	0.035	11 (12.8%)	4 (8.2%)	0.411	35 (10.4%)
Chest pain	20 (17.2%)	15 (17.2%)	1.000	14 (16.3%)	9 (18.4%)	0.756	58 (17.2%)
Gastrointestinal symptom	17 (14.7%)	33 (37.9%)	<0.001	20 (23.3%)	21 (42.9%)	0.017	91 (26.9%)
Nausea or vomiting	11 (9.5%)	21 (24.1%)	0.005	14 (16.3%)	14 (28.6%)	0.090	60 (17.8%)
Diarrhoea	9 (7.8%)	22 (25.3%)	<0.001	8 (9.3%)	13 (26.5%)	0.008	52 (15.4%)
Initial labs
Leukopenia	5 (4.3%)	5 (5.7%)	0.640	7 (8.1%)	11 (22.5%)	0.019	28 (8.3%)
Leucocytosis	32 (27.6%)	15 (17.2%)	0.084	20 (23.3%)	5 (10.2%)	0.060	72 (21.3%)
Thrombocytopenia	18 (15.5%)	19 (21.8%)	0.248	10 (11.6%)	11 (22.5%)	0.095	58 (17.2%)
Thrombocytosis	11 (9.5%)	6 (6.9%)	0.510	11 (12.8%)	1 (2.0%)	0.035	29 (8.6%)
Creatinine	1.19 (0.88–1.95)	1.12 (0.81–1.82)	0.731	1.18 (0.82–2.52)	1.33 (1.01–1.87)	0.819	1.2 (0.86–1.95)
Initial chest X-ray
Clear lung field	43 (40.6%)	15 (17.9%)	<0.001	30 (38.0%)	13 (28.3%)	0.270	101 (32.1%)
Possible multifocal infiltrate	33 (31.1%)	43 (51.2%)	0.005	31 (39.2%)	25 (54.4%)	0.101	132 (41.9%)

Eight variables were included in the final multivariable model. Sick contact and nursing facility residence were the two biggest risk factors for COVID-19, followed by respiratory symptom (cough or shortness of breath), gastrointestinal symptom (nausea, vomiting or diarrhoea), hypoxia at triage, obesity and constitutional symptom (fever, chills or myalgia). Leucocytosis was negatively associated with COVID-19 (Table 3). Based on the coefficients from the final model, three risk scores were created (Table 4). The first model used the rounded value of the coefficients as a score for each variable. Sick contact and nursing facility residence were assigned 10 points each, and obesity was assigned 3 points. The constitutional symptom was assigned 2 points, and respiratory symptom, gastrointestinal symptom and hypoxia on triage were assigned 4 points each. Leucocytosis was assigned minus 3 points. This risk score achieved AUROC of 0.87 (95% CI 0.83–0.92) in the development cohort and 0.85 (95% CI 0.78–0.92) in the validation cohort. Simplified risk scores were created for practical use (models 2 and 3). For the second model, our simplest model, sick contact and nursing facility residence were assigned 2 points each; obesity, respiratory symptom, gastrointestinal symptom, constitutional symptom and hypoxia at triage were assigned 1 point each; and leucocytosis was assigned minus 1 point. The second model achieved AUROC of 0.87 (95% CI 0.83–0.92) in the development dataset and 0.83 (95% CI 0.76–0.90) in the validation dataset. Our final model, model 3, modified model 2 by assigning 3 points each to sick contact and nursing facility residence, and it achieved AUROC of 0.87 (95% CI 0.82–0.92) in the development dataset and 0.85 (95% CI 0.78–0.92) in the validation dataset. The receiver operating characteristic curve for the final model, model 3, is presented in Figure 1. Using the final model, a risk score of ⩾3 achieved a sensitivity of 85.1% and a specificity of 75.0% in the development cohort, and a sensitivity of 79.6% and a specificity of 70.9% in the validation cohort. Positive and negative predictive values were 71.8% and 87.0% in the development cohort and 60.9% and 85.9% in the validation cohort, respectively. Sensitivities, specificities and positive and negative predictive values for different cut-off values are presented in Table 5.

Table 3.

Results of univariable analysis and multivariable analysis

	Univariable analysis			Multivariable analysis
	Crude OR	95% CI	P-value	Adjusted OR	95% CI	P-value
Demographics
Age (year)	0.99	0.97–1.00	0.102
Male sex	0.70	0.40–1.22	0.209
Nursing facility residence	1.88	0.89–3.96	0.096	9.63	3.02–30.67	<0.001
Diabetes	0.71	0.40–1.25	0.232
Hypertension	0.65	0.36–1.16	0.145
Chronic kidney disease	0.84	0.41–1.73	0.640
Obesity	2.87	1.61–5.11	<0.001	2.93	1.32–6.51	0.008
Smoking	0.44	0.23–0.87	0.017
Triage vital signs
Fever	3.49	1.64–7.45	0.001
Tachycardia	1.40	0.80–2.44	0.240
Hypotension	1.52	0.72–3.21	0.274
Tachypnoea	1.37	0.77–2.41	0.282
Hypoxia	4.32	2.39–7.83	<0.001	3.52	1.58–7.83	0.002
Sick contact	11.29	3.76–33.92	<0.001	10.47	2.67–41.04	0.001
Presenting symptoms
Constitutional symptom	5.4	2.94–9.93	<0.001	2.31	1.05–5.10	0.038
Respiratory symptom	5.31	2.66–10.62	<0.001	4.36	1.73–10.98	0.002
Gastrointestinal symptom	3.56	1.82–6.97	<0.001	4.11	1.59–10.64	0.004
Initial labs
Leukopenia	1.35	0.38–4.83	0.641
Leucocytosis	0.55	0.27–1.09	0.086	0.33	0.12–0.90	0.030
Thrombocytopenia	1.52	0.74–3.11	0.250
Thrombocytosis	0.71	0.25–1.99	0.512
Initial chest X-ray
Clear lung field	0.32	0.16–0.63	0.001
Possible multifocal infiltrate	2.32	1.28–4.20	0.005

Table 4.

COVID-19 risk scores and corresponding AUROCs

		Model 1	Model 2	Model 3
Sick contact with suspected or confirmed COVID-19 case		+10 points	+2 points	+3 points
Nursing Facility Residence		+10 points	+2 points	+3 points
Obesity		+3 points	+1 point	+1 point
Constitutional symptom (fever, chills, or myalgia)		+2 points	+1 point	+1 point
Respiratory symptom (cough or shortness of breath)		+4 points	+1 point	+1 point
Gastrointestinal symptom (nausea, vomiting, or diarrhoea)		+4 points	+1 point	+1 point
Hypoxia on triage		+4 points	+1 point	+1 point
Leucocytosis		−3 points	−1 point	−1 point
Total points		37 points	9 points	11 points
AUROC (95% CI)	Development cohort	0.87 (0.83–0.92)	0.87 (0.83–0.92)	0.87 (0.82–0.92)
	Validation cohort	0.85 (0.78–0.92)	0.83 (0.76–0.90)	0.85 (0.78–0.92)

Fig. 1.

Receiver operating characteristic curves of the COVID-19 risk score (model 3) among patients admitted to the hospital from the emergency room in a development cohort (left) and a validation cohort (right).

Table 5.

Sensitivities, specificities, positive predictive values (PPVs) and negative predictive value (NPVs) of the final risk score system for each prediction score cut-off

	Development cohort				Validation cohort
Prediction score	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
⩾0	100.0	5.2	44.2	100.0	100.0	3.5	37.1	100.0
⩾1	98.9	20.7	48.3	96.0	98.0	16.3	40.0	93.3
⩾2	97.7	47.4	58.2	96.5	93.9	37.2	46.0	91.4
⩾3	85.1	75.0	71.8	87.0	79.6	70.9	60.9	85.9
⩾4	64.4	88.8	81.2	76.9	69.4	91.9	82.9	84.0
⩾5	46.0	93.1	83.3	69.7	44.9	96.5	88.0	75.5
⩾6	27.6	98.3	92.3	64.4	28.6	97.7	87.5	70.6
⩾7	14.9	100.0	100.0	61.1	12.2	4.4	100.0	66.7

Discussion

Early and proper identification and isolation of suspected patients with COVID-19 are essential to allow timely treatment, conserve resources, protect patients and healthcare staff, and to avoid the spread of COVID-19 in healthcare facilities [4]. In a resource-limited setting, it is critically important to risk-stratify patients who need testing. In this study, we present a novel bedside score developed to aid the diagnosis of COVID-19. Prior studies suggested a few logistic regression models predicting COVID-19 diagnosis [3, 5, 6]. However, logistic models are not practical to implement in clinical practice due to complex mathematical calculations needed to perform the prediction. We simplified our prediction model by creating a scoring system that is simple and practical for use and internally validated its utility. Our simplified final risk score (model 3) achieved AUROC comparable to the first model that used rounded values of coefficients from the regression model as scores, highlighting the utility of a simple bedside score. We only included variables that are readily available at the initial hospital encounter to enable practical implementation.

In our analysis, sick contact with suspected or confirmed COVID-19 cases and nursing facility residence were found to be two major risk factors for COVID-19 among newly admitted patients, and they were assigned 3 points each in the final model. Nursing facility residents are a high-risk group for COVID-19. They reside in a healthcare setting with an increased risk of contracting the virus, and they are often elderly with multiple underlying medical conditions and therefore at increased risk of worse outcomes from COVID-19. Thus, active screening of this high-risk group is critical. Using a cut-off of three in our final model will allow screening all patients admitted to the hospital from nursing facilities and those who had sick contact with presumed COVID-19 cases. Male sex and chronic medical conditions such as diabetes and hypertension were known to be associated with the worse outcome from COVID-19 [7-10]. However, in this study, we did not find significant differences in the proportion of males, hypertensives and diabetics between COVID-19-positive admissions and negative admissions. Interestingly, hypertension and diabetes were more common in non-COVID-19 admissions than in COVID-19 admissions in both development and validation cohorts. This is likely because our control group was patients who were admitted to the hospital with non-COVID-19 causes, and hypertension and diabetes are important risk factors for hospitalisations from non-COVID-19 reasons such as cardiovascular conditions. Obesity is another risk factor for worse clinical outcomes in COVID-19 [9, 10]. In this study, newly admitted patients with obesity were more likely to have COVID-19. In addition to demographic risk factors and symptom scores, we also identified that patients with leucocytosis were less likely to have COVID-19. While COVID-19 is known to be associated with leukopenia and thrombocytopenia, they were not independent predictors of COVID-19 diagnosis in our cohort [11, 12].

In summary, the indicators used in this risk score stratification are readily available at the time of admission into hospitals throughout the nation. The ability to quickly and appropriately risk stratify and identify suspected patients requiring quarantine and testing may allow physicians to make appropriate decisions in terms of early diagnosis and management.

Nonetheless, our study has limitations. Our study is limited by small cohort size. In this study, we did not find a significant association between chest X-ray findings and COVID-19 status after adjusting for the effects of confounders. This may be due to our small sample size; however, as our study excluded patients who were directly admitted to the ICU from the emergency room, chest X-ray may not be a very strong predictor of COVID-19 among newly hospitalised patients who are not critically ill. Inflammatory markers such as d-dimer, C-reactive protein and ferritin were reported to be often elevated in COVID-19 but these lab values were not available for many study patients and therefore not included in the model [13, 14]. As our scoring system was developed in a cohort of patients admitted to the regular medical floor from the emergency room, and our study result cannot be generalised to other settings, such as outpatient practices, urgent cares or ICUs. SARS-CoV-2 is also known to cause asymptomatic infection, and our score system is designed to risk-stratify newly admitted patients with symptoms concerning for COVID-19 infection, therefore cannot be used to identify asymptomatic patients. Given the above limitations and single-centre study design, the risk score should be further validated in larger and/or multicentre studies.

Conclusion

In summary, we developed a simple, easy-to-implement bedside scoring system for COVID-19 risk stratification among patients who are being admitted to the hospital. The risk score system achieved AUROC of 0.85 in validation and can be used as a supplemental tool to assist clinical decision in the triage, quarantine and testing of patients admitted to the hospital with suspicion of COVID-19 infection.