Validation of a clinical risk-scoring algorithm for severe scrub typhus.

OBJECTIVE: The aim of the study reported here was to validate the risk-scoring algorithm for prognostication of scrub typhus severity.
METHODS: The risk-scoring algorithm for prognostication of scrub typhus severity developed earlier from two general hospitals in Thailand was validated using an independent dataset of scrub typhus patients in one of the hospitals from a few years later. The predictive performances of the two datasets were compared by analysis of the area under the receiver-operating characteristic curve (AuROC). Classification of patients into non-severe, severe, and fatal cases was also compared.
RESULTS: The proportions of non-severe, severe, and fatal patients by operational definition were similar between the development and validation datasets. Patient, clinical, and laboratory profiles were also similar. Scores were similar in both datasets, both in terms of discriminating non-severe from severe and fatal patients (AuROC =88.74% versus 91.48%, P=0.324), and in discriminating fatal from severe and non-severe patients (AuROC =88.66% versus 91.22%, P=0.407). Over- and under-estimations were similar and were clinically acceptable.
CONCLUSION: The previously developed risk-scoring algorithm for prognostication of scrub typhus severity performed similarly with the validation data and the first dataset. The scoring algorithm may help in the prognostication of patients according to their severity in routine clinical practice. Clinicians may use this scoring system to help make decisions about more intensive investigations and appropriate treatments.

Introduction

“Scrub typhus,” one of several potentially fatal tropical rickettsial infections, is caused by Orientia tsutsugamushi and presents as a systemic, vasculitis-like infection. Clinical manifestations are typically fever, eschar, generalized or regional lymphadenopathy, maculopapular rash, severe headache, and myalgia. Nausea and vomiting, diarrhea, and conjunctival injection can also occur.1,2 Without appropriate treatment, infected patients usually experience complications in the second week. Those especially at risk of infection are the elderly, those without previous immunity,2–4 and those with risk factors such as hypoalbuminemia, hyperbilirubinemia, high levels of aspartate aminotransferase (AST) and/or creatinine, and those with lung crepitations.3,5–10 Patients with complications who are admitted to intensive care units with a high Acute Physiology and Chronic Health Evaluation II (APACHE II) score are at high risk of mortality.5 In rural areas where effective treatment is unavailable or delayed, mortality rates have been reported as high as 35%.2

The cost of treatment for scrub typhus patients with complications is much higher due to hospital admissions and more intensive treatments.11 Prognostication of disease severity may help with the identification of disease severity and prompt initiation of treatments.

Prognostication of disease severity has been considered by researchers of various conditions. Examples include a clinical prediction rule for severe community-acquired pneumonia,12 Mortality in Emergency Department Sepsis score,13 a prediction rule for risk of mortality in Clostridium difficile infection,14 and decision-tree algorithms for dengue fever15 and dengue hemorrhagic fever.16 Some of these tools were found to have both advantages and disadvantages when validated.

Previously, we developed a clinical risk-scoring algorithm for scrub typhus severity comprising six predictors from routine practice.17 The objective of the study reported here was to validate the risk-scoring algorithm developed earlier with an independent dataset obtained from similar patients from a few years later.

Materials and methods

Patients

Data were retrieved retrospectively from the medical files of scrub typhus patients. The criteria for inclusion were similar to those used in the previous study17 and were based on history of disease exposure and presentation with acute fever and at least one of the following signs and symptoms: myalgia, headache, conjunctival injection, cough, profuse sweating, maculopapular rash, and/or lymphadenopathy accompanied by the presence of eschar, and/or a positive immuno-chromatographic test for scrub typhus.18

Patients were classified into three groups:

non-severe – patients without any complications

severe – patients with complications involving at least one of the following organ systems:

cardiovascular system – presence of any of any of the following: systolic blood pressure <90 mmHg, myocarditis (defined as elevated creatine kinase-myocardial band isoenzyme above baseline), or abnormal cardiac arrhythmia with no previous history of atrial fibrillation, supra ventricular tachycardia, or frequent premature ventricular tachycardia

respiratory system – presence of acute respiratory distress syndrome, defined as the ratio of partial pressure arterial oxygen and fraction of inspired oxygen (PaO2/FiO2) <200 mmHg, with bilateral interstitial infiltration on chest film with normal cardio/thoracic ratio, or no volume overload of central venous pressure from central venous catheter

central nervous system – presence of any of the following: Glasgow Coma Scale ≤12 without other causes, seizure without other causes, or meningoencephalitis

hematological system – platelet count ≤20,000/mm3

urinary system – presence of acute renal failure, defined as creatinine ≥2mg/dL or creatinine change of >0.5 mg/dL/day

gastrointestinal and hepatobiliary system – presence of hepatitis, defined as elevated AST or alanine aminotransferase (ALT) more than fivefold normal levels.

fatal – patients who died in hospital from scrub typhus.

Patients in any intervention trials during the same period were excluded from the data analysis.

Development data

The development data were for patients admitted to two university-affiliated referral hospitals – Nakornping Hospital, Chiang Mai and Chiangrai Prachanukroh Hospital – in the north of Thailand from 2004 to 2010 (n=526).

Validation data

The validation data were for patients admitted to Chiangrai Prachanukroh Hospital from 2011 to 2012 (n=257).

Data analysis

The characteristics of the development data and the validation data were compared using an exact probability test, Student’s t-test, or Wilcoxon rank-sum test as appropriate. The patient risk scores were assigned following the scoring scheme previously developed (Table 1).17 Patients were categorized into three severity groups: 1) non-severe, 2) severe, and 3) fatal, as described. The predictive performance of the scores was compared by analysis of the area under the receiver-operating characteristic curve (AuROC). Over- and underestimated proportions were calculated and compared.

Table 1

Significant predictors and assigned item scores

Predictor	Category	Assigned score
Age (years)	≤15	0
	>15	3
Pulse rate (/minute)	≤100	0
	>100	2
Crepitation	No	0
	Yes	2
AST (IU/L)	≤160	0
	>160	2
Serum albumin (g/dL)	≤3.0	3
	>3.0	0
Serum creatinine (mg/dL)	≤1.4	0
	>1.4	4

Abbreviation: AST, aspartate aminotransferase.

Ethical approval

The study was approved by the Ethics Committee for Research in Patients, Faculty of Medicine, Chiang Mai University, and the research ethical committees of the two hospitals.

Results

Data for 257 patients comprised the validation dataset. Of those, 160 cases were defined as non-severe, 62 as severe, and 35 as fatal. The distribution of patients into the three severity groups of the development and the validation datasets was similar. The clinical characteristics of the patients were also similar except in terms of 1) sex, 2) symptoms or signs of conjunctival injection, maculopapular rash, lymphadenopathy, eschar, abdominal pain, stiff neck, and crepitation, 3) vital signs of systolic blood pressure, and diastolic blood pressure (Table 2). The laboratory investigation data also showed similar profiles for both datasets, except in terms of platelet count and AST, alanine aminotransferase, alkaline phosphatase, albumin, and potassium levels (Table 3).

Table 2

Demographic and clinical characteristics of scrub typhus patients in the development and validation datasets

Characteristics	Development, n=526 (n [%])	Validation, n=257 (n [%])	P-value
Demographic and history
Male	291 (55.3)	165 (64.2)	0.021
Age (years) (mean ± SD)	29.3±22.0	31.0±21.4	0.115
Underlying diseases	52 (9.9)	23 (8.9)	0.796
Fever duration (days) (mean ± SD)	6.5±3.2	6.6±3.8	0.151
Length of hospital stay (days) (mean ± SD)	4.1±2.6	4.8±3.9	0.246
Definition-classified severity
Non-severe	357 (67.9)	160 (62.3)	0.219
Severe	100 (19.0)	62 (24.1)
Fatal	69 (13.1)	35 (13.6)
Clinical characteristic
Headache	160 (30.4)	96 (37.5)	0.062
Myalgia	105 (20.0)	61 (23.7)	0.228
Conjunctival injection	484 (92.0)	185 (72.0)	<0.001
Maculopapular rash	485 (92.2)	187 (72.8)	<0.001
Lymphadenopathy	373 (70.9)	154 (59.9)	0.003
Eschar	290 (55.1)	41 (15.6)	<0.001
Abdominal pain	178 (33.8)	65 (25.3)	0.017
Nausea, vomiting	122 (23.2)	75 (29.2)	0.079
Diarrhea	102 (19.4)	38 (14.8)	0.136
Hepatomegaly	163 (31.0)	67 (26.1)	0.181
Splenomegaly	53 (10.1)	20 (7.8)	0.360
Jaundice	49 (9.3)	20 (7.8)	0.506
Seizure	26 (4.9)	15 (5.8)	0.611
Stiff neck	14 (2.7)	17 (6.6)	0.011
Crepitation	56 (10.7)	44 (17.1)	0.012
Wheezing	15 (2.9)	5 (2.0)	0.630
Cough	169 (32.1)	88 (34.2)	0.571
Dyspnea	39 (7.4)	29 (11.3)	0.079
Respiration (/minute) (mean ± SD)	25.4±8.8	25.3±10.5	0.964
Pulse (/minute) (mean ± SD)	104.0±22.1	102.1±20.4	0.237
SBP (mmHg) (mean ± SD)	102.8±17.6	106±21.0	0.012
DBP (mmHg) (mean ± SD)	63.4±12.1	66.5±15.7	0.005
Body temperature (°C) (mean ± SD)	38.2±1.2	38.3±2.9	0.496

Abbreviations: DBP, diastolic blood pressure; SBP, systolic blood pressure; SD, standard deviation.

Table 3

Laboratory test findings

Characteristics	Development, n=526(mean ± SD)	Validation, n=257(mean ± SD)	P-value
Hematological
WBC (×103/mm3)	9.9±5.4	10.9±10.6	0.288
Neutrophils (%)	70.2±17.1	71.3±15.8	0.387
Lymphocytes (%)	21.1±14.6	20.1±13.6	0.525
Monocytes (%)	6.1±7.7	6.0±6.3	0.464
Platelets (×103/mm3)	133.1±107.8	167.2±127.5	<0.001
Hematocrit (%)	35.2±6.2	36.5±6.3	0.498
Hemoglobin (g/dL)	11.8±2.1	12.1±2.2	0.156
Biochemistry
BUN (mg/dL)	27.6±26.7	26.7±26.4	0.681
Creatinine (mg/dL)	1.7±1.9	1.8±2.1	0.431
Total bilirubin (mg/dL)	2.6±4.4	1.9±2.5	0.348
Direct bilirubin (mg/dL)	1.3±2.3	0.8±1.2	0.175
Albumin (g/dL)	2.7±0.6	3.1±0.7	0.014
Globulin (g/dL)	3.3±0.7	3.4±0.7	0.063
ALT (IU/L)	125.2±259.7	143.1±505.3	<0.001
AST (IU/L)	215.0±459.5	375.1±2,528.4	<0.001
ALP (IU/L)	251.3±193.5	201.5±164.6	<0.001
Na (mmol/L)	133.3±4.7	134.0±10.8	0.265
K (mmol/L)	3.6±0.6	3.8±2.1	0.028
Cl2 (mmol/L)	100.7±5.6	101.4±6.0	0.163
CO2 (mmol/L)	21.2±4.7	21.8±5.2	0.121
Urine albumin (n [%])
<1+	242 (54.6)	118 (51.5)	0.463
≥1+	201 (45.4)	111 (48.5)
Urine sugar (n [%])
<1+	435 (98.2)	223 (97.4)	0.571
≥1+	8 (1.8)	6 (2.6)

Notes: The data were missing for some parameters: in the development set, urine albumin and urine sugar =83 cases; in the validation set, AST =42 cases, albumin =68 cases, creatinine =48 cases, and urine albumin and urine sugar =28 cases.

Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; AST, aspartate aminotransferase; BUN, blood urea nitrogen; Cl2, chloride; CO2, carbon dioxide; K, potassium; Na, sodium; SD, standard deviation; WBC, white blood cell.

The risk score was assigned to patients with six complete clinical parameters (n=167, 65%). The mean risk scores of patients in the validation data were different to those of patients in the development dataset (mean ± sd 7.1±3.9 and 6.0±3.9, respectively). The proportion of patients classified into the three severity levels was also different (P=0.004; Table 4).

Table 4

Score-classified scrub typhus severity levels in the development and validation datasets

Score-classified severity levels	Development(n=526)	Validation(n=167)	P-value
Mean score ± SD	6.0±3.9	7.1±3.9	0.002
Severity levels (n, %)
Non-severe	278 (52.8)	64 (38.3)	0.004
Severe	143 (27.2)	56 (35.5)
Fatal	105 (20.0)	47 (28.1)

Abbreviation: SD, standard deviation.

In the validation dataset, the score correctly predicted 31.7% (53 out of 78) of patients into the non-severe group (score ≤5) with one-level underestimation of 6.6% (eleven patients) and no two-level underestimation, while 14.4% (24 out of 55) of patients were correctly classified into the severe group (score 6–9), with an underestimation of 5.4% (nine patients) and an overestimation of 13.8% (23 patients). Finally, for the fatal severity group (score ≥10), the score was predicted correctly for 14.9% (25 of 34) of patients, with one-level overestimation of 12.0% (20 patients) and two-level overestimation of 1.2% (2 patients), giving a total overestimation of 13.2% (Table 5).

Table 5

Score-classified severity levels, criterion-classified severity levels, and risk estimation validity

Score-classified severity levels	Score range	Criterion-classified severity levels			Risk estimation validity*
		Non-severe, n=78	Severe, n=55	Fatal, n=34	Over (%)	Correct (%)	Under (%)
Mean ± SD		4.4±2.4	8.4±3.4	11.4±2.5
IQR		3–6	7–12	9–14
Non-severe, n=64	≤5	53	11	0	–	31.7	6.6
Severe, n=56	6–9	23	24	9	13.8	14.4	5.4
Fatal, n=47	≥10	2	20	25	13.2	14.9	–
Total					27.5	61.0	12.0

Note:

Percentage of total patients.

Abbreviations: IQR, interquartile range; SD, standard deviation.

The overall predictive performance (as assessed by the AuROCs) of the validation dataset was insignificantly lower than that of the development dataset, both in the discrimination between non-severe versus (vs) severe and fatal (AuROC =88.74% vs 91.48%, P=0.324) and between fatal versus non-severe and severe (AuROC =88.66% vs 91.22%, P=0.407; Table 6 and Figure 1).

Table 6

Discriminative performance of scrub typhus severity score in the development and validation datasets

Prediction/discrimination	Development (n=526)		Validation (n=167)		P-value
	AuROC (%)	95% CI	AuROC (%)	95% CI
Non-severe versus severe + fatal	91.48	88.58–94.10	88.74	83.78–93.71	0.324
Non-severe + severe versus fatal	91.22	88.23–94.20	88.66	83.52–93.79	0.407

Abbreviations: AuROC, area under the receiver-operating characteristic curve; CI, confidence interval.

Figure 1

Score-predicted probability of severity in the development data (solid lines) and the validation data (dashed lines).

Note: Vertical dotted lines represent score-derived criteria for classifying patients into non-severe, severe, and fatal groups.

Discussion

Validation is needed for any clinical prediction rules before they can be adopted in clinical practice, either internally or externally. The Mortality in Emergency Department Sepsis score, for example, was developed to evaluate the risk of death in patients in emergency departments, in which validation in one-third of the sample yielded similar discrimination (AuROC =0.82 vs 0.78).13 Further, a clinical prediction rule for severe community-acquired pneumonia was developed to screen patients for initial treatments in emergency departments. The rule performed similarly to the development data when prospectively validated in patients from the same hospitals (AuROC =0.92). However, the performance was poorer when externally validated in patients from other hospitals (AuROC =0.80).12 External validation of a prediction rule for risk of mortality in Clostridium difficile infection also yielded poorer performance (AuROC =0.704 vs 0.653).14 It is very common for the performance of the validation data to be poorer than that of the development data.

Our clinical risk-scoring algorithm for severe scrub typhus was validated against patients admitted in only one hospital some years after the patients used in the development dataset. When only patients with complete information on the six predictors (n=167) were selected from the validation dataset, the performance of the algorithm was somewhat poorer than with the development data, both in terms of differentiating non-severe from severe and fatal (AuROC =88.74% vs 91.48%), and fatal from non-severe and severe (AuROC=88.66% vs 91.22%). However, these differences were not statistically or clinically significant. The overall proportion of correct classifications was somewhat lower (61.0% vs 68.3%), but overestimation was higher (27.0% vs 25.8%), which may be considered more beneficial to the patients.17

There were incomplete data on important predictors for 90 of 257 patients (35%). The missing data were AST (n=42, 24.1%), albumin (n=68, 26.5%), and creatinine (n=48, 18.7%). These patients were in the non-severe group, which did not require organ-specific investigations. We did the sensitivity analysis by replacing the missing data by the mean values calculated from existing nonmissing data for each of the corresponding variables. When this was done, the discrimination between non-severe and severe and fatal was reduced (AuROC =84.41%) but was still considered excellent,19 while the discrimination between fatal and non-severe and severe was unchanged. However, sensitivity analysis by multiple imputation of the missing data yielded almost identical performances (data not shown).

These sensitivity analyses imply that the derived risk-scoring algorithm might be successfully and safely applied to classify future patients, especially those with severe and fatal risk of scrub typhus. The score should help physicians in charge determine the need for further investigations and start the necessary treatments for those at these risk levels.

The main advantage of the risk-scoring algorithm is that it requires routinely available patient information, comprising demographic, physical examination, and laboratory test data.17 In endemic areas with restricted health resources,20–22 the score may help inform clinical decision making with regard to both investigations and treatments.

In the present study, data on the validation patients were collected retrospectively and were based on provisional diagnosis made as part of routine clinical practice, in which scrub typhus cases may reflect “suspected” rather than “confirmed” cases, by definition. Therefore, prospective validation to patients with a theoretically defined diagnosis of scrub typhus18 would be helpful to confirm the algorithm’s feasibility for future use in routine practice.

Conclusion

A risk-scoring algorithm developed previously to classify scrub typhus patients into the three severity levels was validated using a different set of patients. The algorithm demonstrated good performance in this subsequent set of patients. The algorithm was clinically acceptable and could be applied in routine clinical practice to help identify patients with severe prognosis in order to judge the need for more intensive investigation and treatments.