A clinical score for identifying active tuberculosis while awaiting microbiological results: Development and validation of a multivariable prediction model in sub-Saharan Africa.

BACKGROUND: In highly resource-limited settings, many clinics lack same-day microbiological testing for active tuberculosis (TB). In these contexts, risk of pretreatment loss to follow-up is high, and a simple, easy-to-use clinical risk score could be useful. METHODS AND
FINDINGS: We analyzed data from adults tested for TB with Xpert MTB/RIF across 28 primary health clinics in rural South Africa (between July 2016 and January 2018). We used least absolute shrinkage and selection operator regression to identify characteristics associated with Xpert-confirmed TB and converted coefficients into a simple score. We assessed discrimination using receiver operating characteristic (ROC) curves, calibration using Cox linear logistic regression, and clinical utility using decision curves. We validated the score externally in a population of adults tested for TB across 4 primary health clinics in urban Uganda (between May 2018 and December 2019). Model development was repeated de novo with the Ugandan population to compare clinical scores. The South African and Ugandan cohorts included 701 and 106 individuals who tested positive for TB, respectively, and 686 and 281 randomly selected individuals who tested negative. Compared to the Ugandan cohort, the South African cohort was older (41% versus 19% aged 45 years or older), had similar breakdown of biological sex (48% versus 50% female), and had higher HIV prevalence (45% versus 34%). The final prediction model, scored from 0 to 10, included 6 characteristics: age, sex, HIV (2 points), diabetes, number of classical TB symptoms (cough, fever, weight loss, and night sweats; 1 point each), and >14-day symptom duration. Discrimination was moderate in the derivation (c-statistic = 0.82, 95% CI = 0.81 to 0.82) and validation (c-statistic = 0.75, 95% CI = 0.69 to 0.80) populations. A patient with 10% pretest probability of TB would have a posttest probability of 4% with a score of 3/10 versus 43% with a score of 7/10. The de novo Ugandan model contained similar characteristics and performed equally well. Our study may be subject to spectrum bias as we only included a random sample of people without TB from each cohort. This score is only meant to guide management while awaiting microbiological results, not intended as a community-based triage test (i.e., to identify individuals who should receive further testing).
CONCLUSIONS: In this study, we observed that a simple clinical risk score reasonably distinguished individuals with and without TB among those submitting sputum for diagnosis. Subject to prospective validation, this score might be useful in settings with constrained diagnostic resources where concern for pretreatment loss to follow-up is high.

Introduction

The World Health Organization (WHO) estimates that 10.0 million new tuberculosis (TB) cases and 1.4 million deaths occurred in 2018, making TB the leading single-agent cause of infectious mortality worldwide [1] Pretreatment loss to follow-up is a major contributor to TB morbidity and transmission: an estimated 13% to 18% of people diagnosed with TB in high-burden settings are lost to follow-up before starting treatment [2]. Rapid treatment of TB is difficult in settings where access to on-site diagnostic testing, radiography, and specialist staff is limited. Empirical (or clinical) TB diagnosis—in other words, diagnosis made without microbiological confirmation—is therefore an important consideration in these settings where the risk of loss to follow-up is high. However, empirical diagnosis is not standardized, may not be routinely made by midlevel clinicians who frequently staff such clinics, and does not correlate well with microbiological results—often leading to inappropriate therapy [3,4]. Simple tools to improve the process of empirical TB diagnosis in severely resource-constrained settings could therefore substantially reduce Mycobacterium tuberculosis transmission and TB mortality.

Molecular tests for TB, including Xpert MTB/RIF (Xpert; Cepheid, Sunnyvale, California, United States of America) and Xpert Ultra, have improved TB diagnosis but remain unavailable in many peripheral clinics, despite the availability of newer single-module systems such as Xpert Omni and Xpert Edge [5-7]. Even where available, results from molecular tests are often not received by the treating clinician on the same day. Most randomized trials of Xpert on clinical outcomes have been conducted in settings where same-day testing and ancillary diagnostic procedures (e.g., chest X-ray) were readily available [7-9]. However, a very common clinical scenario is one in which a patient with presumptive TB is unlikely to receive same-day radiological or microbiological test results and is at high risk of loss to follow-up if treatment is not initiated immediately [8,10,11]. In such settings, a simple clinical score that could rapidly identify high-risk patients for consideration of anti-TB treatment would be exceedingly helpful.

Most existing models for predicting active TB require equipment or infrastructure (e.g., radiology, laboratory testing, and calculations requiring a computer or smartphone) that is generally unavailable in clinical settings that also lack same-day microbiological testing for TB [12-15]. Thus, there exists a need for a prediction score that is simple enough to write on paper and calculate rapidly by hand. We aimed to develop such a simple score for predicting active TB, requiring only information likely to be accessible to clinicians in such resource-limited settings. To ensure that this score could be widely applicable across sub-Saharan Africa, we developed the score in 1 epidemiological context (rural South Africa) and validated it in a very different one (urban Uganda).

Methods

Study setting and population

We derived our clinical score using data from the Kharitode study, a cluster-randomized controlled trial of TB contact investigation strategies (household and incentive-based contact investigation) against facility-based screening (the standard of care) across 56 primary care clinics in 1 rural and 1 semi-urban district of Limpopo Province, South Africa [16]. Limpopo Province has a low population density (46.1 people per km2 in 2016) [17], high TB incidence (301 per 100,000/year in 2015), and high adult HIV prevalence (17%) [18]. The median household income in this study population was approximately $150 per month [19].

For the current analysis, we included participants above 15 years of age who presented with TB symptoms at the 28 primary health clinics randomized to facility-based (i.e., passive) screening between July 2016 and January 2018. According to standard practice in South Africa, all clinic attendees were screened for TB symptoms, and symptomatic individuals were referred for Xpert testing. Among the symptomatic individuals, those who provided informed consent were interviewed about demographic and clinical details and provided follow-up contact information [16]. Our definition of active TB was based on an Xpert–positive sputum specimen. Xpert has imperfect sensitivity for pulmonary TB (estimated 88%) but high specificity (98%), such that the positive predictive value of Xpert is expected to be high [20]. Data were collected on all consenting Xpert–positive cases and a random sample of Xpert–negative individuals designed to be representative of the entire Xpert–negative population. Individuals who did not have an interpretable Xpert result or were treated for TB despite a negative Xpert result (<1%) were excluded from analysis.

Model specification and score generation

This study is reported as per the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guideline developing, validating, or updating a prediction model (S1 Checklist). We considered variables for inclusion in our model that met the following criteria: (1) existing evidence of association with TB disease (i.e., excluding variables measured for reasons other than TB risk); (2) measured in the Kharitode trial with little (<10%) missingness; and (3) likely to be available to clinicians in highly resource-limited settings. Continuous predictors were either categorized based on a priori selection of cutoffs that fell near the midpoint of the observed range (e.g., 2-week duration of symptoms) or handled as ordinal variables (e.g., number of current TB symptoms). We applied multiple imputation with chained equations [21,22] to impute missing data, using the R package “mice.” We then conducted 10-fold cross-validation and fit a multivariable least absolute shrinkage and selection operator (lasso) regression model, using the R package “glmnet” [23,24]. Our goal was to create a simple scoring system that could be used rapidly by clinicians in settings of low infrastructure without sacrificing predictive accuracy. To this end, we generated a point scoring system by assigning points to each variable by first identifying clusters of coefficients in the multivariable lasso regression model (i.e., variables that each had roughly equal associations with the outcome of interest), then taking the median of those clustered coefficients, dividing all coefficients by that median value, then rounding to the nearest integer. This process enabled us to remove variables that were only weakly associated with the outcome (i.e., coefficients less than half the median value of the majority of predictors) and to generate a score based entirely on integer values. Use of the mean, rather than median, value did not change our results. Points were summed to generate the final risk score. The final risk score was then fitted to a logistic regression taking TB status (Xpert result) as the outcome. The predictive accuracy of this simplified risk score was compared against that of a full logistic regression model using actual coefficient values (S4 and S5 Figs, S3 Table). We performed a secondary analysis in which all missing values were assigned the most common value (single imputation) to reflect likely clinical use of the score in the field (S4 Table, S8 Fig).

External validation

We validated the model by assessing its performance in a population of individuals tested for active pulmonary TB in outpatient clinics in densely populated parishes of Kampala, the capital city of Uganda. The estimated population density in our study site is 23,000 people per km2. Among the adult study population, self-reported median household income was $91, the prevalence of pulmonary TB was estimated between 420 and 940 per 100,000 population, and the prevalence of HIV was estimated 18%. We enrolled patients with presumptive TB presenting at 4 healthcare facilities in Kampala, Uganda between May 22, 2018 and December 31, 2019. This external validation population consisted of adults (≥15 years) who tested positive for TB on Xpert and a random selection of adults who presented with TB symptoms on the same day as the case but tested negative on Xpert and were not treated for active TB. Cases and non-cases were not matched on any other variable. Missing data were found in only 1 individual who was removed from the analysis. We repeated the same model development process de novo with the bootstrap sample of Ugandan population using a split internal validation approach to assess the degree of difference between the clinical score system as developed in South Africa and the score as developed in Uganda. The de novo analysis was then also validated in the South African population (S5 and S6 Tables, S11 and S12 Figs). All analyses were conducted using R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).

An important aspect of the derivation and validation populations is that a random selection of all non-cases (rather than a full cohort) was enrolled in each study. In South Africa, laboratory registers were used to randomly select 1 Xpert–negative patient for every Xpert–positive patient enrolled; in Uganda, clinical registers of patients with presumptive TB were used to select 2 Xpert–negative patients per Xpert–positive patient. In both studies, controls were matched to cases based only on facility and date range of presentation. Thus, to mitigate against spectrum bias, we performed sensitivity analyses of all outcomes in simulated cohorts where the data from non-cases were replicated to provide prespecified levels of TB prevalence (5%, 10%, and 20%) among people being tested.

Model calibration, discrimination, and clinical utility

We assessed agreement between model-based risk predictions and observed TB status (calibration) using the Cox linear logistic regression in the validation population [21,25]. This regression technique compares observed TB status (positive versus negative) to the log odds of predicted TB risk, assuming a linear relationship. The intercept of this regression can therefore be interpreted as the overall degree of under- or overestimation of risk, while the slope can be interpreted as evaluating whether the accuracy of risk prediction is different, comparing those with low predicted risk to those with high predicted risk. The intercept of the fitted regression model was modified to reflect the different sampling fraction in each population [26,27]. Discrimination was evaluated using receiver operating characteristic (ROC) curve analysis and calculation of the c-statistic (area under the ROC curve). We conducted decision curve analysis to help weigh the relative benefits of treating patients with true active TB against the harms of treating patients with Xpert–negative results using different cutoffs of the clinical score [28,29]. For both predictive accuracy and decision curve analyses, we weighted data from Xpert–negative individuals in Uganda to represent a scenario of 10% underlying TB prevalence. Finally, to provide a clinically relevant decision tool, we used the sensitivity and specificity of the model to estimate the probability of TB for individuals with each possible risk score (1 to 10) assuming pretest probabilities (i.e., prevalence of TB in the underlying population being tested) of 5%, 10%, and 20%.

Analysis plan

This analysis, which relied on the availability of overlapping data from 2 independent studies, was not laid out in a prospective protocol. Data to inform the score were determined a priori based on available variables deemed to be measurable in a highly resource-constrained setting, and the score was developed based on consensus assessment of lasso regression coefficients (i.e., how to combine those coefficients into a simple numerical score), prior to performing the validation. The variables contributing to the score (and the score itself) remain the same as those made in a priori decisions, but numerous supporting analyses were performed or modified in response to comments made in peer review. These revisions included a modified assessment of calibration (use of Cox linear logistic regression with modification of the intercept to reflect sampling fractions), use of multiple imputation for missing data, inclusion of decision curve analysis, and performance of multiple sensitivity analyses (including simulated cohorts to evaluate different underlying TB prevalence, use of restricted cubic splines to model age, and restriction to patients with chronic cough only). Data sufficient to replicate all analyses are available at https://doi.org/10.7281/T1/W2AG3A, and model code is available at https://github.com/ybaik10/clinicalriskscore.

Ethics statement

The Kharitode study was approved by the University of the Witwatersrand’s Human Research Ethics Committee and the Limpopo Provincial Government Department of Health. The external validation study was approved by the Institutional Review Board (IRB) of the Makerere University College of Health Sciences. Both studies were approved by the IRB of the Johns Hopkins Bloomberg School of Public Health through an Authorization Agreement with the local IRBs. All study participants provided written informed consent. Parental consent (with participant assent) was obtained for participants between 15 and 18 years of age.

Results

A total of 1,387 adults with symptoms of active TB were included in the South African derivation cohort: 701 individuals with Xpert-confirmed TB and 686 individuals who presented to the same clinics were tested for TB with Xpert but tested negative and were not started on TB treatment (Fig 1). The Ugandan external validation population included 106 Xpert–positive and 281 Xpert–negative adults. Participants with TB were more likely to report classic TB symptoms (other than cough) and had a longer duration of symptoms (median 4 weeks versus 2 weeks) compared to people with negative Xpert results in both populations (Table 1). Six of the predictors—male sex, age between 25 and 44 years, HIV positivity (based on clinical registers or self-reported answers), the number of WHO TB symptoms (cough, fever, and night sweats within the past few days from the day of clinic visit and weight loss of more than 5 kg or enough to make clothes loose) currently experiencing [16], any TB symptoms duration of over 2 weeks, and self-reported history of diabetes—had sufficiently strong associations to be included in the simple clinical score, which ranged from 1 to 10 (Table 2). Fig 1 illustrates the use of this score in settings of different pretest probability (TB prevalence among people being tested). In settings where the pretest probability is 10%, a patient with a risk score of 3 would have a predicted TB risk of 4%, whereas a person with a risk score of 7 would have a predicted risk of 43% (Fig 2). Optimal cutoffs for clinical use were at score of 4 (sensitivity = 0.85, specificity = 0.63) or 5 (sensitivity = 0.69, specificity = 0.80), out of 10 (Fig 3).

Fig 1

Flow chart of the South African and Ugandan study population.

Left panel shows the flow chart of the South African study population. Laboratory registers were used to randomly select 1 Xpert–negative patient for every Xpert–positive patient enrolled. After excluding those who did not report any TB symptoms, the final ratio between Xpert–positive TB cases (n = 701) and Xpert–negative individuals (n = 686) became 1.02:1. Right panel shows the flow chart of the Ugandan study population. Clinical registers of patients with presumptive TB were used to select 2 Xpert–negative patients per Xpert–positive patient. The final ratio of Xpert–positive TB cases (n = 106) to Xpert–negative individuals (n = 281) was 0.38:1. In both studies, controls were matched to cases based only on facility and date range of clinical presentation. TB, tuberculosis.

Table 1

Characteristics of model derivation (South Africa) and external validation (Uganda) populations.

	South African derivation population,N (%)		Ugandan external validation population,N (%)
	Xpert–positive,701 (51)	Xpert–negative,686 (49)	Xpert–positive,106 (27)	Xpert–negative,281 (73)
Age category, years
15–24	84 (12)	119 (17)	20 (19)	66 (24)
25–34	166 (24)	118 (17)	42 (40)	87 (31)
35–44	202 (29)	133 (19)	30 (28)	69 (25)
45–54	147 (21)	154 (22)	14 (13)	39 (14)
≥55	102 (14)	162 (24)	0 (0)	20 (7)
Sex
Female	275 (39)	389 (57)	38 (36)	161 (57)
Male	426 (61)	297 (43)	68 (64)	120 (43)
HIV status
HIV–negative or unknowna	281 (40)	484 (71)	68 (64)	188 (67)
HIV–positive, on antiretroviral therapy	322 (46)	166 (24)	26 (25)	89 (32)
HIV–positive, not on antiretroviral therapy	48 (7)	14 (2)	12 (11)	4 (1)
HIV–positive, unknown antiretroviral therapy	50 (7)	22 (3)	0 (0)	0 (0)
Presence of classical TB symptoms
Cough	608 (87)	653 (95)	105 (99)	279 (99)
Fever	305 (44)	165 (24)	47 (44)	67 (24)
Weight loss	431 (62)	94 (14)	76 (72)	95 (34)
Night sweats	414 (59)	160 (23)	42 (40)	45 (16)
Total number of classical TB symptoms
1	174 (25)	420 (60)	20 (19)	136 (48)
2	163 (23)	173 (25)	32 (30)	96 (34)
3	198 (28)	66 (9)	30 (28)	38 (14)
4	166 (24)	27 (4)	24 (23)	11 (4)
Duration of TB symptomsb
≤2 weeks	206 (31)	378 (55)	10 (9)	113 (40)
>2 weeks	464 (69)	283 (41)	96 (91)	168 (60)
Any other non-TB symptomsc	440 (63)	301 (44)	62 (59)	111 (40)
Self-reported comorbidities
Diabetes mellitusb	25 (4)	21 (3)	1 (1)	2 (1)
Obstructive pulmonary diseaseb	15 (2)	21 (3)	1 (1)	7 (3)
Previous TB diagnosis (self-report)b	128 (18)	86 (13)	24 (23)	32 (11)
Educationb
High school or less	511 (73)	478 (70)	35 (33)	96 (34)
Any post-high school education	186 (27)	200 (29)	71 (67)	185 (66)
Smoking historyb
Never	416 (59)	478 (70)	67 (63)	225 (80)
Ever	283 (40)	200 (29)	39 (37)	56 (20)
Occupationb
Regularly employed	128 (18)	148 (22)	58 (55%)	148 (53%)
Irregular work, student, or housewife	110 (16)	206 (30)	31 (29%)	89 (32%)
Unemployed or retired	458 (66)	322 (48)	17 (16%)	44 (16%)
Incomeb,d	1,820 (1,140–3,200) ZAR	2,125 (1,140–3,500) ZAR	375,000 (200,000–600,000) Shilling	340,000 (200,000–600,000) Shilling

HIV, human immunodeficiency virus; TB, tuberculosis.

aA total of 11% of patients whose HIV status was unknown or unreported were included in South African study population.

bMissing rate is 4% for duration of TB symptoms; 2% for diabetes and obstructive pulmonary diseases; 1% for education, previous TB, smoking, and occupation; and 43% for income in South Africa.

cIncludes chest pain, pain elsewhere, skin symptoms, genitourinary symptoms, gastrointestinal symptoms, and “any other symptom” by self-report.

d1 ZAR, South African currency = US$0.06; 1 Shilling, Ugandan currency = US$0.0003.

Table 2

Association of key variables with Xpert-confirmed pulmonary TB.

	Unadjusted odds ratioa (95% CI)	Adjusted odds ratiob (95% CI)	p-value	Lasso regression coefficient	Scorec
Age category, years
15–24	1.17 (0.81, 1.7)	1.65 (1.03, 2.65)	0.04	0.39
25–34	2.3 (1.64, 3.23)	2.66 (1.72, 4.11)	<0.001	0.92	1
35–44	2.52 (1.81, 3.5)	1.77 (1.15, 2.73)	0.01	0.51	1
45–54	1.6 (1.15, 2.24)	1.28 (0.84, 1.97)	0.25	0.21
≥55	Reference	Reference		Reference
Sex
Female	Reference	Reference		Reference
Male	2.03 (1.64, 2.51)	2.90 (2.07, 4.05)	<0.001	0.93	1
HIV status
HIV–negative or unknown	Reference	Reference		Reference
HIV–positive	3.63 (2.91, 4.53)	3.55 (2.65, 4.75)	<0.001	1.22	2
HIV/antiretroviral therapy status
HIV–negative or unknown	Reference	-		-
HIV–positive, on antiretroviral therapy	3.38 (2.66, 4.28)	-		-
HIV–positive, not on antiretroviral therapyd	6.02 (3.26, 11.11)	-		-
Classical TB symptoms		-		-
Cough	0.33 (0.22, 0.5)	-		-
Fever	2.44 (1.94, 3.07)	-		-
Weight loss	10.08 (7.73, 13.14)	-		-
Night sweats	4.74 (3.76, 5.98)	-		-
Total number of classical TB symptoms
1	Reference	Reference		Reference	1 (0)
2	2.27 (1.72, 3.00)	1.92 (1.40, 2.64)	<0.001	0.69	2 (1)
3	7.24 (5.21, 10.07)	5.38 (3.70, 7.83)	<0.001	1.70	3 (2)
4	14.84 (9.52, 23.12)	10.00 (6.08, 16.46)	<0.001	2.31	4 (3)
Duration of TB symptoms
≤2 weeks	Reference	Reference		Reference
>2 weeks	3.02 (2.41, 3.79)	2.41 (1.83, 3.16)	<0.001	0.85	1
Any other non-TB symptomse	2.16 (1.74, 2.68)	1.35 (1.03, 1.77)	0.03	0
Self-reported comorbidities
Diabetes mellitus	1.11 (0.62, 1.99)	2.02 (0.92, 4.42)	0.08	0.76	1
Obstructive pulmonary disease	0.69 (0.35, 1.35)	-		-
Previous TB diagnosis (self-report)	1.49 (1.11, 2.00)	1.18 (0.81, 1.71)	0.39	0.13
Educationf
High school or less	Reference	-		-
Any post-high school education	0.86 (0.68, 1.08)	-		-
Smoking history
Never	Reference	Reference		Reference
Ever	1.61 (1.29, 2.01)	0.78 (0.55, 1.11)	0.16	−0.23	0

95% CI, 95% confidence interval; HIV, human immunodeficiency virus; lasso, least absolute shrinkage and selection operator; TB, tuberculosis.

aEstimated from univariate logistic regression.

bEstimated from the multivariable logistic regression, adjusting for all other variables with a population prevalence of at least 10% and a statistically significant association with TB on univariate regression. Individual TB symptoms were removed in favor of total number of symptoms based on an a priori decision.

cTo transform the coefficients to simple relative points, each point in this simple clinical score is estimated by dividing the corresponding lasso coefficient by the median value of coefficients (0.9, taking 1 coefficient closer to clustered values when a variable has more than 2 categories) and rounding to the nearest integer. One point was added to the score for number of TB symptoms to increase usability, as all participants had at least 1 symptom.

dThis category includes HIV–positive with unknown antiretroviral therapy status.

eParticipants were asked about chest pain, pain elsewhere, skin symptoms, genitourinary symptoms, gastrointestinal symptoms, and “any other symptom.”

fOccupation and median household income were also explored as indicators of socioeconomic status but excluded based on uncertain applicability in the clinical setting.

Fig 2

A simple clinical risk score for empiric diagnosis of active TB.

Shown is a 1-page, easy-to-use tool for use in clinical settings where same-day microbiological testing for pulmonary TB is unavailable and risk of loss to follow-up is high. For illustrative purposes, we have chosen cutoffs of 10% and 40% risk of TB as potential clinical decision points, based on natural breaks in predictive probability and on intuition that TB treatment is unlikely to be started empirically for patients with less than a 1 in 10 chance of having TB. Other cutoffs could be selected based on local resources and probabilities of loss to follow-up if untreated. TB, tuberculosis.

Fig 3

Accuracy of a simple clinical risk score for active pulmonary TB.

This figure shows the sensitivity (solid black), specificity (dashed black), positive predictive values (red), and negative predictive values (blue) of a simple clinical risk score for pulmonary TB in the derivation South African cohort, at different cutoffs for a positive test. The positive and negative predictive values are estimated assuming a prevalence of TB among symptomatic individuals presenting for care (i.e., pretest probability) of 5% (dotted lines), 10% (dashed lines), and 20% (solid lines). The score ranges from a minimum of 1 to 10; see Fig 2. Due to the small sample size of individuals with scores higher than 8, we combined scores equal to or higher than 8 into 1 category. Accuracy and predictive values are calculated relative to Xpert MTB/RIF as a gold standard. TB, tuberculosis.

In the model derivation and external validation population, the median risk score among people with TB was 5 (interquartile range [IQR] 4 to 6), compared to 3 (IQR 2 to 4) and 4 (IQR = 3 to 5), respectively, in people without TB. Model calibration was acceptable in the context of a very simple score (intercept = −0.25 [95% CI: −0.50, 0.01] and slope = 0.76 [95% CI: 0.65, 0.96]). (Fig 4A) The simplified risk score demonstrated moderate discrimination in the derivation and external validation population, with a c-statistic of 0.82 (95% CI 0.81 to 0.82) and 0.75 (95% CI 0.69 to 0.80), respectively. This was comparable to the discrimination of the full model using regression coefficients (c-statistic 0.83, 95% CI 0.82 to 0.83 in the derivation population and 0.77, 95% CI 0.71 to 0.82 in the validation population) (Figs 4B and S4). Estimated TB prevalence and mode of imputing missing data did not influence model discrimination (S8 Fig).

Fig 4

Calibration and discrimination of a simple clinical score for diagnosis of active TB in sub-Saharan Africa.

Panel A shows model calibration using Cox linear logistic regression in the external validation (Ugandan) population. An intercept of 0 and slope of 1 is consistent with good calibration. In this plot, the red line represents perfect calibration, the black line corresponds to calibration of the simple clinical score, the dotted blue line corresponds to a smoothed (Loess) calibration, and the gray region corresponds to the 95% confidence band of the Loess calibration. The black line falling below the red line indicates that the simple score mildly overestimates the probability of TB in the validation population. Calibration curves were generated after adjusting for the different sampling fraction of TB in the derivation and validation populations, as described in the text. Panel B shows the ROC curve—a measure of discrimination—in the South African derivation (black line) and Ugandan external validation (red line) cohorts. Due to the small sample size of individuals with scores over 8, we combined scores equal to or higher than 8 into 1 category. The number on each dot represents the risk score at which sensitivity and specificity are estimated. For example, at a score of 4, specificity and sensitivity are 0.63 and 0.85, respectively, in the derivation population and 0.47 and 0.88, respectively, in the validation population. The reported c-statistics did not differ with adjustment of the sampling fractions to a population with 10% estimated prevalence. ROC, receiver operating characteristic; TB, tuberculosis.

Decision curve analysis suggested that use of the clinical score to inform empiric treatment decisions offered a positive net benefit (compared to a treat-all or treat-none strategy) when the ratio of benefit of a true-positive diagnosis to cost of a false-positive diagnosis ranged from 4:1 to 50:1 (Fig 5).

Fig 5

Clinical utility of a simple clinical score for diagnosis of active TB in sub-Saharan Africa.

A decision curve analysis compares the standardized net benefit of different treatment strategies. The standardized net benefit was estimated as total benefit (treating true TB) minus total harm (treating false-positive TB), standardized to a maximum benefit of 1, assuming a population with 10% underlying prevalence of TB. The standardized net benefit was examined under different considerations of the relative benefit of a true-positive diagnosis versus the risk of a false-positive diagnosis (x-axis). This x-axis or threshold probability corresponds to the posttest probability given a TB prevalence of 10% among people being tested (i.e., as shown in Fig 2) at which the harms and benefits of empiric TB treatment are considered to be equivalent (i.e., the number of people without TB started on empiric treatment (false-positives) that would be tolerated to start empiric treatment on 1 additional person with TB (true-positive). The black numbers on top of the x-axis are the scores that correspond to each of the posttest probabilities. A treatment strategy with the highest net benefit at the particular threshold probability has the highest clinical value. The decision curve is based on the external validation population, with 95% confidence bands shown as dotted lines. Areas on the x-axis where the lower bound of the 95% CI is higher than the “treatment for all” line (i.e., threshold probability > 6%) offers a statistically significant benefit over treating all individuals. Areas on the x-axis where the lower bound of the 95% CI does not cross 0 (i.e., threshold probability ≤ 23%) illustrate settings in which use of the clinical risk score offers a statistically significant benefit relative to no empirical treatment. Numbers corresponding vertical dotted lines denote the threshold probabilities where the lower 95% confidence limit of “treatment based on clinical risk score” line is higher than the expected net benefit of treatment for all (blue line) or no empiric treatment (black line). These threshold probabilities (6%, 23%) include a threshold for treatment of a risk score of ≥4 (9% probability of TB) or ≥5 (17% probability of TB). Use of the clinical risk score would therefore offer higher net benefit than alternative treatment strategies (e.g., treatment for all or treatment for none) if the benefit of empirically treating someone with TB is deemed to be 3.3 to 15.7 times greater than the harm of empirically treating someone without TB. CI, confidence interval; TB, tuberculosis.

The de novo model developed in the Ugandan population included a total of 9 points, of which 8 appeared in the primary model developed in the South African population (age, sex, HIV status, the number of WHO TB symptoms [1-4], and duration of symptoms) and 2 did not (antiretroviral therapy (ART) and cough observed during the interview). These 2 variables either were not collected (observed cough) or were rare (HIV not on ART) in the rural South African cohort. The c-statistic of this model in the internal validation population in Uganda was 0.81 (95% CI = 0.79 to 0.83) and was 0.78 (95% CI = 0.77 to 0.79) in the external validation population in South Africa (S5 and S6 Tables, S11 and S12 Figs).

After replicating non-cases in simulated cohorts, the sensitivity analyses showed no material change in discrimination; calibration was acceptable when the external validation population was modified in this fashion to match the prevalence of TB in the derivation population (see below), and estimates of clinical utility are provided across a range of underlying TB prevalence estimates to enhance transportability across settings. All estimates of variance, however, reflect only the primary data available (without replication of the Xpert–negative population).

Discussion

This analysis of 2 different clinical populations in sub-Saharan Africa illustrates the successful development and validation of a clinical risk score for predicting the presence of active TB in primary health clinics where same-day microbiological testing may be unavailable. This score is easy to calculate by hand and relies only on data that are readily available in the clinical setting. Despite its simplicity, the clinical risk score showed reasonable predictive accuracy, including in an external validation population. Use of this score was estimated to add clinical utility for patients in whom the benefits of an immediate diagnosis were similar or somewhat greater than the risks of false-positive diagnosis. De novo construction of a similar tool in the validation population yielded remarkably similar results, further suggesting that this clinical tool might be transportable across sub-Saharan African primary care settings. We have constructed a corresponding 1-page clinical tool that can be easily implemented in such settings. This clinical tool could reduce losses to follow-up (and corresponding transmission and mortality) by facilitating immediate initiation of treatment among patients who have a high clinical probability of having TB. Before recommending wide use, however, this tool must first be validated in other populations, and a strategy of using such a score to inform empiric treatment decisions must be evaluated in terms of its implementation and impact on patient outcomes (e.g., pretreatment loss to follow-up, treatment success, and mortality) in field settings.

A major potential benefit of this clinical TB score is its reliance on characteristics—namely younger age, male sex, number of TB symptoms (and duration >2 weeks), and HIV status—that are (1) easy to measure; (2) known to be associated with TB; and (3) confirmed to enhance predictive accuracy in 2 very different clinical populations. Our risk score formally combines these characteristics in a way that would be easy to calculate in the clinical setting and also easy to incorporate into policy. Regarding age, young adults may not experience higher absolute TB risk compared to older adults. Our data rather may reflect that alternative diagnoses (e.g., chronic obstructive pulmonary diseases and cancer) are less likely in younger patients who are seeking care for severe illness. Self-reported diabetes status contributed to the clinical score but may be slightly more difficult to standardize. Furthermore, the prevalence of diabetes and obesity is higher in South Africa than in other sub-Saharan African countries [30] and may not contribute substantially to predictive accuracy in those settings; as evidence of this, self-reported diabetes did not appear in the de novo model developed from the Ugandan population.

It is helpful that a simple clinical score performed nearly as well as a model using full regression coefficients, that imputation of the most common value (as might be done in the field) performed equally well as multiple imputation, and that predictive accuracy was not substantially degraded in external validation. Also reassuring is that the scores developed de novo in 2 separate populations converged on the majority of characteristics included. While these results imply the potential transportability of this clinical TB score to other populations (especially other populations seeking primary care for TB symptoms in sub-Saharan Africa), further studies are required to confirm such generalizability before incorporating this score into widespread practice.

The value of this clinical score for active TB must be interpreted in light of its intended use. This score is only meant to guide management while awaiting microbiological (e.g., Xpert) results, not intended as a community-based triage test (i.e., to identify individuals who should receive further testing). Thus, while the score does not meet minimum WHO target product profile criteria for a community-based TB triage test (90% sensitivity and 70% specificity) [31], its value should be assessed not on its use as a triage test, but rather on the basis of its potential clinical utility in patients for whom a microbiological test is indicated but may not reliably or rapidly return. For example, in settings (e.g., high-income countries and most cities in middle-income countries) where microbiological results are likely to be consistently and rapidly available, a clinical prediction rule may not add substantively to the existing standard of care. If implemented in clinical settings, it is possible that this score could be misused (i.e., to forego microbiological testing in high-risk patients); any implementation plan would need to be accompanied by strong instructions and perhaps incentives to prevent such misuse.

However, in highly resource-constrained settings where Xpert results may not consistently return, this clinical score may help guide management. Many such settings employ midlevel clinicians and utilize highly standardized approaches to care. If a patient presents with TB symptoms, for example, a sputum specimen will typically be collected, but treatment for TB will generally be deferred until the corresponding results return. In this context, pretreatment loss to follow-up is common [2]. If initiation of empirical TB treatment could reduce long waiting times, repeated visits, and delays in receiving results [2], it might facilitate immediate initiation of treatment (with the aim of reducing losses to follow-up) for high-risk patients. Indeed, many known risk factors exist for pretreatment losses to follow-up, including long turnaround time and travel times, high patient costs, and perceived stigma [2,32,33]. Clinicians might therefore selectively apply this score for patients with high risk of loss to follow-up. Alternatively, a similar algorithm to predict the risk of pretreatment loss to follow-up might be useful. Key priorities for future studies are therefore to better prospectively identify patients at high risk of pretreatment loss to follow-up and to better quantify the risk–benefit ratio of initiating empiric TB treatment, as functions of TB risk (as described here), clinical severity, and risk of loss to follow-up [34].

In selecting an appropriate cutoff for intervention on the basis of this clinical score, certain tradeoffs must be considered. If using a low cutoff of empiric treatment (for example, > score of 4 in a setting of ≥10% TB prevalence, yellow cells in Fig 2), the vast majority of individuals above this cutoff will not have TB. In such a setting, it would be critical to ensure that TB therapy could be de-escalated if microbiological results came back negative and clinical improvement were not consistent with TB (e.g., rapid improvement in a matter of days or no improvement over weeks). If such practice could be followed, same-day initiation of empiric TB treatment might improve clinical outcomes and reduce losses to follow-up, as seen with rapid initiation of ART for HIV [35,36]. Decision curve analysis (Fig 5) suggests that this is the context in which the score is likely to add greatest value, but future prospective studies would be necessary to confirm whether better outcomes could actually be achieved using this approach. If de-escalation of care is thought not to be feasible, an alternative use of this score would be to initiate empiric TB treatment only for those patients with a very high probability of TB (for example, 40% probability as defined by the red cells in Fig 2). In this context, future de-escalation of treatment might be less essential, but as suggested by the decision curve, this approach might not add substantial value over the current standard of care in which many such patients will already be treated.

The predictive accuracy of this simple clinical score was compatible with, or even superior to, previously published prediction models that incorporated criteria including chest X-ray [12] (c-statistic = 0.70, 0.79), body mass index (BMI) [13,15] (c-statistic = 0.70), hemoglobin level [14,15] (c-statistic = 0.66), HIV viral load and ART [37] (c-statistic = 0.59, 0.69), CD4+ T cell count [12,13,37] (c-statistic = 0.70), and Karnofsky score [14] (c-statistic = 0.75). These measures may not be readily available in highly resource-constrained settings that may lack even reliable scales and stadiometers. Of note, some prediction rules [12,37] have been explicitly designed for use in HIV–positive populations, in whom TB is often more difficult to diagnose. As such, the expected performance of these prediction rules might be lower, and they might be limited in their ability to generalize to HIV–negative populations. Some other prediction rules [38,39] have made in the context of contacts where not many of those experience TB symptoms. Our aim was to expand on this work by developing a tool that could be readily employed in peripheral health settings where a large fraction of individuals with TB symptoms may initially present to care [10], but where resources are extremely limited.

Our study has limitations. First, the South African derivation cohort included only 60% of all individuals who presented with TB symptoms during the study period. The majority of non-participants (70%) were not reachable by phone follow-up for collection of additional data. While the external validation population had much lower refusal rates (3%), these losses might induce bias in our results. Second, both the development and validation cohorts included a random sample of people without TB, rather than evaluating a full cohort—and the sampling fractions differed between the derivation and validation cohorts. Non-cases were sampled to be representative of all people presenting with symptoms (i.e., not matched to cases), and we confirmed all results in simulated cohorts designed to match the underlying prevalence of TB; these analyses help to mitigate against spectrum bias but cannot remove it entirely. Third, we recruited people with presumptive pulmonary TB who presented to primary healthcare facilities. This clinical TB score may therefore not be applicable to such as active case finding where many people do not have TB symptoms, hospitals where individuals are acutely ill, or settings for predicting extrapulmonary TB. The utility of this score may also depend strongly on the underlying population’s access to care—which can affect both symptoms on presentation and probability of returning for confirmatory microbiological test results [2]. Fourth, we omitted some factors that are known to be associated with increased TB risk (e.g., self-reported weight, clinician’s subjective assessment of underweight status, or recent contact with a known TB case). These variables were either not measured in our population (self-reported weight and height and subjective assessment of underweight) or not found to improve predictive accuracy (recent TB contact). We did measure BMI in Uganda and found underweight status (BMI < 18.5kg/m2) as an additional score of 2, to improve discrimination; future studies should consider including these variables where height and weight can be reliably measured and BMI readily calculated. Fifth, the sensitivity of Xpert is imperfect, and many individuals in both cohorts were treated for TB without a positive Xpert result. By excluding these individuals from our analyses, our results may be biased in favor of better predictive accuracy (as these individuals may have higher clinical risk scores but may not have true underlying TB), may be subject to spectrum bias (as the clinical presentation of Xpert–positive and Xpert–negative TB cases may differ), and may exclude the very population for whom the clinical score might be most useful. This bias is likely to be small in the South African population because the prevalence of unknown Xpert status or empiric treatment was <1%. Sixth, this risk score may not be transportable to settings with aging populations, high prevalence of TB among non-native–born populations, or lower HIV prevalence. Finally, although the prediction model is simple, calculation of posttest probabilities requires knowledge of pretest probabilities (i.e., the proportion of patients tested with Xpert at a given facility who test positive); these numbers may not be widely known by treating clinicians.

In summary, we have developed a simple clinical TB score to inform empirical treatment decisions in resource-limited peripheral health settings of sub-Saharan Africa. This score is easy to calculate, consists only of easily measured variables, and performed well in 2 distinct populations (rural South Africa and urban Uganda). Use of this score may help to guide clinical management while awaiting results of microbiological testing, especially in highly resource-constrained settings where Xpert results may not be reliably available, and pretreatment losses to follow-up are high.

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline.

(DOCX)

Click here for additional data file.

Photos of the study sites.

Top is the study site in Limpopo Province, South Africa. Limpopo Province is widely rural and has a low population density (46 people per km2). It has high TB incidence (301 per 100,000) and high adult HIV prevalence (17%). Two photos in the bottom are the study site in the capital city, Kampala, Uganda. In our Ugandan study site, the population density was estimated 23,000 people per km2, TB prevalence was estimated 939 per 100,000 by GeneXpert Ultra catridge (418 per 100,000 after excluding GeneXpert Ultra trace positive but culture negative), and HIV prevalence was 18% among adults.

(TIFF)

Click here for additional data file.

A simple clinical risk score for empiric diagnosis of active tuberculosis in other pretest TB probability settings.

For illustrative purposes, we have chosen cutoffs of 10% and 40% risk of TB as potential clinical decision points, based on natural breaks in predictive probability and on intuition that TB treatment is unlikely to be started empirically for patients with less than a 1 in 10 chance of having TB. Other cutoffs could be selected based on local resources and probabilities of loss to follow-up if untreated. The utility of this score would markedly decrease in settings where the pretest TB probability is below 2.5% because the score higher than 7 is ≥15% at 2.5% TB setting.

(TIFF)

Click here for additional data file.

Calibration of a clinical risk score using actual regression coefficients rather than a simple 1 to 10 score.

The figure shows model calibration using Cox linear logistic regression in the external validation (Ugandan) population. An intercept of 0 and slope of 1 is consistent with good calibration. In this plot, the red line represents perfect calibration, the black line corresponds to calibration of the simple clinical score, the dotted blue line corresponds to a smoothed (Loess) calibration, and the gray region corresponds to the 95% confidence band. Calibration curves were generated after adjusting for the different sampling fraction of TB in the derivation and validation populations, as described in the main text.

(TIFF)

Click here for additional data file.

Discrimination of a more detailed clinical risk score using actual regression coefficients rather than a simple 1 to 10 score, including age modeled as a restricted cubic spline (RCS).

Panel A shows the receiver operating characteristic (ROC) curve—a measure of discrimination—in the South African derivation (black line) and Ugandan external validation (red line) cohorts. The reported c-statistics did not differ with adjustment of the sampling fractions to a population with 10% estimated prevalence. Panel B investigates the nonlinear relationship between the probability of Xpert MTB/RIF result and the continuous variable of age. The R package “Hmisc” was used to plot the estimated restricted cubic splines (RCS; with 5 knots placed at the natural quantiles of age). Arrows on the x-axis show the location of the knots. Of note, the second knot fell very near the median of our highest-risk category (i.e., age 25 to 44 years). Dots show a plot of nonparametric estimates (or smoothing parameters). The solid line shows the estimated spline transformation, and the dotted line shows the 95% confidence lines. Panel C compares the ROC curves using the actual regression coefficients; one with the age modeled as an RCS (solid line) and the other (dotted line) with the age modeled as categorical by decade (i.e., 15 to 24, 25 to 34, 35 to 44, 45 to 54, and 55+). The discrimination of the full model with age as a categorical variable was significantly higher than with age modeled as an RCS but was not statistically different from that of the simple 1 to 10 risk score presented in the main text.

(TIFF)

Click here for additional data file.

Discrimination of the model using each of actual classical TB symptoms as individual binary variables, instead of the total number of TB symptoms.

This receiver operating characteristic (ROC) curve shows the discrimination of the model using each of actual classical TB symptoms as individual binary variables, instead of the total number of TB symptoms. The black curves show the clinical score’s discrimination in the derivation (South African) population, and the red curves show the corresponding analysis in the external validation (Ugandan) population. The number on each dot represents the risk score at which sensitivity and specificity are estimated. The discrimination of the model using each symptom individually was not significantly higher than that of the main simple risk score in which symptoms were simply counted, shown in Fig 4B.

(TIFF)

Click here for additional data file.

Discrimination of the model within individuals who presented chronic cough regardless of any other TB symptoms.

In South African population, 665 (49%) reported chronic (>2weeks) cough, 61% of which were Xpert–positive TB cases in South African population. In Ugandan population, 262 (68%) reported chronic cough in Ugandan population, 36% of which were Xpert–positive TB cases. This receiver operating characteristic (ROC) curve shows the discrimination of the model within individuals who presented chronic cough regardless of any other TB symptoms. The black curves show the clinical score’s discrimination in the derivation (South African) population, and the red curves show the corresponding analysis in the external validation (Ugandan) population.

(TIFF)

Click here for additional data file.

Calibration prior to adjustment for different sampling fractions.

The clinical risk score for active tuberculosis (TB) as derived in the South African population is expected to systematically overestimate the predicted risk of TB in the external validation Ugandan population because of the different sampling fractions used. (Specifically, in South Africa, approximately 1 Xpert-individual was sampled per case, whereas in Uganda, nearly 2.7 Xpert–negative individuals were sampled per case.) To appropriately evaluate model calibration, therefore, we adjusted the intercept of the Cox linear–logistic model by multiplying the log odds ratio of TB in the validation population versus the derivation population. This intercept is a measure of the difference between the predicted probability and the actual outcome (i.e., calibration-in-the-large). Adjusting the intercept (for everyone in the study population) does not affect the order of predicted risks of individuals, and hence, it does not change the predictive accuracy in terms of discrimination (i.e., c-statistics). Panel A shows model calibration using Cox linear logistic regression in the external validation (Ugandan) population before adjusting the intercept, using simple scores as the prediction model. An intercept of 0 and slope of 1 is consistent with good calibration. In this plot, the red line represents perfect calibration, the black line corresponds to calibration of the simple clinical score, the dotted blue line corresponds to a smoothed (Loess) calibration, and the gray region corresponds to the 95% confidence band. The divergence between the black and red lines illustrates the poor calibration prior to adjustment of the intercept; as described above, this poor calibration is expected on the basis of different sampling fractions. Panel A shows calibration of the simple 1 to 10 score, whereas Panel B shows calibration of the model using full regression coefficients. Both of these plots were generated in the external validation (Ugandan) population before adjusting the intercept.

(TIFF)

Click here for additional data file.

Discrimination of the clinical risk score with missing values imputed as the most common value rather than by multiple imputation.

This receiver operating characteristic (ROC) curve shows the discrimination of the model after imputing missing data with the most common value (as might be done in clinical practice), rather than using multiple imputation. The black curves show the clinical score’s discrimination in the derivation (South African) population, and the red curves show the corresponding analysis in the external validation (Ugandan) population. The solid lines and dotted lines show the simple 1-to-10 scoring and scoring system using full regression coefficients, respectively. Comparing this figure to Fig 4B in the main text illustrates that the simple risk score retains the same discrimination (c-statistic 0.75) in the external validation population when using a simple imputation technique (replacing missing values with the most common value in the population), which could be performed easily in clinical practice.

(TIFF)

Click here for additional data file.

Discrimination of the clinical risk score after including empirically diagnosed TB patients as TB positive or as TB negative.

In Ugandan study population, 27 TB cases were clinically diagnosed; 16 of which were empirically treated without microbiological confirmation and 11 of which were empirically treated regardless of negative Xpert results. Among 27 patients treated empirically, 7 (26%) patients ultimately had Xpert–positive (4/7) or culture positive (3/7). The risk score of empirically diagnosed individuals (median = 6, IQR = 5–6) was as high as the score of Xpert-confirmed TB cases (5, 4–7) compared to the Xpert–negative group (4, 3–5). The receiver operating characteristic (ROC) curves show the discrimination of the model after including empirically diagnosed TB patients in the study population. Left panel is the ROC curve when empirically diagnosed TB patients were considered as TB positive together with Xpert-confirmed TB-positive cases. Right panel is the ROC curve when empirically diagnosed TB patients were considered as TB negative. The curve shows the clinical score’s discrimination in the external validation (Ugandan) population. The points on the line show the simple 1-to-10 scoring. Comparing these figures to Fig 4B in the main text illustrates that the simple risk score retains the similar discrimination power (c-statistic 0.76 versus 0.75) when treating empirically diagnosed TB patients same as Xpert-confirmed TB-positive cases. As expected, including these empirically diagnosed individuals (slightly) increased the predictive power of our risk score.

(TIFF)

Click here for additional data file.

Clinically utility of a simple clinical score for diagnosis of active tuberculosis in sub-Saharan Africa assuming a population with 5% underlying prevalence of TB.

The standardized net benefit (y-axis) was estimated as total benefit (treating true TB) minus total risk (treating false-positive TB), standardized to a maximum benefit of 1, assuming a population with 5% underlying prevalence of TB. The standardized net benefit was examined under different considerations of the relative benefit of a true-positive diagnosis versus the risk of a false-positive diagnosis, or threshold probabilities (x-axis). The decision curve is based on the external validation population, with 95% confidence bands shown as dotted lines. Black numbers on top of the x-axis are the posttest probability we estimated in the main Fig 1 under 5% pretest probability. Red numbers correspond to the threshold probabilities where the lower 95% confidence limit of “treatment based on clinical risk score” line is higher than the upper 95% confidence limit of other lines of 2 different treatment strategies. Use of the clinical risk score of 4 or 5 would offer higher net benefit than alternative treatment strategies (e.g., treatment for all or no empiric treatment) between 3% and 11%. This figure can also be used to evaluate the potential impact of incorrectly assumed pretest probability of TB—for example, if practitioners believe that the pretest probability is 10%, when in reality it is 5%. In this example, the only score in which a misspecification would lead to an incorrect treatment decision is a score of 5. This implies a posttest probability of 17% and an empiric treatment consideration at a 10% pretest probability setting, when in fact the posttest probability is 9% and no empiric treatment consideration required at a 5% pretest probability setting. About 20% of symptomatic individuals may be falsely treated until the microbiological testing confirmation is available.

(TIFF)

Click here for additional data file.

Calibration of a simple clinical score for empirical diagnosis of active tuberculosis developed from the Ugandan study population.

Panel A shows model calibration using Cox linear logistic regression in the internal validation (Ugandan) population, among those whose data did not contribute to model development. An intercept of 0 and slope of 1 is consistent with good calibration. In this plot, the red line represents perfect calibration, the black line corresponds to calibration of the simple clinical score, the dotted blue line corresponds to a smoothed (Loess) calibration, and the gray region corresponds to the 95% confidence band. Panel B shows model calibration in the external validation (South African) population. The calibration curve for Panel B was generated after adjusting for the different sampling fraction of TB in the derivation and validation populations, as described in the text.

(TIFF)

Click here for additional data file.

Discrimination of a simple clinical score and more detailed clinical risk score (using actual regression coefficients) for empirical diagnosis of active tuberculosis developed from the Ugandan study population.

Panel A shows the receiver operating characteristic (ROC) curve in the South African derivation cohort (red line), internal validation cohort (blue line), and Ugandan external validation cohort (black line). The number on each dot represents the risk score at which sensitivity and specificity are estimated. For example, at a score of 5, sensitivity and specificity are 0.66 and 0.82, respectively, in the derivation and internal validation populations, versus 0.79 and 0.64, respectively, in the external validation population. The reported c-statistics did not differ with adjustment of the sampling fractions to a population with 10% estimated prevalence. Panel B shows the ROC curve using more detailed score incorporating full regression coefficients. The discrimination power was retested under an assumption of 10% TB prevalence (by replicating the TB-negative population to fit the target prevalence); the c-statistics remained the same.

(TIFF)

Click here for additional data file.

Interview questions for self-reported tuberculosis symptoms and HIV status.

(DOCX)

Click here for additional data file.

Observed and predicted probability of active pulmonary TB in the Ugandan external validation population.

The table provides the observed percentage of the population in the external validation population (urban Uganda) who had Xpert-confirmed TB, given each clinical risk score. The observed percentages are compared to the percentages predicted by the clinical risk score (right-most column). The optimal cutoff for clinical decision-making was at a score of ≥4 or ≥5; above these cutoffs, observed and predicted probabilities were similar.

(DOCX)

Click here for additional data file.

Lasso regression coefficients and the simple scoring system after modeling actual classical TB symptoms as individual binary variables (instead of the total number of TB symptoms).

We explored use of the actual regression coefficients from the lasso model (rather than a simple 1-to-10 scoring system), including both a categorical expression of age and a representation of age using restricted cubic splines. For each of these alternative scoring systems, derivation and external validation were performed in the same fashion as for the primary score described in the main text.

(DOCX)

Click here for additional data file.

Lasso regression coefficient and simple scoring systems, using a single imputation method (i.e., mode) for the missing predictors.

For the clinical risk score to be useful as a tool in the field, clinicians must be able to directly impute missing data when those data are unavailable. We therefore performed a secondary analysis in which all missing values were assigned the most common value (or single imputation) to reflect likely clinical use of the score in the field. For example, when a clinician does not have information about duration of TB symptoms, the clinician can use the dominant value, 0 or 1, among patients. In our study, a missing duration of TB symptoms, any other non-TB symptoms, and diabetes mellitus was replaced to 1, 1, and 0. After replacing the missing values with the most common value of each variable, the simple score rule based on the least absolute shrinkage and selection operator (lasso) regression coefficients remained the same. We then compared the discrimination of this model using simple (clinical) imputation versus multiple imputation.

(DOCX)

Click here for additional data file.

Additional characteristics of the model derivation population (Uganda) after bootstrapping and random splitting of the model derivation and the internal validation population.

To further assess the transportability of the components of the clinical risk score, we performed a de novo model development using the population from Kampala, Uganda, as the derivation population and the population from South Africa as the external validation population. For this analysis, we used a 20-fold bootstrap sample of the Ugandan population (given its smaller sample size) to optimize statistical power. Then, we applied a split internal validation approach by randomly selecting two-thirds of the population as the model training cohort and internally validating the model on the remaining one-third of the testing population (whose data did not contribute to model development).

(DOCX)

Click here for additional data file.

Association of key variables with Xpert-confirmed pulmonary tuberculosis in the bootstrapped Ugandan population.

(DOCX)

Click here for additional data file.

19 May 2020

Dear Dr Baik,

Thank you for submitting your manuscript entitled "A simple clinical score for predicting active tuberculosis when same-day microbiological testing is unavailable" for consideration by PLOS Medicine.

Your manuscript has now been evaluated by the PLOS Medicine editorial staff and I am writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by .

Login to Editorial Manager here: https://www.editorialmanager.com/pmedicine

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosmedicine@plos.org if you have any queries relating to your submission.

Kind regards,

Thomas J McBride, PhD,

PLOS Medicine

22 Jun 2020

Dear Dr. Baik,

Thank you very much for submitting your manuscript "A simple clinical score for predicting active tuberculosis when same-day microbiological testing is unavailable" (PMEDICINE-D-20-01816R1) for consideration at PLOS Medicine.

Your paper was evaluated by a senior editor and discussed among all the editors here. It was also discussed with an academic editor with relevant expertise, and sent to independent reviewers, including a statistical reviewer. The reviews are appended at the bottom of this email and any accompanying reviewer attachments can be seen via the link below:

[LINK]

In light of these reviews, I am afraid that we will not be able to accept the manuscript for publication in the journal in its current form, but we would like to consider a revised version that addresses the reviewers' and editors' comments. Obviously we cannot make any decision about publication until we have seen the revised manuscript and your response, and we plan to seek re-review by one or more of the reviewers.

In revising the manuscript for further consideration, your revisions should address the specific points made by each reviewer and the editors. Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments, the changes you have made in the manuscript, and include either an excerpt of the revised text or the location (eg: page and line number) where each change can be found. Please submit a clean version of the paper as the main article file; a version with changes marked should be uploaded as a marked up manuscript.

In addition, we request that you upload any figures associated with your paper as individual TIF or EPS files with 300dpi resolution at resubmission; please read our figure guidelines for more information on our requirements: http://journals.plos.org/plosmedicine/s/figures. While revising your submission, please upload your figure files to the PACE digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at PLOSMedicine@plos.org.

We expect to receive your revised manuscript by Jul 13 2020 11:59PM. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

We ask every co-author listed on the manuscript to fill in a contributing author statement, making sure to declare all competing interests. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. If new competing interests are declared later in the revision process, this may also hold up the submission. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT. You can see our competing interests policy here: http://journals.plos.org/plosmedicine/s/competing-interests.

Please use the following link to submit the revised manuscript:

https://www.editorialmanager.com/pmedicine/

Your article can be found in the "Submissions Needing Revision" folder.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see http://journals.plos.org/plosmedicine/s/submission-guidelines#loc-methods.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

We look forward to receiving your revised manuscript.

Sincerely,

Emma Veitch, PhD

PLOS Medicine

On behalf of Clare Stone, PhD, Acting Chief Editor,

PLOS Medicine

plosmedicine.org

-----------------------------------------------------------

Requests from the editors:

*We'd suggest revising the title per PLOS Medicine's style, ideally this should include the study design in the subtitle (after a colon), eg ": A randomized controlled trial," "A retrospective study," "A modelling study," etc.

*Please structure the abstract using the PLOS Medicine headings (Background, Methods and Findings, Conclusions; "Methods and Findings" should be a single subsection. In the last sentence of the Abstract Methods and Findings section, please include a brief summary of any key limitation(s) of the study's methodology.

*At this stage, we ask that you include a short, non-technical Author Summary of your research to make findings accessible to a wide audience that includes both scientists and non-scientists. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract. Please see our author guidelines for more information: https://journals.plos.org/plosmedicine/s/revising-your-manuscript#loc-author-summary

*Please clarify in the paper (Methods section) whether this analysis, reported in the paper, corresponded to that laid out in a prospective protocol or analysis plan? Please state this (either way) early in the Methods section.

a) If a prospective analysis plan (from your funding proposal, IRB or other ethics committee submission, study protocol, or other planning document written before analyzing the data) was used in designing the study, please include the relevant prospectively written document with your revised manuscript as a Supporting Information file to be published alongside your study, and cite it in the Methods section. A legend for this file should be included at the end of your manuscript.

b) If no such document exists, please make sure that the Methods section transparently describes when analyses were planned, and when/why any data-driven changes to analyses took place.

c) In either case, changes in the analysis-- including those made in response to peer review comments-- should be identified as such in the Methods section of the paper, with rationale.

*The authors may wish to consider using a reporting guideline corresponding to their specific study design to aid fuller reporting of the methods and findings of this paper. The TRIPOD statement may be the most appropriate (https://www.tripod-statement.org/TRIPOD/TRIPOD-Checklists) and the authors may like to look at whether this guideline is helpful; if so the checklist provided on the TRIPOD website could be used to assist reporting - in which case please fill out the checklist and upload as supporting information when resubmitting. In addition to this the authors might like to cite either the TRIPOD statement or other reporting guideline they used in enhancing study reporting.

-----------------------------------------------------------

Comments from the reviewers:

Reviewer #1: Dear Authors,

Thank you for the opportunity to review this great and important paper, which I have recommended for publication. I attach a letter with some minor comments for consideration prior to publication but have let the editor know that I do not need to see another version of this manuscript.

Congratulations on your excellent, practical, research.

Yours sincerely,

Tom Wingfield

-----------------------------------------------------------

Reviewer #2: Comments to the authors

I enjoyed reading this manuscript, which makes an important and novel contribution to the literature. The study aims to address the critical question of how to manage patients presenting to primary care settings with TB symptoms for whom same day microbiological testing is unavailable. This is a thorough and impressive piece of work, with sound statistical methods that are strengthened significantly by the extensive additional analyses presented in the supporting information. Some of the issues I would have highlighted have already been addressed by the authors in these supplementary analyses or have been thoughtfully discussed in the limitations. However, I have a number of comments and suggestions that may help to further strengthen the manuscript that mostly relate to the clinical utility of the score.

Cut offs and implications for practice. The authors present standard parameters used to evaluate diagnostic tests. However, I think that what the use of this score in practice might explicitly mean should be more clearly demonstrated, and the implications of specific strategies further considered. There is a slight contradiction in the manuscript because in the risk score figure, the authors are careful to frame clinical recommendations as "consider or strongly consider treatment". However, in the discussion, the authors highlight that in many resource-constrained settings mid-level clinicians utilize standardised approaches to care. If a standardised approach were to be used, using a score cut off of four for giving empirical TB treatment gives a sensitivity of 85% and a specificity of 63% (optimistic scenario using the derivation data). Translating these figures into a population of 1000 people where prevalence of TB in tested individuals is 10% means that 85 would be appropriately treated for TB, 15 would not be treated immediately despite having TB, and a further 333 would also be given TB treatment who subsequently have a negative Xpert result. Using a different cut-off point would mean different results and this trade-off between sensitivity and specificity is very important and could be further elaborated upon. If the score were to be applied in this standardised way with a cut off of four, this would mean a very large number of people starting TB treatment empirically who subsequently have a negative Xpert result (four times the number starting TB treatment who subsequently have a positive Xpert result). Some of these "negative but treated" people will truly actually have TB, as a single Xpert test on sputum has imperfect sensitivity for pulmonary TB and is inadequate to diagnose extra-pulmonary TB. However, in most cases people who tested negative for TB will truly not have TB and may have an alternative diagnosis such as a simple upper respiratory tract infection. Although the vision is that clinicians will be able to de-escalate treatment, I'm not sure this is likely to happen in practice and feel that once a decision to start TB treatment has been made, it is infrequent for that decision to be reversed, especially in the absence of microbiological/radiological testing. Furthermore, by the time clinicians evaluate people with a negative result to decide about TB treatment de-escalation, a simple upper respiratory tract infection might have spontaneously resolved, appearing to mimic a good response to TB treatment and increasing the chances of the clinician simply continuing TB treatment, inappropriately. An alternative approach of only treating people empirically who have very high-risk scores has its own limitations and I think that these complex trade-off issues merit further discussion in the manuscript.

Loss to follow-up. As the authors state, the score is not designed and should not be used to replace definitive microbiological testing, but rather aims to identify people who are at high risk of TB to facilitate empirical management with the aim of reducing loss to follow-up for the highest-risk patients. Therefore, I think the utility of this score depends both on the risk of TB, and on the risk of loss to follow-up for that individual person. Perhaps the authors could discuss in further detail some of the reasons why people are lost to follow-up (e.g. long travel times, costs, stigma) and the potential for derivation of similar algorithms to predict loss to follow-up that could be used alongside their tool to inform optimal decision making. They might also reflect upon other strategies that could be used to reduce pre-treatment loss to follow-up such as incentives and social support interventions. Overall, I think that there is a clear need for more research to address these important questions because the decision to start empirical treatment versus waiting for results/illness evolution depends on many factors, including local epidemiology, severity of illness, availability of further testing/chest radiography, and risk of loss to follow-up.

Decision curve analysis. The use of a decision curve analysis is appropriate and interesting. However, I think that people inherently find the interpretation of these analyses and graphics confusing. I think the authors have done a good job in trying to explain and provide context to the findings, but I wonder if they might be able to go even further to make these findings more accessible to people not familiar with the analysis strategy. I also wonder whether "cost:benefit" should be rephrased as "risk:benefit". Obviously a false-positive diagnosis incurs economic costs to both the person and health system, but also more broader risks associated with inappropriate treatment, adverse drug reactions, and failure to diagnose alternative pathologies. As highlighted above, perhaps some of these trade-off issues could be further explored in the discussion. I was also unclear what statistically significant means in the context of the decision curve analysis, and whether 95% confidence intervals are necessary.

Minor comments.

1. Apologies if I have missed it but it wasn't clear to me what symptoms triggered TB testing in the derivation cohort, and how this differed from how presumptive TB was defined to trigger TB testing in the validation cohort? Table 1 seems to suggest that people had to have at least one of cough, fever, weight loss, or night sweats to receive an Xpert test? Were the same criteria used in both cohorts? In some settings where this score might be most valuable, patients only receive a TB diagnostic test if they have productive cough for over two weeks. Could the authors potentially comment upon how this might affect the interpretation and use of their score?

2. Please provide more detail on the symptoms included in the model. Should clinicians ask about any cough of any duration, or specifically productive cough? Does weight loss have to be over a specified amount or is it simply self-reported? Did the authors have data available on haemoptysis? Do symptoms refer to a certain time period, i.e. if you have had one episode of fever three weeks ago but none since, does this count as having presented with fever?

3. Relatedly, how variables are measured is likely to impact on predictive performance. Diabetes and HIV are self-reported, but these diseases are substantially underdiagnosed in many of the settings where the authors propose the score could be used. I note that HIV status is known for all participants in this study, was that because rapid testing was done at recruitment and so clinicians had up to date results at the time of risk score assessment? What should be done in settings where somebody last had an HIV test three years ago and the assessing clinician does not have access to rapid testing now? Do we assume they are negative, potentially misclassifying them and underestimating their risk of TB? This is complex as in some sub-Saharan African countries HIV prevalence is extremely high, especially in people presenting to healthcare with symptoms. Similarly for diabetes.

4. I think Table 1 should be checked and include missing data for each variable. Some of the numbers don't add up, e.g. duration of TB symptoms 206+465=671. But n=707 for the column.

5. I think the number 1634 (total population in derivation cohort) is an error in the results. Also, the authors first report non-participation in the discussion. I think a flow chart demonstrating study design and recruitment would significantly add to the manuscript and help readers to understand what the authors did. Relatedly I would be interested to know what the prevalence of Xpert confirmed TB was in people presenting with symptoms at these clinics, to have a clearer understanding of what sampling fraction the authors recruited. How were the Xpert negative people randomly selected? These data and questions could all be easily presented and addressed in a figure.

6. I think a great strength of this study is the derivation and validation of the score in two diverse cohorts. Perhaps the authors could supplement the above suggested figure with some population-level data demonstrating the key differences between these two populations e.g. underlying HIV prevalence, poverty estimates. Photos may also be a nice addition.

7. I was surprised to see that both smoking and previous TB were associated with TB diagnosis in univariable analysis (as you would expect), but not in multivariable analysis. Could the authors comment upon which factors included in the final model are felt to be confounding or mediating these associations?

8. I agree with the importance of deriving a model that could be easily used in a resource-constrained primary health care clinic. However, I can think of a number of variables that might predict TB that have not been included, which would be easily measurable. For example, epidemiological factors such as if the person was known to have a recent TB contact. Relatedly, I think that although BMI might be difficult to measure in many settings, clinicians can easily visually estimate whether a person is underweight versus normal weight versus overweight. If BMI data are available to analyse then I would be interested to see if a variable of weight category predicts TB in these populations, and importantly whether it adds any predictive power to their model.

9. I think the discrimination of the score in the derivation cohort should be mentioned in the main text of the results and not just the figure.

10. Are the sensitivity and specificity in figure 4 based on the performance of the score in the derivation cohort, or the validation cohort? I suspect derivation cohort from studying the data.

11. I think the last sentence of the first paragraph of the discussion should be reworded, emphasising the need for further operational/impact research to investigate the use of a score-based strategy. Further validation alone will not demonstrate that the score can reduce loss to follow-up, transmission, and mortality.

12. I would be interested to know if smear microscopy was available in these clinics and if the authors have data on this? This is still the most commonly used method to diagnose TB globally and is available in many of the settings where this score might be most useful.

13. I think the sentence "These sensitivity analyses showed no material change in discrimination…." would be more appropriate in the results rather than the methods.

None of these comments should take away from the excellent and comprehensive work the authors have done. Please accept my apologies for the rather long review.

Open review

Signed: Matthew Saunders

-----------------------------------------------------------

Reviewer #3: This paper addresses the important question of whether a simple clinical risk score can be used to guide empirical TB treatment in patients presenting to primary care in low resource settings, whilst waiting for the results of microbiological TB tests. The authors develop a pragmatic score using data from a cohort in rural South Africa, which is they then validate in a prospective study in a very different population in Uganda, and find it performs similarly. Overall the study is well designed, and the paper very well written.

Comments:

-It would be good if the authors expanded on how they randomly selected the Xpert-negative patients in both cohorts (this can be added to Supplement if needed).

-The authors modified the intercept of the fitted regression model in the Ugandan cohort to mitigate for the different sampling fractions from the 2 cohorts. Is there a reason the sample fraction couldn't be increased from the SA cohort to be closer to the Ugandan cohort?

-Please can you further explain the Cox's linear-logistic regression model used in the validation cohort to assess calibration (it is not a regression model I am familiar with)

-Could the authors explain the decision curve analysis in more detail (again, this could go in the supplement if needed)? Specifically, a brief explanation of threshold probability and benefit may help the readers

-Linked to this, in figure 2 I suggest the authors present threshold probability as a %. Can the authors also clarify of the 'treatment for none' line is mean to be 'no empirical treatment' as is suggested in the figure legend (his maybe misinterpreted to mean no TB treatment, even if subsequent positive TB tests). it may also be clearer to colour the areas relating to the 'benefit over treating all' and 'benefit to no treatment'.

-Can the author's justify including a 20% TB prevalence in the models- it seems too high to me for TB prevalence in primary care settings (even amongst patients with TB symptoms). Is the TB prevalence in the 2 cohorts known? If not please cite relevant recent studies to justify TB prevalence in primary care settings. It may be make more sense to include a lower pre-test probability in the clinical tool and analyses.

-Do the authors have data on how many patients in the 2 cohorts were started on TB treatment 'empirically' (ie prior to microbiological test result)? It would be useful to compare the performance of the score to clinicians decision making on empirical treatment without a score.

-Similarly, the authors recognise the exclusion of patients treated for TB in the absence of positive microbiological results as a limitation. Can the authors report how many patients in each cohort were treated for TB but had negative Xpert and/or other microbiological TB tests? How many patients had radiological evidence of TB but negative tests? IT may be worth considering including these patients in another sensitivity analysis.

-There is a significant difference between self-reported diabetes prevalence in the South African and Ugandan cohorts (I suspect this will be similar for many settings outside South Africa, or in more rural settings). Can the authors expand on this, including reasons why it might be the case, in the discussion

-The authors used a simple imputation model for missing data using the mode value. Can the authors expand on how this translates to use in the field- eg if HIV status is unknown, how should that be scored when calculating risk score (maybe this could be added to figure 3 as a footnote)?

-Figure 4 would benefit from a legend explaining the different lines

-----------------------------------------------------------

Reviewer #4: "A simple clinical score for predicting active tuberculosis when same-day microbiological testing is unavailable" describes the derivation of a simple clinical score for use in assessing risk of tuberculosis (TB). This clinical score is expressedly designed not to require any computational resources, so as to be rapidly tallied by hand if needed, using just six broad [categories of] features (Table 2; age, sex, HIV status, classical TB symptoms, symptom duration, diabetes). Therefore, it is particularly suitable for deployment in under-resourced countries.

Initial model derivation was done using training data from 28 clinics in rural South Africa. Regression was performed on training data, and regression coefficients were converted into a scoring scheme, with further calibration and checks of clinical utility performed. External validation was performed on four clinics in urban Uganda. Simulation of different TB prevalence levels was performed as sensitivity analyses on the robustness of the model during validation, and comprehensive de novo analyses were performed to assess generalizability of the model construction (Appendix Section V)

It may be noted that a fair number of similarly-targeted low-resource clinical scores for TB prediction have been proposed previously, as discussed in the paragraph from Line 350 onwards. This proposal claims to be an improvement by not requiring criteria that require additional equipment as far as possible (e.g. scales, for body mass index), and also to provide rules that are also more generally applicable, rather than particularly to HIV-positive populations.

1. The authors might consider emphasizing that this clinical score is most appropriate for a sub-Saharan African population in the manuscript title (since this is where the training/validation data was exclusively drawn from), unless it can be convincingly shown to apply generally.

2. The authors might consider introducing and emphasizing the actual proposed scoring system (Figure 3) early on in the text, since it would appear to be of overriding interest for interested readers.

3. A brief description of the expected accuracy/sensitivity/specificity of the Xpert sputum test considered as gold standard here, might be included in the methods (it is stated in the limitations that Xpert is an imperfect test)

4. Appendix Section IV describes single imputation (with the most common value for each predictor), as likely practice in the field. However, it is not clear as to what the modal value for each predictor is, either in Figure 3 or the relevant tables. These model "default" values might be included (and explained) in Table 2/Figure 3.

5. The definitions of the classical WHO TB symptoms (cough/fever/night sweats/weight loss) appear to be rather underspecified. Is there an unspoken understanding amongst clinicians as to what qualifies for each of these symptoms, or are these symptoms expected to be wholly based on self-reporting? For example, the participant questionnaire for the Kharitode TB study [16] data that this publication is based on appears to specifically define weight loss as "more than 5 kg, or enough to make my clothes loose" - might this be relevant here?

6. Likewise, the clarity of the "Symptom duration > 14 days" predictor might be made more explicit, as to whether it refers to any symptom (assumed), or to some/all (i.e. Question 15 or 16 of Kharitode TB questionnaire)?

7. For Table 2, while the intent behind the score derivation is clear, the details remain slightly opaque (i.e. footnote c: "Each point in this simple clinical score is estimated by dividing the corresponding LASSO coefficient by the median value of four clustered coefficients (0.9) and rounding to the nearest integer..."); this might be explained slightly further - in particular, what is the significance of the four clustered coefficients, and of their median?

8. Figure 2 on cost-benefit decision curve analysis appears to indicate that the proposed clinical scoring system has potential to provide additional utility over basic "treat all" and "treat none" strategies (it is assumed that these costs & benefits are defined in economic terms?). For completeness, the relevant assumed values for costs & benefits given each patient scenario (i.e. "treat" and "TB", "treat" but "no TB", "no treat" and "no TB", "no treat" but "TB") might be specified and justified.

9. On Figure 2, it appears that various points on the "treatment based on clinical risk score" curve correspond to discrete patient scores; if so, the authors might consider labelling these scores on the curve, as with Figure 1 Panel B.

10. The reported predictive accuracies of the various previous models mentioned from Line 350 onwards, might be provided in the text for convenience.

11. The limitation that the proposed clinical scoring system should not be intended for community-based triage (Line 328) appears fairly significant, and might be emphasized in the abstract.

12. The limitation on requiring pre-test probability of true/Xpert TB appears fairly major in practice. The authors might consider further motivating the scoring system for decision curve analysis based on different assumed pre-test TB probabilities (i.e. Figure 2, for 5%/20% TB probability), and with incorrectly-assumed TB probabilities (i.e. assumed 10% probability, actually 20%). This is because it appears likely that the pre-test probability might not be estimated particularly accurately, especially where clinics do not encounter large numbers of patients.

Minor comments:

13. In Line 356, it is stated that "As such, the expected performance of these prediction rules might be lower, but they might also be limited in their ability to generalize to HIV-negative populations"; the authors might consider clarifying that the lower performance is due to the HIV-positive population, and/or change the "but" to "and".

-----------------------------------------------------------

Any attachments provided with reviews can be seen via the following link:

[LINK]

Submitted filename: PLOS Medicine review TW 20200525 2pm.docx

Click here for additional data file.

Submitted filename: 20200601_PLOS_riskscore_review.docx

Click here for additional data file.

23 Sep 2020

Dear Dr. Baik,

Thank you very much for re-submitting your manuscript "A simple clinical score for predicting active tuberculosis when same-day microbiological testing is unavailable: a multivariable prediction model using data from a clinical trial in South Africa and a population-based survey in Uganda" (PMEDICINE-D-20-01816R2) for review by PLOS Medicine.

I have discussed the paper with my colleagues and the academic editor and it was also seen again by three of the original reviewers. I am pleased to say that provided the remaining editorial and production issues are dealt with we are planning to accept the paper for publication in the journal.

The remaining issues that need to be addressed are listed at the end of this email. Any accompanying reviewer attachments can be seen via the link below. Please take these into account before resubmitting your manuscript:

[LINK]

Our publications team (plosmedicine@plos.org) will be in touch shortly about the production requirements for your paper, and the link and deadline for resubmission. DO NOT RESUBMIT BEFORE YOU'VE RECEIVED THE PRODUCTION REQUIREMENTS.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

In revising the manuscript for further consideration here, please ensure you address the specific points made by each reviewer and the editors. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments and the changes you have made in the manuscript. Please submit a clean version of the paper as the main article file. A version with changes marked must also be uploaded as a marked up manuscript file.

Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. If you haven't already, we ask that you provide a short, non-technical Author Summary of your research to make findings accessible to a wide audience that includes both scientists and non-scientists. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract.

We expect to receive your revised manuscript within 1 week. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

We ask every co-author listed on the manuscript to fill in a contributing author statement. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

If you have any questions in the meantime, please contact me or the journal staff on plosmedicine@plos.org.

We look forward to receiving the revised manuscript by Sep 30 2020 11:59PM.

Sincerely,

Thomas McBride, PhD

Senior Editor

PLOS Medicine

plosmedicine.org

------------------------------------------------------------

Requests from Editors:

1- Thank you for editing the title, please revise further to: “A clinical score for identifying active tuberculosis without microbiological testing: development and validation of a multivariable prediction model”

2- Second point of Author Summary, “*require* data…”

3- Thank you for including the TRIPOD checklist. Please add the following statement, or similar, near the start of the Methods: "This study is reported as per the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline developing, validating, or updating a prediction model (S1 Checklist). Please also replace the page numbers in the checklist with paragraph numbers per section (e.g. "Methods, paragraph 1"), since the page numbers of the final published paper may be different from the page numbers in the current manuscript.

4- Thank you for noting your data are available. At this time, please update your data statement to provide all the information necessary for researchers to access this dataset (eg, accession number, url or email address to contact if an application is required).

5- Please add a bit more context of why the study is important to the Abstract Background (i.e. note the limited availability of same-day microbiological testing an the risk of losing patients to follow-up).

6- In the Abstract Methods and Findings, please include the dates of the two cohorts, and some relevant demographic information (e.g., age, sex, HIV positivity).

7- The sentence on lines 44-46 needs a bit more context for what the scores of 3 and 7 mean.

8- Thank you for adding the limitations of the score on lines 46-47, please also include limitations of the study itself, though.

9- Similar to the Abstract Background section, the Abstract Conclusions could use a bit more context as well. Please address the study implications without overreaching what can be concluded from the data; the phrase "In this study, we observed ..." may be useful.

10- Around line 51, you could add "subject to prospective validation" or similar

11- Please move the reference brackets in front of punctuation. E.g.: “... the leading single-agent cause of infectious mortality worldwide [1].”

12- Table 1: please make clear that the numbers in the parentheses are percentages. Also, “irregular” is misspelt.

13- Table 2: please include p-values alongside the 95% CIs.

14- The numbers above the x-axis on figure 4 are a bit difficult to interpret. Please consider reformatting the “Score” and “Post-test probability” labels with smaller fonts and re-aligning. Perhaps the scales below could be smaller to allow more space?

15- Please use the "Vancouver" style for reference formatting, and see our website for other reference guidelines https://journals.plos.org/plosmedicine/s/submission-guidelines#loc-references

> Reference 7, 14, 15 may need reformatting.

> For reference 17, 18, please provide the date accessed.

> Missing information from Reference 31?

16- Figure S1 could be moved to the main text. The background renders in dark grey on my computer, which makes the arrows difficult to see.

17- The third image from Figure S2 may need to be edited to blur the face of the person on the right. I think he’s probably identifiable.

18- References to the “Appendix” throughout the the main text should be changed to reference the relevant supplementary file(s).

19- Please remove the funding information from the "Acknowledgements" at the end.

Comments from Reviewers:

Reviewer #2: No remaining requests.

Reviewer #3: I'm happy with the responses to the reviewer comments and revisions to the paper.

Reviewer #4: We thank the authors for addressing most of our points from the first review round, particularly the requested informative additions to Figure 4 (former Figure 2):

3. The expected sensitivity/specificity for Xpert (0.88/0.98) is now provided.

4. Single imputation values might be derived from Table 1 on hindsight, which might or might not be further emphasized.

5. WHO TB symptoms were briefly expounded on.

6. Symptoms > 14 days was clarified as being any TB symptoms.

7. Table 2 Footnote c was clarified by more fully describing a method to translate the LASSO coefficients to actionable scores.

8. The cost-benefit analysis of Figure 4 (former Figure 2) has been more quantitatively justified.

9. Sample patient scores were labeled on Figure 4 (former Figure 2) as requested.

10. Reported predictive accuracies of various models were mentioned as c-statistics.

11. The limitation on community-based triaged has been emphasized as suggested.

12. Additional caveats have been described discussion appropriate adjustments, were microbiological results not as expected according to the assumed TB probabilities. Ideally, the effect of incorrect TB probability assumptions might be more quantitatively explored.

Any attachments provided with reviews can be seen via the following link:

[LINK]

7 Oct 2020

Submitted filename: Letter to reviewers_final.docx

Click here for additional data file.

14 Oct 2020

Dear Dr Baik,

On behalf of my colleagues and the academic editor, Dr. Ankur Gupta-Wright, I am delighted to inform you that your manuscript entitled "A clinical score for identifying active tuberculosis while awaiting microbiological results: development and validation of a multivariable prediction model in sub-Saharan Africa" (PMEDICINE-D-20-01816R3) has been accepted for publication in PLOS Medicine.

PRODUCTION PROCESS

Before publication you will see the copyedited word document (within 5 busines days) and a PDF proof shortly after that. The copyeditor will be in touch shortly before sending you the copyedited Word document. We will make some revisions at copyediting stage to conform to our general style, and for clarification. When you receive this version you should check and revise it very carefully, including figures, tables, references, and supporting information, because corrections at the next stage (proofs) will be strictly limited to (1) errors in author names or affiliations, (2) errors of scientific fact that would cause misunderstandings to readers, and (3) printer's (introduced) errors. Please return the copyedited file within 2 business days in order to ensure timely delivery of the PDF proof.

If you are likely to be away when either this document or the proof is sent, please ensure we have contact information of a second person, as we will need you to respond quickly at each point. Given the disruptions resulting from the ongoing COVID-19 pandemic, there may be delays in the production process. We apologise in advance for any inconvenience caused and will do our best to minimize impact as far as possible.

PRESS

A selection of our articles each week are press released by the journal. You will be contacted nearer the time if we are press releasing your article in order to approve the content and check the contact information for journalists is correct. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact.

PROFILE INFORMATION

Now that your manuscript has been accepted, please log into EM and update your profile. Go to https://www.editorialmanager.com/pmedicine, log in, and click on the "Update My Information" link at the top of the page. Please update your user information to ensure an efficient production and billing process.

Thank you again for submitting the manuscript to PLOS Medicine. We look forward to publishing it.

Best wishes,

Thomas McBride, PhD

Senior Editor

PLOS Medicine

plosmedicine.org