Modelling study of the ability to diagnose acute rheumatic fever at different levels of the Ugandan healthcare system.

OBJECTIVE: To determine the ability to accurately diagnose acute rheumatic fever (ARF) given the resources available at three levels of the Ugandan healthcare system.
METHODS: Using data obtained from a large epidemiological database on ARF conducted in three districts of Uganda, we selected variables that might positively or negatively predict rheumatic fever based on diagnostic capacity at three levels/tiers of the Ugandan healthcare system. Variables were put into three statistical models that were built sequentially. Multiple logistic regression was used to estimate ORs and 95% CI of predictors of ARF. Performance of the models was determined using Akaike information criterion, adjusted R2, concordance C statistic, Brier score and adequacy index.
RESULTS: A model with clinical predictor variables available at a lower-level health centre (tier 1) predicted ARF with an optimism corrected area under the curve (AUC) (c-statistic) of 0.69. Adding tests available at the district level (tier 2, ECG, complete blood count and malaria testing) increased the AUC to 0.76. A model that additionally included diagnostic tests available at the national referral hospital (tier 3, echocardiography, anti-streptolysin O titres, erythrocyte sedimentation rate/C-reactive protein) had the best performance with an AUC of 0.91.
CONCLUSIONS: Reducing the burden of rheumatic heart disease in low and middle-income countries requires overcoming challenges of ARF diagnosis. Ensuring that possible cases can be evaluated using electrocardiography and relatively simple blood tests will improve diagnostic accuracy somewhat, but access to echocardiography and tests to confirm recent streptococcal infection will have the greatest impact. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

To our knowledge, this is the first study to describe ability to predict diagnosis of acute rheumatic fever using resources available at different levels of the health system in a low resource setting.

Data used in the modelling were obtained by research staff trained in acute rheumatic fever diagnosis; sensitivity and specificity of the predictive models would likely change if data was obtained by front-line health workers.

Data obtained from this study can be used in formulating alternative strategies for diagnosis of acute rheumatic fever in low-resource settings.

Introduction

Acute rheumatic fever (ARF) is an autoimmune disease that results from infection with Group A streptococcus in susceptible individuals.1 Recurrent episodes of ARF lead to development of rheumatic heart disease (RHD). RHD is the gravest and only long-term sequela characterised by irreversible damage of heart valves. RHD affects over 40 million people worldwide and is responsible for more than 300 000 deaths annually.2 The greatest burden of RHD is found in the world’s poorest nations where most patients present for the first time with severe disease and mortality is high.3 4 Despite high rates of RHD in these settings, few patients are diagnosed with ARF. Early detection of ARF ensures timely initiation of benzyl benzathine penicillin G that can prevent recurrent ARF and worsening of RHD5 6

The reasons for underdetection of ARF are likely multifactorial but have not been systematically explored. Theories include (1) Inaction to receive appropriate care, that is, poor health seeking behaviour that results from low levels of community awareness, (2) clinical overlap with common diseases that are more familiar to health workers, (3) subtle presentation of the disease in endemic settings, and (4) the inability of providers to make a diagnosis of ARF in low-resource settings due to limited diagnostic resources. The latter concern was the focus of this study.

The diagnosis of ARF is complex. There is no single diagnostic test for ARF, thus diagnosis relies on the application of clinical criteria, most commonly the Jones criteria, and the ability to rule out competing diagnoses.7 Echocardiography and laboratory criteria are part of the evaluation of major and minor manifestations in the Jones criteria, and confirmation of recent streptococcal infection is needed to enter the diagnostic pathway for most cases. In many low-resource settings, these tests are simply not available at the community level, and the ability to predict ARF without them is not known.

We assessed the ability to accurately predict ARF given the resources that are commonly available at three levels/tiers of the Ugandan healthcare system. We conducted this study within the context of a broader ARF epidemiological study in Uganda, where gold standard diagnosis applying Jones criteria was available.

Methods

Study design

This was a predictive modelling study of the ability to diagnose ARF at different levels of the Ugandan healthcare system.

Predictive model derivation

Data source

We obtained data from a large epidemiological study on rheumatic fever conducted in three districts of Uganda (Lira, Kampala and Mbarara) between June 2017 and June 2020.8 Following community sensitisation about signs and symptoms of ARF participants were included in the above epidemiological study if they were aged 3–17 years, presented with fever and joint pain or suspected to have carditis or chorea. A cardiologist’s review of all enrolled participants was done and participants assigned to four categories following application of the 2015 revised Jones criteria for moderate/high risk populations (online supplemental table 1): (1) definite ARF (diagnosis made based Jones criteria), (2) possible ARF (defined as participants who had evidence of streptococcal infection but partially fulfilled Jones criteria), (3) known alternate diagnosis (participants who had evidence of an alternate diagnosis) and (4) unknown diagnosis (all other participants)

For this particular study, we included participants that presented with fever and joint pain, the most common, challenging and non-specific diagnostic presentation of ARF. We excluded participants presenting with fulminant carditis or chorea as these are standalone criteria for ARF, and more easily distinguished on physical exam alone. Participants with final diagnosis of definite ARF, known alternate diagnosis or unknown alternate diagnosis were included in the predictive model derivation cohort, but those with possible ARF, given the non-specific phenotype, were excluded.

Diagnostic resources typically available at three tiers of the Ugandan health system were determined based on a health facility survey conducted as part of the broader research program.9

Variables

Variables that might positively or negatively predict rheumatic fever, including but not limited to those in the Jones criteria, were selected (table 1). Tier 1 (community healthcare centre) variables included only features that could be determined by history and clinical exam. Tier 2 (district hospital) variables included those in tier 1 plus basic laboratory testing typically available at the district level. Tier 3 included variables in tiers 1 and 2, as well as advanced tests that are available at the National referral hospital. Antideoxyribonuclease-B antibody testing, one of the accepted forms of confirming recent group A streptococcal infection, was not included in the model as this testing is not currently available in-country.

Table 1

Variables selected from the rheumatic fever database

Tier 1Community health centres	Tier 2District hospital	Tier 3National referral hospital
Demographics Sex History Sore throat in the past 4 weeks Family history of ARF/RHD Number of fever days Medication given for joint pain prior to visit Joint pain history (type of involvement) History of viral symptoms (rhinorrhoea/cough) Physical exam Heart murmur Tachycardia Joint assessment Erythema marginatum Subcutaneous nodules	Tier 1 variables plusLaboratory White cell count Haemoglobin level Malaria test (point of care or blood smear) ECG PR interval	Tier 1 and 2 variables plusLaboratory ESR CRP Streptococcal evidence (Throat culture, ASO titres) Echocardiogram Rheumatic carditis on echocardiogram

Criteria in bold reflect those included in the Jones criteria.

ARF, Acute rheumatic fever; ASO, antistreptolysin O; CRP, C reactive protein; ESR, erythrocyte sedimentation rate; RHD, rheumatic heart disease.

Patient and public involvement

There was no patient or public involvement in research design, recruitment, study conduct, or study supervision/advisory. Participants and their guardians were informed of the parent study (ARF incidence) results individually and radio messaging was used to communicate the results of the parent study to the community at large.

Statistical analysis

Medians with IQRs, or frequencies with percent of total, were used to describe patient demographic, clinical, and diagnostic characteristics. Multivariable logistic regression was used to estimate ORs and 95% CIs for ARF according to the three tiers of predictors using the lrm function in the rms package (V.6.0.0) in R (V.3.6.1). Models were built sequentially, with the first tier model (model 1) including information on patient history and clinical exam available at the community healthcare centre. The second tier model (model 2) additionally incorporated laboratory and ECG testing available at the district hospital. Advanced laboratory and echocardiogram testing available at the national hospital were further added to the third tier model (model 3). Clinical variables of interest with extremely low counts (subcutaneous nodules and erythema marginatum) were not included. Restricted cubic spline terms (three knots placed at the 10th, 50th and 90th percentiles) were included for continuous predictors to account for potential non-linear associations with ARF. Two-way interactions were assessed for each tier and retained where the Wald Chi-square P value was less than 0.3 resulting in the interaction of days of fever by white cell count (WCC) retained in the second and third tier models. The Akaike information criterion (AIC), adjusted R2, concordance statistic (C statistic; a.k.a. area under the receiver operating characteristic (AUC) curve), Brier score and adequacy index (log-likelihood base model/ log-likelihood new model) were used to assess model performance. Formal testing was performed by likelihood ratio tests (LRT) of nested models and improvement in the AUC curve. Optimism-corrected values were obtained using 200 bootstrap resamples to provide estimates of the out-of-sample performance using the validate function in the rms package. Given the variation in units of measurement, ORs for a unit increase equal to the IQR are presented in tabular form for all continuous variables to aid interpretation. Plots of predicted probabilities of ARF calculated at the median value for continuous covariates, and the most common category for categorical covariates, are provided to highlight the modelled functional form for continuous predictors. Sensitivity and specificity were calculated from the predicted probabilities using Youden’s index to evaluate classification accuracy.

Results

The 503 subjects are included in the epidemiologic study database. Of these, 143 (30%) had definite ARF. Presence of a heart murmur was more common in participants with definite ARF (31% vs 16%) as was having a family history of RHD (7.7% vs 0.6%). Polyarthralgia was more common in participants without ARF while polyarthritis was more common in those with ARF (table 2).

Table 2

Predictor distribution according to ARF status and tier

	No ARF (n=360)	Definite ARF (n=143)
Tier 1
Female sex	190 (52.8)	68 (47.6)
Sore throat in past 4 weeks	106 (29.4)	40 (28.0)
Heart murmur	58 (16.1)	44 (30.8)
Family history of ARF/RHD	2 (0.6)	11 (7.7)
Tachycardia for age	103 (28.6)	47 (32.9)
Days of fever	3.00 (1.75, 5.00)	3.00 (1.25, 5.00)
Medication given for joint pain prior to visit
Yes	224 (62.2)	99 (69.2)
No	135 (37.5)	44 (30.8)
Missing	1 (0.3)	0 (0.0)
Viral symptoms (rhinorrhoea/cough)	257 (71.4)	105 (73.4)
Joint Assessment Monarthritis	15 (4.2)	13 (9.1)
Polyarthritis	42 (11.7)	46 (32.2)
Polyarthralgia	264 (73.3)	85 (59.4)
Subcutaneous nodules*	0 (0.0)	1 (0.7)
Erythema marginatum*	0 (0.0)	0 (0.0)
Tier 2
PR interval ≥180 ms
Yes	12 (3.3)	10 (7.0)
No	343 (95.3)	131 (91.6)
Missing	5 (1.4)	2 (1.4)
White cell count	5.56 (4.26, 7.60)	7.11 (5.50, 10.59)
Haemoglobin	12.70 (11.40, 13.62)	11.90 (10.40, 13.10)
Confirmed malaria infection	135 (37.5)	23 (16.1)
Tier 3
ESR (mm/hour)	15.00 (7.00, 33.00)	40.00 (13.25, 65.00)
CRP (mg/L)	3.80 (0.66, 28.76)	31.61 (6.48, 80.22)
Streptococcal evidence	28 (7.8)	90 (62.9)
Carditis on echocardiogram	3 (0.8)	32 (22.4)

*Denotes variable with low count and not used in modelling. Categorical variables n (%) and Continuous variable median (IQR). Missing days of fever for five patients with definite ARF and eight patients with no ARF. Missing ESR (mm/hour) for one patient with definite ARF and four patients with no ARF. Missing CRP (mg/L) for one patient with no ARF.

ARF, acute rheumatic fever; CRP, C reactive protein; ESR, erythrocyte sedimentation rate; RHD, rheumatic heart disease.

Predictive performance of the models

Table 3 and figure 1 show the predictive performance of the models comprising the tiered predictors. Models 1 and 2 had acceptable predictive performance; however, model 3 had the lowest AIC score and superior discriminative performance (corrected c-statistic=0.91). The adequacy index for the tier 1 and 2 predictor set, when compared with the tier 3 predictor set, was 0.39 (-2 log likelihood model 2 / (−2 log likelihood model 3) highlighting that the tier 1 and 2 predictors explained a modest fraction of the log likelihood obtained with the full model. The adequacy index for the tier 1 vs tier 2 predictor sets was 0.62. The LRT and AUC p values showed sequential improvement in performance for the predictors included at later tiers (p<0.001). Comparison of the apparent and optimism corrected estimates suggested limited overfitting based on the resampled data.

Table 3

Model performance

	Model 1	Model 2	Model 3
AIC	533.48	492.81	311.84
C statistic
Apparent	0.72	0.80	0.94
Corrected*	0.69	0.76	0.91
Adjusted R square
Apparent	0.20	0.32	0.68
Corrected*	0.15	0.22	0.59
Brier score
Apparent	0.17	0.15	0.08
Corrected*	0.18	0.17	0.10
Sensitivity	0.66	0.77	0.84
Specificity	0.68	0.67	0.87
Adequacy index		0.62	0.39
LRT p value†		<0.001	<0.001
AUC p value†		<0.001	<0.001

Adequacy index: adequacy of the model ignoring the new predictors (log-likelihood base model/log-likelihood new model).

*Out-of-sample estimates obtained using bootstrap resampling (200 samples)

†Values for model 2 compare the model 2 versus model 1. Values for model 3 compare model 3 versus model 2

AIC, Akaike information criterion; AUC, area under the curve; LRT, likelihood ratio tests.

Figure 1

Observed receiver operating curves. AUC, area under the curve.

Predictors of ARF

Table 4 shows the ORs and 95% CIs for ARF according to tiered predictors. Having a heart murmur (OR=2.19 (95% CI 1.32 to 3.63)), family history of ARF/RHD (OR=11.82 (95% CI 2.43 to 57.50)), monarthritis (OR=4.93 (95% CI 1.85 to 13.15)), polyarthritis (OR=9.75 (95% CI 3.76 to 25.28)), and polyarthralgia (OR=2.57 (95% CI 1.08 to 6.1)) were associated with ARF in model 1 (tier 1 predictors). Tier 1 predictors generally retained positive associations with ARF with the inclusion of tier 2 (model 2) and tier 3 (model 3) predictors. For tier 2 predictors, a PR interval ≥180 units exhibited an OR of 3.47 (95% CI 1.2 to 10.06) for ARF, and a comparison of the 75th vs 25th percentile values for WCC exhibited an OR of 1.76 (95% CI 1.07 to 2.9), when added to the tier 1 predictors. All tier 3 predictors were associated with an increased odds of ARF including ESR (OR=4.38 (95% CI 2.00 to 9.60)), CRP (OR=3.63 (95% CI 1.62 to 8.14)), strep (OR=32.73 (95% CI 13.87 to 77.28)) and carditis (OR=143.55 (95% CI 23.13 to 890.93)); however, the presence of carditis on echocardiography was estimated with limited precision. The non-linear conditional association for the continuous variables in each model are provided in online supplemental figure 1.

Table 4

ORs and 95% CIs for ARF according to tiered predictors

	Model 1	Model 2	Model 3
	OR (95% CI)	OR (95% CI)	OR (95% CI)
Female sex	0.80 (0.52 to 1.24)	0.74 (0.46 to 1.18)	0.91 (0.47 to 1.74)
Sore throat in past 4 weeks	1.00 (0.61 to 1.62)	0.97 (0.57 to 1.64)	1.55 (0.74 to 3.23)
Heart murmur	2.19 (1.32 to 3.63)	2.18 (1.20 to 3.94)	1.03 (0.40 to 2.65)
Family history of ARF/RHD	11.82 (2.43 to 57.50)	10.12 (1.84 to 55.82)	7.64 (0.54 to 107.63)
Tachycardia	1.32 (0.82 to 2.12)	1.22 (0.72 to 2.06)	0.93 (0.44 to 1.95)
Days of fever	0.87 (0.57 to 1.32)	0.84 (0.53 to 1.32)	0.47 (0.24 to 0.91)
Medication for joint pain prior to visit	1.40 (0.88 to 2.22)	1.17 (0.71 to 1.93)	1.19 (0.59 to 2.42)
Viral symptoms (rhinorrhoea/cough)	1.21 (0.74 to 1.98)	1.32 (0.78 to 2.23)	1.44 (0.70 to 2.96)
Joint Assessment
Monarthritis	4.93 (1.85 to 13.15)	7.50 (2.52 to 22.34)	4.82 (0.91 to 25.64)
Polyarthritis	9.75 (3.76 to 25.28)	16.32 (5.58 to 47.73)	4.29 (0.90 to 20.46)
Polyarthralgia	2.57 (1.08 to 6.10)	4.12 (1.57 to 10.79)	2.35 (0.58 to 9.56)
PR interval>=180 ms		3.47 (1.20 to 10.06)	1.82 (0.37 to 8.82)
White blood cell count		1.76 (1.07 to 2.90)	0.96 (0.50 to 1.84)
Haemoglobin		0.78 (0.55 to 1.10)	0.98 (0.65 to 1.47)
Confirmed malaria infection		0.26 (0.15 to 0.47)	0.07 (0.03 to 0.17)
ESR (mm/hour)			4.38 (2.00 to 9.60)
CRP (mg/L)			3.63 (1.62 to 8.14)
Streptococcal evidence			32.73 (13.87 to 77.28)
Carditis on echocardiogram			143.55 (23.13 to 890.93)

ARF, acute rheumatic fever; CRP, C reactive protein; ESR, erythrocyte sedimentation rate; RHD, rheumatic heart disease.

Discussion

In this study, we have described the ability to accurately predict ARF given the resources that are commonly available at three levels of the Ugandan healthcare system. These data demonstrate that the average clinician working at the health centre or district hospital level in Uganda, when faced with a child suffering with fever and joint pain, has inadequate resources to make the diagnosis of ARF. They also highlight that while the 2015 Jones criteria revision improved the sensitivity of ARF diagnosis in moderate/high risk areas, in practical terms,7 the ability to apply these criteria are severely limited by resource constraints and lack of access to diagnostic testing required for this common problem in regions most endemic for RHD.

While improving the prevention of RHD in low-income and middle-income countries (LMICs) will require a multifaceted approach, accurate diagnosis of ARF at the community level is exceedingly important. For children experiencing their first attack of ARF, initiation of secondary antibiotic prophylaxis can reduce recurrent ARF by 70%–80%.10 Missed ARF diagnosis results in late or absent detection of these children, many of whom will go on to present only when they have progressed to advanced stages of RHD.11

Improving ARF diagnosis at the community level is no simple task. Our data suggest that testing that is commonly available at the district level (tier 2), including ECG, WCC, haemoglobin and malaria confirmation, adds somewhat to the diagnostic accuracy found at community health centres (tier 1), where evaluation is largely limited to history and physical exam. Testing available at the tier 3, National Referral Hospital level, improved the ability to predict ARF, with confirmation of streptococcal exposure and demonstration of carditis on echocardiogram contributing most to the model performance. Unfortunately, for most children with suspected ARF, the distance to the National Referral Hospital, most often located in capital cities in LMICs, is prohibitive, and these resources remain out of reach.

The infrastructure needed for confirmation of recent streptococcal infection is complex. No single test has high predictive value, and often a combination of throat culture, requiring a microbiological laboratory, and streptococcal antigen testing is required to find a positive result. Recently, progress has been made in development of a dried blood spot antigen test for antistreptolysin O titres (ASOT) that could bring these critical tests to LMIC communities.12 Ongoing work is focused on expanding the antigens that could be detected by this testing method and in operationalising these tests, so that they could be scaled for community application.

Expanding echocardiography to the community level has been explored and may be the best opportunity to identify those at highest risk. In both Rwanda13 and Uganda,14 integration of echocardiography in the hands of nurses and other frontline health providers, has been tested. In Uganda, an innovative model of telemedicine mentorship resulted in a high accuracy of front-line providers to diagnose many forms of heart failure, including valvular heart disease.14 Cost analysis of non-communicable disease clinics, with integrated nurse-led echocardiography in Rwanda, was also favourable.15 Investment in echocardiography and ultrasound in general, could improve and support health service delivery such as the example of integrated cardiac and obstetrical ultrasound piloted in two communities in rural Uganda.16 Much work is ongoing in this space globally. Artificial intelligence, both for image acquisition and diagnosis also hold promise to help expand access to echocardiography.17 Artificial intelligence combined with point of care ultrasound scan could be used by frontline health workers to improve diagnosis of ARF within the community.

These data also urge us to consider non-resource intensive strategies that might improve ARF diagnosis and appropriate triage and referral, even without infrastructure changes. Improved healthcare worker and community education is essential to improve ARF awareness.18 19 Consideration of alternate diagnostic strategies that will enable children to receive diagnosis close to home is critical. While the Jones criteria remain the gold standard for ARF diagnosis, the inability to apply them in most LMIC settings, renders them practically ineffective. Simplified diagnostic strategies that rely on a high suspicion for ARF in children presenting with fever and joint pain, and provide protection through the initiation of secondary prophylaxis, until full evaluation can be completed, might bridge this gap. The American Heart Association in partnership with the global RHD advocacy organisation REACH, is currently working on a frontline healthcare provider toolkit that could achieve this goal (stopRHD.org).

This study has several limitations. Physical exam data and history used in the modelling was obtained from evaluations done by research staff trained in ARF diagnosis. The knowledge level and skill present in frontline healthcare workers is likely lower, which would further reduce the performance of tier 1 variables. Similarly, the echocardiographic images and ECG obtained from the parent study were interpreted by cardiologists in the USA and Uganda. If diagnosis was made locally, it is likely that the sensitivity and specificity would be affected, changing the performance of this model in a practical roll-out.

We conclude that ultimately reducing the burden of RHD in LMICs requires overcoming the challenge of ARF diagnosis. While the inability to diagnose ARF at the community level was predictable, these data provide the evidence needed to further explore infrastructure investments and novel strategies for community healthcare strengthening that can address this gap. Further research in this area is needed and will have broad implications for the development of RHD national action plans, which the world committed to in the 2018 RHD global resolution.20