Evaluation of the RABBIT Risk Score for serious infections.

OBJECTIVE: To evaluate the Rheumatoid Arthritis Observation of Biologic Therapy (RABBIT) Risk Score for serious infections in patients with rheumatoid arthritis (RA).
METHODS: The RABBIT Risk Score for serious infections was developed in 2011 on a cohort of RA patients enrolled in the German biologics register RABBIT between 2001 and 2007. To evaluate this score, we used data from patients enrolled in RABBIT after 1 January 2009. Expected numbers of serious infections and expected numbers of patients with at least one serious infection per year were calculated by means of the RABBIT Risk Score and compared with observed numbers in the evaluation sample.
RESULTS: The evaluation of the score in an independent cohort of 1522 RA patients treated with tumour necrosis factor α (TNFα) inhibitors and 1468 patients treated with non-biological disease-modifying antirheumatic drugs (DMARDs) showed excellent agreement between observed and expected rates of serious infections. For patients exposed to TNF inhibitors, expected as well as observed numbers of serious infections were 3.0 per 100 patient-years (PY). For patients on non-biological DMARDs the expected and observed numbers were 1.5/100 PY and 1.8/100 PY, respectively. The score was highly predictive in groups of patients with low as well as with high infection risk.
CONCLUSIONS: The RABBIT Risk Score is a reliable instrument which determines the risk of serious infection in individual patients based on clinical and treatment information. It helps the rheumatologist to balance benefits and risks of treatment, to avoid high-risk treatment combinations and thus to make informed clinical decisions. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

Serious infections are a major concern in patients treated with cytokine inhibitors or other biological agents. Randomised clinical trials as well as observational cohort studies have shown that there is an increased risk of serious infection in patients treated with tumour necrosis factor α (TNFα) inhibitors. The risk, however, seems to decrease over time: at the start of treatment, an up to 4.5-fold risk was reported,5 whereas after 1 year of treatment no increase in risk over conventional therapy was seen in observational cohort studies.1 5 6 When interpreting data from observational studies, it is important to distinguish the changes in overall risk in the cohorts from the changing risks occurring in individual patients. We showed previously that the decline in risk over time observed in cohorts can be attributed to (i) the increased loss to follow-up of patients with higher susceptibility to infections, as well as (ii) a risk decline in individual patients attributable to improvement in clinical status and reduced glucocorticoid (GC) intake. Taking baseline characteristics and time-dependent changes in the clinical status and treatment of individual patients into account, we were able to calculate the expected infection risk for specific patient profiles and for individual therapies at any time point during the course of treatment.1 This risk estimate allows rheumatologists to align their therapy to the expected risk of individual patients.

However, the risk calculation was based on one single cohort. The aim of this analysis was to evaluate the score with a new cohort of patients not included in its development.

We used data from the German biologics register RABBIT, which is the German acronym for ‘Rheumatoid Arthritis (RA) Observation of Biologic Therapy’.

Methods

The RABBIT Risk Score was developed on patients enrolled in RABBIT before 1 January 2007.1 For the evaluation of the risk score, we used an independent cohort of patients who were enrolled in RABBIT between January 2009 and January 2012 at start of treatment with a TNFα inhibitor or a non-biological disease-modifying antirheumatic drug (nbDMARD). Using the RABBIT Risk Score, we calculated for each patient the expected likelihood of a serious infection based on the individual risk profile.

In our previous analysis we had estimated the number of serious infections per 100 patient-years by means of generalised estimation equations with a Poisson link function. The Poisson link is appropriate for counts such as numbers of infections. A more intuitive measure, however, is the percentage of patients with at least one infection per year. We therefore calculated in addition a modified version of the risk score using the development sample of patients enrolled before 1 January 2007. A complementary log–log link function was applied to estimate the probability of at least one serious infection per year per patient. Both versions of the RABBIT Risk Score contain the following risk factors which had proven predictive of the infection risk in our previous analysis: age (up to or above 60), functional status (assessed by the Hannover Functional Status Questionnaire, Funktionsfragebogen Hannover, FFbH), specific co-morbidities (chronic renal or lung disease), treatment with GCs (≤7.5 mg/day, 7.5–15 mg/day or above), number of previous DMARD treatment failures, previous serious infection, and current treatment with a TNFα inhibitor or an nbDMARD. The expected numbers of serious infections as well as the expected numbers of patients with at least one infection were calculated according to the two versions of the RABBIT Risk Score and then compared to the observed numbers in the cohort or within defined subgroups of patients.

Table 1 shows the risk factors included in the score, their weights and the formula for the calculation of both versions of the score. Since the German instrument FFbH for the measurement of functional capacity can be transformed into Health Assessment Questionnaire (HAQ) scores using a formula published by Lautenschlager et al,7 the RABBIT Risk Score can also be calculated with HAQ values.

Table 1

Calculation of the RABBIT Risk Score

Risk factors		V.1: Number of serious infections per 100 PYs	V.2: Percentage of patients with at least one infection per year
Intercept	Always add	−3.996	−4.191
Age	If age >60 add	0.479	0.470
Function (FFbH)	Add	−0.01014*FFbH	−0.00865*FFbH
Alternatively: HAQ	Add	0.362(HAQ-3.16)	0.309(HAQ-3.16)
Chronic lung disease	If yes add	0.522	0.484
Chronic renal disease	If yes add	0.441	0.415
Previous serious infection	If yes add	0.748	0.992
Number of treatment failures	If >5 add	0.443	0.397
Mean glucocorticoid dose	If 7.5–14 mg/day add	0.756	0.782
Mean glucocorticoid dose	If ≥ 15 mg/day add	1.554	1.355
Treatment with TNF inhibitor	If yes (last 3 months) add	0.593	0.589
Calculate the sum of the corresponding values		Sum1	Sum2
Rabbit Risk Score	Calculate

FFbH, Hannover Functional Status Questionnaire, Funktionsfragebogen Hannover; HAQ, Health Assessment Questionnaire; PY, patient-years; RABBIT, Rheumatoid Arthritis Observation of Biologic Therapy; TNF, tumour necrosis factor.

To evaluate the agreement between observed and expected event rates, we applied the Hosmer–Lemshow test to subgroups of patients was well as to 10 groups established according to the expected risk (figure 1).

Figure 1

Expected and observed rates of serious infections per 100 patient-years by deciles of expected rates.

Results

Data on 1522 RA patients treated with TNFα inhibitors and 1468 patients treated with nbDMARDs were available. The median observation time was 1.6 years. Table 2 compares baseline characteristics of patients enrolled in the development cohort to those enrolled in the evaluation cohort. The differences in patient characteristics, co-medication and previous treatments between the original sample and the evaluation cohort indicate the changes in treatment strategies for RA in Germany. Patients enrolled in the register between 2001 and 2007 had higher disease activity in both the anti-TNF and the nbDMARD cohorts, they had lower functional capacity, more previous DMARD failures, were more often rheumatoid factor positive and were more often treated with GCs ≥7.5 mg/day than patients enrolled between 2009 and 2012. However, there was no difference in the prevalence of co-morbid chronic lung or renal disease.

Table 2

Baseline characteristics of patients

	Original sample		Evaluation sample
	Anti-TNF	nbDMARD	Anti-TNF	nbDMARD
N	3271	1773	1522	1468
Female, n (%)	2556 (78.1)	1394 (78.6)	1143 (75.1%)	1106 (75.3%)
Age	53.8 (12.3)	56.2 (11.5)	55.8 (12.9)	58.2 (12.5)
Disease duration (years), median (IQR)	9 (5, 16)	6 (3, 12)	7 (3, 14)	4 (2, 9)
Follow-up time (years), median (IQR)	3.1 (2.1, 4.9)	3.3 (2.5, 5.0)	1.5 (0.6, 2.1)	1.6 (0.9, 2.4)
Rheumatoid factor positive, n (%)	2624 (80.2)	1271 (71.7)	1095 (72.8)	866 (59.0)
DAS28	5.7 (1.2)	5.1 (1.3)	5.1 (1.3)	4.6 (1.3)
FFbH	57.0 (23.0)	66.6 (21.5)	64.8 (22.8)	71.2 (21.8)
Smoking never, n (%)	1027 (47.0)	585 (45.6)	667 (44.5)	693 (48.0)
No. of previous DMARDs	3.3 (1.3)	1.8 (1.0)	2.5 (1.1)	1.6 (0.9)
No. of previous biologicals	0.23 (0.6)	0.01 (0.1)	0.24 (0.57)	0.07 (0.34)
Glucocorticoids 7.5–14 mg/day, n (%)	1027 (31.4)	386 (21.8)	383 (25.2)	171 (11.6)
Glucocorticoids ≥15 mg/day, n (%)	491 (15.0)	147 (8.3)	117 (7.7)	50 (3.4)
Chronic lung disease, n (%)	246 (7.5)	112 (6.3)	111 (7.3)	88 (6.0)
Chronic renal disease, n (%)	139 (4.3)	31 (1.8)	57 (3.8)	27 (1.8)

Values are means and SDs if not otherwise specified.

; nbDMARD, non-biological disease-modifying antirheumatic drug; TNF, tumour necrosis factor.

We applied the original RABBIT Risk Score based on the number of infections per 100 patient-years to the evaluation cohort (table 3, columns 4–7). According to the score, 69.6 infections were expected under TNF inhibitors. With 69 infections observed in the evaluation cohort (corresponding to a rate of 3.0 per 100 patient-years) the agreement with the estimate was high. This was also the case in subgroups of patients with specific risk factors where we found a high concordance between expected and observed rates (p=0.32 for disagreement). When we applied the second version of the score which gives the percentage of patients affected by at least one infection per year, thus including the first infection per patient only, we again found high agreement between numbers of observed and expected cases (p=0.24 for disagreement). In the patient groups with higher risks, the observed numbers of infections were insignificantly higher than expected.

Table 3

Expected and observed numbers and rates of serious infections for V.1 (per 100 patient-years) and V.2 (% of patients with at least one serious infection during 12 months)

	n	PY	V.1: Number of serious infections per 100 PY (CI)				V.2: Number and percentage of patients affected by at least one serious infection per year (CI)
			Exp. n	Obs. n	Exp. rate/100 PY	Obs. rate/100 PY	Exp. n	Obs. n	Exp. per year (%)	Obs. per year (%)
TNFα inhibitor, no risk factor	764	1059	16.8	16	1.6	1.5 (0.9–2.5)	15.3	16	1.4	1.5 (0.9–2.5)
nbDMARD, no risk factor	632	816	7.0	6	0.9	0.7 (0.3–1.6)	6.4	6	0.8	0.7 (0.3–1.6)
TNFα inhibitor, ≥1 risk factor*, no GC	635	871	28.9	37	3.3	4.3 (3.0–5.9)	25.4	33	2.9	3.8 (2.6–5.3)
nbDMARD, ≥1 risk factor*, no GC	674	939	15.5	19	1.7	2.0 (1.2–3.2)	14.0	19	1.5	2.0 (1.2–3.2)
TNFα inhibitor + GC, no other risk factor	225	196	8.6	5	4.4	2.6 (0.8–6.0)	7.4	4	3.8	2.0 (0.6–5.2)
nbDMARD + GC, no other risk factor	128	86	1.9	2	2.2	2.3 (0.3–8.4)	1.7	2	2.0	2.3 (0.3–8.4)
TNFα inhibitor, ≥1 risk factor* + GC	206	160	15.3	11	9.6	6.9 (3.4–12.3)	12.8	11	8.0	6.9 (3.4–12.3)
nbDMARD, ≥1 risk factor* + GC	141	96	5.1	8	5.3	8.3 (3.6–16.4)	4.4	8	4.6	8.3 (3.6–16.4)
TNFα inhibitor total	1830	2286	69.6	69	3.0	3.0 (2.3–3.8)	60.9	64	2.7	2.8 (2.2–3.7)
nbDMARD total	1575	1937	29.6	35	1.5	1.8 (1.2–2.5)	26.6	35	1.4	1.8 (1.2–2.5)

Bold: The confidence intervals of the observed rates overlap with the expected rates. There is no other test for insignificant difference necessary or sensible.

*At least one of: chronic lung disease, chronic renal disease, age above 60 years, previous serious infection, high number of DMARD failures. GC: treatment with glucocorticoids ≥7.5 mg/day prednisolone equivalent.

exp, expected; obs, observed; nbDMARD, non-biological disease-modifying antirheumatic drug; PY, patient-years; TNF, tumour necrosis factor.

The comparison of expected and observed event rates in deciles of risk scores using the Hosmer–Lemshow test resulted in a high agreement as shown in figure 1. The p values for difference between observed and expected were 0.85 for all patients, 0.29 for patients exposed to TNF inhibitors and 0.49 for those exposed to nbDMARDs.

We developed an online risk score calculator which can be accessed via the RABBIT website (http://www.biologika-register.de/risk score). The following example shows how the score can be used in daily practice to balance risks from different treatments: if we consider a patient aged 65 with RA and chronic obstructive pulmonary disease, this patient's estimated risk under methotrexate and 7.5 mg/day GCs is 4.7%. If the GC dose is increased to 15 mg/day, the risk increases to 8.2%. Adding a TNF inhibitor would further increase the risk to 14.3%. However, if therapy is effective and the GC dose can be tapered down to 5 mg/day, the risk declines to 3.9%.

The calculator also contains weights for the more recently licensed substances rituximab, tocilizumab and abatacept, based on data from 1343, 825 and 444 patients exposed to the respective agents since 2007. Using the formula for TNF inhibitors, we calculated the difference between expected and observed incidences of at least one infection per year. The observed incidences were very similar to those expected (factors compared to TNF inhibitors: 1.15 for tocilizumab, 0.92 for rituximab and 0.82 for abatacept). The weights for the newer substances need further evaluation in a replication cohort.

Discussion

In daily clinical practice, choices have to be made between treatment strategies with different benefit–risk ratios. Scoring systems that enable calculation of absolute numbers of expected adverse events per year or per 100 patient-years help the treating rheumatologist to make informed decisions, taking the overall risk resulting from treatments and their alternatives as well as specific risks resulting from the demographic and clinical situation of individual patients into account.

Using data from the Rochester cohort, Doran et al8 showed in 2002 that, among others, RA severity, functional status and co-morbidity were predictors of serious infection. This group recently developed a risk score for serious infection, based on 584 patients enrolled in the Rochester cohort between 1955 and 1994, and evaluated the score in 410 patients enrolled between 1995 and 2007.9 The contribution of specific risk factors such as age, previous infection, co-morbidity or GC dose was similar as in our data. They found an HR of 3.6 for more than 10 mg GCs/day while we found a fourfold increased risk for >15 mg/day compared to <7.5 mg/day. Each co-morbidity doubled the risk which is comparable to our data. A direct comparison of the two scores is not possible since the Rochester score does not include treatment with biological or non-biological DMARDs and our score does not include erythrocyte sedimentation rate and extra-articular manifestations of RA.

A second infection risk score was developed very recently by Curtis et al10 using two administrative databases in the USA. Predictors of serious infection were, as in our data, older age, co-morbid conditions, higher dosages of GCs and previous serious infection. Diabetes mellitus was also associated with a moderately increased risk in this study. We did not include diabetes into our model due to low predictive capacity. Diabetes as a co-morbid condition may play a greater role in the USA: in the Curtis study, its prevalence was >30% in the older age group and >10% in the younger one, whereas only 8% of our patients had diabetes. The risk score is also not fully comparable to ours since parameters of RA disease activity or severity were not available in the US study. On the other hand, the Curtis score includes a number of items such as cancer screening or treatments for other chronic diseases that are not available to us.

A strength of our risk score is that it includes time-varying parameters of disease activity and disability which results in more precise risk estimates on the individual level than treatment and sociodemographic factors only. The limitation is that a risk score is not generalisable to populations with substantially different risk distributions. Therefore, we consider our risk score valid for the Caucasian population with moderate to severe RA in Germany and probably also in Western Europe. Re-evaluation of the score in other RA populations would be useful to test its reliability beyond the RABBIT register.