Analysis of scoring systems for primary immunodeficiency diagnosis in adult immunology clinics.

Failure to spot the signs of primary immunodeficiency (PID) often results in delayed diagnosis. Scoring systems to identify PID exist, such as the immunodeficiency disease-related (IDR) score. This research aims to analyse and improve the diagnostic sensitivity and specificity of the IDR scoring system in a small preselected group of adult patients referred to immunology with clinical suspicion of a PID. Records of all patients presenting for the first time to an adult immunology clinic in 2018 at Addenbrooke's Hospital, Cambridge, were scored using the unmodified IDR score and modified versions of it. Included records were searched for a subsequent diagnosis of PID, and the diagnostic sensitivity and specificity of the scoring systems were analysed. Of 400 patients, 213 were excluded: 141 due to secondary immunodeficiency, 69 due to no clinical suspicion of a PID, and hence no investigation for PID, and three due to ongoing diagnostic investigations. Of 187 included patients, 71 were found to have a clinically significant PID. The unmodified IDR score was useful in discriminating between those with and without PID. Modification of the scoring system with seven additional criteria improved the sensitivity and specificity for PID diagnosis to the greatest extent. A modified IDR score with seven additional criteria validated in adults referred to immunology with suspicion of a PID could be used clinically to aid PID diagnosis, although further validation in different patient cohorts is required before it is used in other contexts.

Introduction

Primary immunodeficiencies (PIDs) can present clinically in a wide variety of ways, making diagnosis difficult. Patients with PIDs may live many years before a formal diagnosis is made [1]. This delay in diagnosis can result in patients experiencing ill health that would have otherwise been prevented had a diagnosis of PID been made earlier. Most patients referred to a general immunology clinic with clinical suspicion of a PID invariably have their immune function tested, as it can be difficult to rule out a PID on the basis of symptoms and past medical history alone.

Scoring systems and lists of PID warning signs have been developed in order to aid physicians in knowing when to suspect a PID and refer to immunology, thereby reducing the delay in diagnosis. These include the immunodeficiency disease‐related (IDR) score [2] (Table 2), developed to identify undiagnosed PID in minority groups in the United States, as well as those published by the European Society for Immunodeficiencies (ESID) [3], the Jeffrey Modell Foundation (JMF), the German Patients’ Organization for Primary Immunodeficiencies (DSAI), the Association of the Scientific Medical Societies in Germany (AWMF) and the Duesseldorf Warning Signs [4]. Most of these scoring systems include criteria tailored towards identifying PID in children, such as a score for failure to thrive, which would not apply in an adult context.

Table 2

The unmodified IDR scoring criteria [2]

Criteria	Score
Meningococcal meningitis	3
Sepsis, identified organism	3
Viral meningitis, identified organism	3
Pneumocystosis	3
Bacterial meningitis, identified organism	3
Viral pneumonia, identified organism	3
Pneumococcal pneumonia	3
Bacterial pneumonia, identified organism	3
Pneumonia, other	3
Bronchopneumonia	3
Influenza	3
Bronchiectasis	3
Empyema	3
Lung abscess	3
Rectal abscess	3
Liver abscess	3
Osteomyelitis	3
Giardiasis	2
Haemolytic anaemia	2
Thrombocytopenia	2
Neutropenia	2
Cellulitis	2
Lymphadenitis	2
Splenomegaly	2
Thrush / candidiasis	1
Other mycoses	1
Otitis media	1
Chronic otitis	1
Chronic mastoiditis	1
Acute sinusitis	1
Acute bronchitis	1
Chronic sinusitis	1
Chronic bronchitis	1
Non‐infectious gastroenteritis	1
Malabsorption	1
Fever, unknown origin	1
Loss of weight	1
Failure to thrive	1
Enlarged lymph glands	1
Diarrhoea	1

Chronic scores only counted once in a 60‐day period are shown in bold type. Scores with lower pulmonary tract involvement are shown in italic type.

Previous studies assessing the utility of these scoring systems in diagnosing PID often focus upon paediatric rather than adult populations [4, 5, 6, 7], although the IDR score has been validated in a mixed paediatric and adult cohort [8]. Furthermore, research into novel tools for the diagnosis of PID often focuses upon paediatric rather than adult populations [9, 10]. The aim of this research is to analyse the usefulness of the IDR score in diagnosing PID specifically in adults who have already been referred to immunology, and improve the scoring system by analysing the effect of modifications to it.

Receiver operating characteristic (ROC) curves can be constructed for a diagnostic scoring system, with the area under the ROC curve taken as a measure of the overall ability of a test to discriminate between those with and without a condition [11]. In selecting the optimal cut‐off score for a diagnostic test, the Youden index (sensitivity + specificity – 1) can be used be find the optimal trade‐off between sensitivity and specificity [11].

Materials and methods

Patient cohort and exclusion criteria

The cohort of patients analysed in this study consisted of all patients presenting for the first time to a general adult immunology clinic at Addenbrooke’s Hospital in Cambridge, during the calendar year of 2018. Patient records were excluded if there was no clinical suspicion of PID, due to the fact that these patients did not undergo subsequent tests for PID. This included if the patient was referred to the clinic primarily for urticaria, angioedema or other presentations unrelated to immunodeficiency. Patient records were also excluded if there was evidence of secondary immunodeficiency (Table 1), or if investigations into their immune system were ongoing and inconclusive at the time of writing.

Table 1

Criteria for exclusion due to secondary immunodeficiency, based on known causes of secondary immunodeficiency [14]

Category	Name	Details
Medications	Oral steroids	Exclude scoring while the patient is taking oral steroids if dose exceeds 20 mg/day prednisolone equivalent for more than 1 month, or 40 mg/day prednisolone equivalent for more than 1 week. For children under 20 kg, exclude if dose exceeds 1 mg/kg prednisolone equivalent
	Chemotherapy	Exclude scoring while the patient is undergoing chemotherapy
	Other immunosuppressants	Exclude scoring while the patient is taking immunosuppressant medication, including methotrexate, azathioprine, rituximab, infliximab, anti‐convulsants; only where the clinician believes immunodeficiency is secondary to the medication
Procedures	Surgery	Exclude scoring in the month following invasive surgery
	Splenectomy	Exclude patients who have undergone splenectomy
	Stem cell transplant	Exclude scoring for infections in the month following stem cell transplant
Infection	HIV	Exclude patients with a diagnosis of HIV
	Systemic EBV	Exclude scoring during a confirmed active systemic EBV infection
	Systemic CMV	Exclude scoring during a confirmed active systemic CMV infection
Metabolic	Diabetes mellitus	Exclude patients with any form of diabetes mellitus
	Uraemia	Exclude scoring during an episode of uraemia
Nutrition	Alcohol excess	Exclude patients with a history of drinking more than 35 units/week for women or more than 50 units/week for men
	Malnutrition	Exclude scoring during episodes of malnutrition
	Vitamin or mineral deficiency	Exclude scoring during episodes of untreated vitamin or mineral deficiencies
Protein‐losing conditions	Kidney disease	Exclude patients with kidney disease including AKI, CKD and nephrotic syndrome
	Protein‐losing enteropathy	Exclude scoring patients during episodes of gastrointestinal pathology resulting in protein loss
	Hepatic cirrhosis	Exclude patients with diagnosed alcoholic or non‐alcoholic liver cirrhosis
Other	Sickle cell disease	Exclude patients with sickle cell disease
	Cancer	Exclude patients with any form of malignancy where it is ongoing, or where the clinician believes immunodeficiency is related to the malignancy. Exclude scoring during past episodes of malignancy that have fully resolved
	Radiation	Exclude scoring during or following an episode of acute radiation exposure
	Pregnancy	Exclude scoring during pregnancy.
	Hepatic or pancreatic disease	Exclude scoring during episodes of acute hepatitis, pancreatitis or liver failure requiring active treatment
	Splenic dysfunction	Exclude patients with known splenic dysfunction
	Aplastic anaemia	Exclude patients with a past or current history of bone marrow failure
	Burns and ulcers	Exclude scoring infections during or following a breach of the skin

EBV = Epstein–Barr virus; CMV = cytomegalovirus; AKI = acute kidney injury; CKD = chronic kidney disease.

Systemic autoimmune diseases such as systemic lupus erythematosus are not listed in Table 1 as exclusion criteria due to increasing evidence that autoimmunity can present alongside PID [12, 13], although patients taking immunosuppressive medication were excluded. If one of the causes of secondary immunodeficiency listed in Table 1 was chronic, for example, a patient with type I diabetes, the patient was excluded entirely. If one of the causes of secondary immunodeficiency listed in Table 1 was an acute episode that fully resolved, for example, a patient on a short‐term course of immunosuppressive medication, then any scoring resulting from the episode, such as an infection during the course of immunosuppression, was discounted, but the patient records were not excluded entirely.

Scoring system

All included patient records were scored using the IDR scoring system in its original unmodified form (Table 2). As per the original IDR scoring system [2], time restrictions were applied such that each score can only be counted once in a 30‐day period, or once in a 60‐day period for ‘chronic codes’ (shown in bold type in Table 2) in order to avoid multiple counting of the same episode of illness. When it was clear from the records that a patient suffered from the same chronic problem over many years, such as chronic candidiasis involving multiple readmissions, the problem was only scored once. However, multiple unrelated episodes of the same diagnosis, such as three admissions for pneumonia each 3 months apart, were scored separately. Non‐specific comments in the records, such as ‘many episodes of pneumonia’, were only scored once. Loss of weight was only scored if it was unexplained.

Recent evidence suggests autoimmunity may be a presenting feature of PID [12, 13]. Any autoimmune presentation or immune system dysfunction not included in the scoring system (including atopic conditions) but noted in included patient records was recorded (Supporting information, Table S1).

All included patient records were also scored using the IDR scoring system with three different modifications. Modification 1 added seven additional criteria to the scoring system (Table 3). Modification 2 discounted all scores with lower pulmonary tract involvement (shown in italic type in Table 2) in patients with either a history of smoking or chronic lung disease, including asthma, chronic obstructive pulmonary disease (COPD) and interstitial lung disease. Modification 3 applied both modifications 1 and 2. Thus, each included patient record was given four separate scores, corresponding to the unmodified IDR score, and the IDR score in three modified formats.

Table 3

The seven additional criteria added to the IDR score in Fig. 2 according to modification 1

Criteria	Score	Comments
Need for intravenous (i.v.) antibiotics	3	Score not given if i.v. antibiotics were required for conditions listed in the exclusion criteria (Fig. 1), such as burns or following surgery
Infection with an atypical organism	3	Including all organisms that do not normally cause the presenting infection in clinical practice. For example, atypical organisms causing pneumonia would include Mycoplasma, Chlamydophila, Legionella and fungi [15] to name just a few
Abscess(es) of any organ	3	Including deep or recurrent abscesses of the skin, internal organs and other tissues
Attenuated vaccine response	3	Any abnormal immunological response, such as low functional immunoglobulins following vaccination, or any abnormal clinical response that requires medical attention, such as admission following vaccination
Hypogammaglobulinaemia	3	Only scored if previously noted in medical history, not from subsequent immunological investigations. Note that these criteria may also constitute a primary immunodeficiency in themselves, such as a previously noted hypogammaglobulinaemia in a patient subsequently found to have a clinically significant primary hypogammaglobulinaemia
Lymphopaenia	3
Family history of primary immunodeficiency	3	Only scored if a blood relative has been diagnosed with primary immunodeficiency

Fig. 1

Flow diagram to illustrate the reasons for exclusion of patients from this cohort, and the results of testing for primary immunodeficiencies in included patients.

The seven additional criteria of modification 1 (Table 3) are included in other lists of warning signs for PID [3, 4], including those published by the ESID, the JMF, the DSAI and the AWMF. Episodes of intravenous (i.v.) antibiotic use or infections with atypical organisms were scored multiple times, but only once within a 30‐day period. All other criteria in Table 3 were only ever scored once per patient.

Modification 3 applied both the seven additional criteria in Table 3 while also excluding scores with lower pulmonary involvement in patients with chronic lung disease or a smoking history. In such a patient where the additional criteria in Table 3 involve the lower respiratory system, such as i.v. antibiotics for pneumonia or a pneumonia with an atypical organism, scores for the additional criteria were not given.

Each patient record was scored using the notes from their attendance at their first immunology clinic appointment and any post‐appointment letters recorded on the epic computer system in use at Addenbrooke’s Hospital. Patients were scored based on the entirety of their past medical history until the point of first presentation to the immunology clinic.

The follow‐up immunology clinic notes, letters and blood test results for each patient record were then analysed, and each patient was either deemed to have been diagnosed with a clinically significant PID or not. For the purposes of this study, a ‘clinically significant’ PID is defined as any overt or subtle primary defect of the immune system, for which clinical attention is required, ranging from increased surveillance for infection to immunoglobulin replacement. Patients requiring clinical intervention for identified immune deficiencies without a definitive diagnosis of any one PID in particular were still counted as having a ‘clinical significant’ PID. Patients with an identified mild primary immune system defect, such as immunoglobulin (Ig)A deficiency, were not considered to have a clinically significant PID if they required no clinical attention and were discharged from care following all investigations.

This research does not involve experimentation on human or animal subjects, and therefore no ethical approval was required.

Data analysis

GraphPad Prism version 8 was used to construct grouped frequency graphs of scores for those with and without clinically significant PIDs, and to construct ROC curves, for the diagnosis of PID using the four different versions of the IDR scoring system. The area under each ROC curve was calculated, and the sensitivity, specificity, positive predictive value and negative predictive value of every cut‐off score for each of the four versions of the IDR scoring system were calculated. The Youden index (sensitivity + specificity – 1) was also calculated for every cut‐off score for each of the four versions of the scoring system.

Results

Of a total of 400 patients presenting for the first time to the immunology clinic, 252 (63%) were female and 148 (37%) were male. Excluding a single 8‐year‐old (who was excluded from the study due to lack of clinical suspicion of a PID), the ages of the remaining 399 patients ranged from 16 to 90 years, with a mean age of 50·4. Sixty‐nine were excluded due to lack of clinical suspicion of a PID (and hence, lack of testing for PID) and 141 were excluded due to secondary immunodeficiency. Three were excluded due to ongoing diagnostic investigations into their immune system at the time of writing. This left a small cohort of 187 patients, 71 of whom were subsequently found to have a clinically significant PID, and 116 were not. In this smaller cohort of 187 included patients, 122 (65·2%) were female and 65 (34·8%) were male, with ages ranging from 16 to 84 years and a mean age of 47·4 years. Of those found to have a clinically significant PID, seven were found to have common variable immunodeficiency (CVID), 54 were found to have other hypogammaglobulinaemias, two were found to have isolated lymphopaenias, three required clinical intervention due to poor antibody responses to vaccines, three had PIDs associated with PIK3CD gene mutations, one had properdin deficiency and one had PID resulting from serum autoantibodies to granulocyte–macrophage colony‐stimulating factor.

Common reasons for exclusion due to lack of testing for a PID included presentation with suspected angioedema or chronic urticaria. Common reasons for exclusion due to suspected secondary immunodeficiency included type II diabetes, ongoing malignancies and immunosuppression due to medication.

Supporting information, Table S1 summarizes immune system dysfunctions, including some autoimmune and atopic conditions, that are not included in any of the scoring systems but were noted in the patient records that were included.

Grouped frequency graphs and ROC curves

Grouped frequency graphs of scores given to those with and without a clinically significant PID, for each of the four versions of the IDR scoring system, are shown in Fig. 2. ROC curves for the four different versions of the IDR scoring system are shown in Fig. 3. A summary table to show the area under the curve (AUC), standard error, 95% confidence interval and P‐value (testing the null hypothesis that the AUC equals 0·5) for each ROC curve is shown in Table 4.

Fig. 2

Grouped frequency graphs of scores given to those with and without a clinically significant primary immunodeficiency (PID), for each of the four versions of the immunodeficiency disease‐related (IDR) scoring system.

Fig. 3

Receiver operating characteristic curves for the four different versions of the immunodeficiency disease‐related (IDR) scoring system.

Table 4

Table showing the AUC for each ROC curve, alongside the standard error and 95% CI for this calculation

	AUC	Standard error	95% CI	P‐value
Unmodified IDR score	0·7142	0·03968	0·6364–0·7920	< 0·0001
IDR score with Modification 1	0·8534	0·02722	0·8001–0·9068	< 0·0001
IDR score with Modification 2	0·5729	0·04550	0·4837–0·6620	0·0949
IDR score with Modification 3	0·8110	0·03199	0·7483–0·8737	< 0·0001

AUC = area under the curve; ROC = receiver operating characteristic; CI = confidence interval.

The P‐value tests the null hypothesis that the AUC equals 0·5.

The area under the ROC curve for the unmodified IDR scoring system was greater than 0·5, a statistically significant result. The area under the ROC curve was increased to the greatest extent when seven additional criteria were added to the scoring system (modification 1). By contrast, discounting lower respiratory scores in those with chronic lung disease or a smoking history (modification 2) reduced the area under the ROC curve. Combining both modifications (modification 3) resulted in an increased area under the ROC curve, and hence increased diagnostic discriminatory ability compared to the unmodified IDR scoring system, but to a lesser extent than applying modification 1 alone.

Efficacy of cut‐off scores for PID diagnosis

Supporting information, Table S2 shows the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and Youden index of cut‐off scores for each of the four versions of the IDR scoring system in the diagnosis of PID. A Youden index > 0·5 was obtained in four instances: when the IDR score with modification 1 was used with cut‐off scores of > 4·5, > 5·5 and > 6·5, and when the IDR score with modification 3 was used with a cut‐off score of > 2·5.

The maximum Youden index occurred when the IDR score with the seven additional criteria (modification 1) was applied with a cut‐off score of > 6·5 for the diagnosis of PID. This scoring resulted in a sensitivity of 0·69, a specificity of 0·88, a PPV of 0·78 and a NPV 0·82 for the diagnosis of a clinically significant PID, as shown in Supporting information, Table S2.

Discussion

These results validate the use of the original unmodified IDR scoring system as a tool for PID diagnosis at first presentation to an adult immunology clinic in the United Kingdom. The greatest increase in the area under the ROC curve obtained when seven additional criteria were added to the scoring system (modification 1) suggests that this new modified scoring system has the greatest diagnostic discriminatory ability in our patient cohort of all the versions of the IDR scoring system tested. The Youden index calculations for the scoring system with the seven additional criteria (modification 1) suggest that a cut‐off score of > 6·5 would optimize its diagnostic sensitivity and specificity. Overall, the seven additional criteria (modification 1) represent an improvement to the IDR scoring system for the diagnosis of clinically significant PID in this small patient cohort.

Although the original unmodified IDR score successfully identified undiagnosed PID in a more generalized cohort of hospitalized patients [2], it is important to note that this research does not validate use of the modified scoring system in any context other than for patients presenting for the first time to an adult immunology clinic. In other patient cohorts the sensitivity and specificity of the scoring system for PID diagnosis may change considerably. However, as our clinic is a typical UK immunology clinic, we believe our findings will be generalizable, at least in the current National Health Service environment. Further research is needed to establish whether the IDR scoring system with the seven additional criteria (modification 1) improves the discriminatory ability of the score in diagnosing PID in other patient cohorts. In particular, validation of the scoring system in patients presenting with features suggestive of an immune defect to general practice, prior to referral to immunology, would enable use of the modified IDR score by general practitioners when deciding whether or not to refer a patient to immunology. Use of the scoring system in this context may present new issues, due to differences between the general population and the highly preselected group of patients used in this research. In particular, patients in the general population will have a much lower probability of a PID diagnosis compared to those who have already been referred to immunology. A much larger number of patient records may, therefore, need to be analysed in order to validate the scoring system. Computer‐based scoring of patient records would make this process significantly less labour‐intensive. The frequency of presentations included in the scoring system but not due to PID in the general population, which has not been explored in this research, would also affect the specificity of the scoring system in this context.

Discounting lower respiratory scores in those with chronic lung disease or a smoking history (modification 2) reduced the diagnostic discriminatory ability of the scoring system. Further investigation is required before concluding that those with both chronic lung disease or a smoking history and PID suffer greater lower respiratory tract illness than those with chronic lung disease or a smoking history alone.

This research is a retrospective validation of the scoring systems involving past records, in which all patients who were not tested for a PID had to be excluded. A prospective approach, whereby patients are scored in the clinic at first presentation and then all tested for PID, would further validate the scoring system. Other limitations to this research include the fact that a large number of patient records were excluded, leaving a relatively small patient cohort of only 187 included patients, all of whom presented over a short time‐period (the calendar year 2018) to immunology. A larger computer‐aided scoring and analysis of a greater number of patient records over a longer time‐period would further validate the scoring system. Additionally, past medical records were used to score patients, which may be incomplete or written by different clinicians who may include more or less of the patients’ past medical history in their written notes, thereby introducing some subjectivity into the retrospective scoring system.

Conclusion

The records of 400 patients presenting for the first time to an adult immunology clinic at Addenbrooke’s Hospital in 2018 were analysed, and the 187 records included in the study were scored to test the efficacy of four different scoring systems in the diagnosis of a clinically significant PID. While the unmodified IDR score was found to be a useful diagnostic tool, a modified version of the IDR score with seven additional criteria improved its diagnostic sensitivity and specificity. The optimal trade‐off between sensitivity and specificity was obtained when the modified IDR score with the seven additional criteria was used with a cut‐off score of > 6·5 for the diagnosis of a clinically significant PID. This modified IDR scoring system may be used clinically to aid PID diagnosis in adults who are referred for the first time to immunology with suspicion of a PID, although further prospective validation in other larger patient cohorts is required before it can be used in any other contexts, including prior to immunology referral.

Disclosures

The authors declare no conflicts of interest or commercial associations.

Author contributions

K. T. carried out the data collection and analysis and wrote the manuscript. E. G. K. and D. K. initiated and supervised the research, and reviewed the manuscript.

Table S1. Table summarising immune system dysfunction, from most to least frequent, including autoimmunity, atopic conditions, significant allergies and intolerances, that were noted in the records of included patients but are not included in the IDR scoring system or its modifications.

Table S2. Table showing the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and Youden Index of cut‐off scores for the four versions of the IDR scoring system, in the diagnosis of PID. Only cut‐off scores where these values change numerically have been shown. The four instances resulting in a Youden Index >0.5 are in bold.

Click here for additional data file.