Development and validation of the food allergy severity score.

BACKGROUND: The heterogeneity and lack of validation of existing severity scores for food allergic reactions limit standardization of case management and research advances. We aimed to develop and validate a severity score for food allergic reactions.
METHODS: Following a multidisciplinary experts consensus, it was decided to develop a food allergy severity score (FASS) with ordinal (oFASS) and numerical (nFASS) formats. oFASS with 3 and 5 grades were generated through expert consensus, and nFASS by mathematical modeling. Evaluation was performed in the EuroPrevall outpatient clinic cohort (8232 food reactions) by logistic regression with request of emergency care and medications used as outcomes. Discrimination, classification, and calibration were calculated. Bootstrapping internal validation was followed by external validation (logistic regression) in 5 cohorts (3622 food reactions). Correlation of nFASS with the severity classification done by expert allergy clinicians by Best-Worst Scaling of 32 food reactions was calculated.
RESULTS: oFASS and nFASS map consistently, with nFASS having greater granularity. With the outcomes emergency care, adrenaline and critical medical treatment, oFASS and nFASS had a good discrimination (receiver operating characteristic area under the curve [ROC-AUC]>0.80), classification (sensitivity 0.87-0.92, specificity 0.73-0.78), and calibration. Bootstrapping over ROC-AUC showed negligible biases (1.0 × 10-6 -1.23 × 10-3 ). In external validation, nFASS performed best with higher ROC-AUC. nFASS was strongly correlated (R 0.89) to best-worst scoring of 334 expert clinicians.
CONCLUSION: FASS is a validated and reliable method to measure severity of food allergic reactions. The ordinal and numerical versions that map onto each other are suitable for use by different stakeholders in different settings.

95% confidence interval

area under the curve

bronchial

British Society for Allergy & Clinical Immunology

Best‐Worst Scaling

critical medical treatment

cardiovascular

European Academy of Allergy and Clinical Immunology

food allergy

Food Allergy Severity Score

Hospital Clinico San Carlos

Integrated Approaches to Food Allergen and Allergy Risk Management

intravenous fluids

laryngeal

Network for Online Registration of Anaphylaxis

negative predictive value

nervous system

oral allergy symptoms

ordinal FASS

ordinal FASS with 3 grades

ordinal FASS with 5 grades

odds ratio

positive predictive value

first quartile

third quartile

receiver operating characteristic

Plant food allergies, field to table strategies for reducing their incidence in Europe

sensitivity

standard error

Spanish Society of Allergy and Clinical Immunology

specificity

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis

World Allergy Organization

INTRODUCTION

Food allergy (FA) has become a significant medical problem in the last decades. , Food allergic reactions can affect different organs and systems, and present with a wide range of severity from mild transient oral symptoms, to severe and even fatal anaphylaxis. In fact, FA is one of the main causes of anaphylaxis in the community. , , , , The severity of reactions varies between individuals, and within repeated reactions in the same individual. Severe reactions are not predictable, and allergists have limitations to accurately identify the patients at greatest risk of life‐threatening reactions.

The treatment of acute allergic reactions is driven by severity. The longer‐term management is also guided by the severity of the previous reaction(s), and/or by the presumed risk of having a future severe reaction. With increasing risk (or risk perception) more rigorous avoidance of their problem food is needed, and patients should carry rescue medication including an adrenaline autoinjector. The uncertainty and fear of the severity and outcome of any future reaction have a profound negative impact on patients' health‐related quality of life. Current FA risk management strategies are focused on clinical, regulatory and industrial collaborations to develop reference doses that will or not trigger an allergic reaction and to use them to guide precautionary allergen labeling, but the severity of the reaction is not considered in the process. , Similarly, novel interventions in FA such as allergen immunotherapy or anti‐IgE therapy aim to reduce the risk of accidental reactions by increasing the amount of allergen tolerated, but the severity of the reactions is not included as a primary outcome. , , ,

Severity is therefore a key parameter in FA that needs to be measured as accurately as possible. Several scoring systems have been proposed by different groups to grade severity of anaphylaxis, , , , , , , , and of allergic reactions induced by foods, , , , , , , , , drugs, , , insect venoms, , , , or allergen immunotherapy. , , , All the instruments are organ based, most of them generated using expert opinion with only one using a Delphi methodology. The great majority classifies severity using ordinal scales (3 to 6 grades) that are not equivalent making comparisons difficult. , , Furthermore, none of the current systems has been validated (ie, the performance has never been assessed).

The European Union‐funded project Integrated Approaches to Food Allergen and Allergy Risk Management (iFAAM) in collaboration with a task force of the European Academy of Allergy and Clinical Immunology (EAACI) critically reviewed the available systems and proposed in an EAACI Position Paper an approach to develop a system to measure severity of allergic reactions. Here, we report the development and validation of the Food Allergy Severity Score (FASS), aligned with the EAACI Position Paper. The report follows the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

METHODS

Development of the Food Allergy Severity Score (FASS)

Premises that FASS must meet

In line with the iFAAM‐EAACI expert consensus FASS had to meet the following premises:

Have an anatomical basis: as the number of organs/systems involved increase, severity increases.

Consider laryngeal (Lx), bronchial (BR), cardiovascular (CV), and nervous system (NS) involvement as potentially life‐threatening, even if isolated, and thus more severe than that of other organs/systems.

Use variables (symptoms/signs) that are easily/routinely recorded.

Be applicable to all patient populations (all ages, all foods).

Be used in different countries.

Have ordinal (oFASS) and numerical (nFASS) formats that map consistently, to facilitate use by different stakeholders in different scenarios.

The development of FASS needs to identify the organs/systems affected and the specific symptoms and signs associated with their involvement in an allergic reaction, since these are the variables used to build the score. The identification of organs and systems based on anatomy does not pose a problem. However, there are multiple ways to describe symptoms and signs of allergic reactions and a common international terminology is lacking. To overcome this, the symptoms as described in the PRACTALL consensus  have been taken as reference.

Ordinal Food Allergy Severity Score (oFASS)

The oFASS was built by experts' consensus of different stakeholders (expert allergy clinicians, epidemiologists, basic scientists, representatives of food allergic patients) after several rounds of discussion at iFAAM project meetings. It has two versions of five (oFASS‐5) and three categories (oFASS‐3) (Table 1). We use the term oFASS to refer to both oFASS‐5 and oFASS‐3. In oFASS‐5, grade 1 includes reactions restricted to the oral cavity. Grades 2 to 5 may include oral symptoms, but other target organs are affected. Grading is based on the organ/system involved regardless of the type or number of specific symptoms present of that organ/system. Grades 2 and 3 include skin, eye/nose, digestive, and/or uterine involvement, either 1 or more than 1 of them, respectively. Lx and/or BR involvement (even isolated) classifies a reaction as Grade 4, and CV and/or NS involvement (even isolated) as grade 5. In grades 4 and 5, other target organ/systems of lower grades may be affected. oFASS‐3 is a simplified version, where mild corresponds to grade 1, moderate to grades 2 and 3, and severe to grades 4 and 5 of the oFASS‐5.

TABLE 1

Ordinal Food Allergy Severity Score (oFASS) versions: oFASS‐3 and oFASS‐5

oFASS−3	oFASS−5	Organs/Systems involved
		Oral cavity	Skin Nose/Eye Digestive Uterus	Larynx Bronchi	Cardiovascular Nervous system
Mild	Grade 1	Yes	No	No	No
Moderate	Grade 2	Yes/No	1	No	No
	Grade 3	Yes/No	>1	No	No
Severe	Grade 4	Yes/No	Yes/No	1 or both	No
	Grade 5	Yes/No	Yes/No	Yes/No	1 or both

Abbreviations: no, not involved; yes, involved; yes/no, it can be involved or not.

Numerical Food Allergy Severity Score (nFASS)

The nFASS model was constructed from scratch and had to satisfy several conditions: (i) the nFASS must consistently map onto the oFASS (ie, higher oFASS implies higher nFASS, and a given nFASS value can only correspond to one level of oFASS); (ii) a higher number of symptoms within the same organ/system should increase the score.

The nFASS is computed in 3 steps as shown in the example of Table 2. Each organ/system has an assigned exponent, ε o, (−1 for oral; 0 for GI, eye/nose, skin; 2 for BR, LX; 4 for CV/NS) and each symptom has an equivalence with the PRACTALL reference and an assigned weight, λ (Table S1). The small individual weights of the symptoms (λ < 0.1), combined with the organ/system exponents (from −1 to 4) that multiply the expression, guarantee correct mapping from the nFASS to the oFASS. Further details are in Appendix S2 and Table S2.

TABLE 2

Computing nFASS

Food reaction: boy 5 years, 15 min after eating peanut presents urticaria, red eyes, nausea, wheeze, and dizziness
Step 1					Step 2
Identify organ/system involved and the symptom PRACTALL equivalence to select the corresponding organ exponent and symptom weight					Compute one organ/system contribution (nFASSo)
Symptom in the reaction	Organ System	Exponent ε	Symptom equivalence PRACTALL	Weight λ	nFASS_o=2^ε_01+λ_1+⋯+λ_n
Urticaria	Skin	ε_O=0	Generalized involvement (>10)	λ_1=0.08	nFASS_Skin=2^ε_01+λ_1=2^01+0.08= 1.08
Red eyes	Eye	ε_1=0	Intermittent rubbing of eyes	λ_1=0.05	nFASS_Eye=2^ε_11+λ_1=2^01+0.05=1.05
Nausea	GI	ε_2=0	Complaints of nausea OR abdominal pain	λ_1=0.03	nFASS_GI=2^ε_21+λ_1=2^01+0.03= 1.03
Wheeze	Lower respiratory	ε_3=2	Wheezing: Inspiratory and expiratory wheezing to auscultation	λ_1=0.07	nFASS_LoResp=2^ε_31+λ_1=2^21+0.07= 4.28
Dizzy	NS	ε_4=4	Weak, dizzy	λ_1=0.05	nFASS_NS=2^ε_41+λ_1=2^41+0.05=16.8
Step 3
Sum values of all the organs/systems affected and apply transformation to reduce the scale
nFASS=log_2∑_oϵOrgansnFASS_o+2
nFASS=log_2(nFASS_Skin+nFASS_Eye+nFASS_GI+nFASS_LoResp+nFASS_NS)+2=log_2(1.08+1.05+1.03+4.28+16.8)+2=4.59+2=6.59

Abbreviations: GI, gastrointestinal; LoResp, lower respiratory; NS, nervous system.

Evaluation cohort

The evaluation was done in the EuroPrevall outpatient clinic dataset that comprises 2112 patients and 8232 immediate food allergic reactions (characteristics in Tables S3 and S4, and the equivalence of symptoms with the PRACTALL reference in Table S5). Patients reporting reactions to foods were selected in 12 allergy clinics across Europe and evaluated following the same protocol previously described. There was no exclusion based on age, type of food, or severity of reaction.

External validation cohorts

The external validation cohorts covered infants, children, and adults (n = 2930) selected across Europe, with immediate allergic reactions (n = 3622) to any type of food and of any severity (Table S3). These cohorts included patients from the EuroPrevall general population survey phase 3 (147 subjects, 635 food reactions), , patients recruited in the iFAAM project (iFAAM cohort, 356 patients, 453 reactions) , ; food‐induced anaphylaxis from the Network for Online Registration of Anaphylaxis (NORA) (1959 subjects, 2020 anaphylaxis) , ; infants and toddlers with egg and milk allergies recruited at Hospital Clinico San Carlos (Madrid, Spain) (HCSC cohort, 83 children, 129 food reactions); and apple allergic patients recruited in the EU‐funded SAFE project (385 subjects, 385 apple reactions). The equivalence of symptoms with the PRACTALL reference is presented in Table S5, and further information on the cohorts in Appendix S2.

Predictors and outcomes

The predictors are the 3 versions of FASS: oFASS‐3, oFASS‐5, and nFASS. In order to assess the ability of FASS (the acronym FASS comprises all three versions) to reflect reaction severity, we selected in the evaluation cohort outcomes (or severity indicators) that reflected management decisions taken by patients, carers and/or health professionals in the routine assistance of the 8232 food reactions included in the cohort. We assumed that a patient who requested emergency care assistance for a food reaction had a perception of higher severity than a patient who did not. Similarly, we assumed that a reaction that was treated with medication/s was considered more severe than a reaction not treated at all. We therefore selected from the information collected in the record forms of the validation cohort the following outcomes: request of emergency care, use of any medication, antihistamines, corticosteroids, and adrenaline. Any medication includes all the drugs used to treat a reaction. An outcome named critical medical treatment (CMT) was built which included use of intravenous fluids (IVF), vasopressors, oxygen, and/or mechanical ventilation. In the external validation cohorts, we used the outcomes for which FASS exhibited the best discrimination, classification, and calibration in the evaluation cohort. Outcomes different from the ones used in the evaluation phase could not be included in the external validation.

Missing data

We excluded food reactions with missing information on the outcomes in the evaluation and validation cohorts and no imputation was performed (ie, complete case analysis). Information on missing values is presented in Tables S6 and S7.

Survey of allergy healthcare professionals to appraise the performance of FASS

Separately, we undertook a global survey of allergy healthcare professionals to rate the severity of different food‐induced allergic reactions, using “Best‐Worst Scaling” (BWS) which avoids user scale bias. In brief, an online “MaxDiff” survey was developed in which respondents were asked to rate the severity of different pairs of allergic reaction scenarios and choose the pair that, in their opinion, reflected the maximum difference in severity. The pairs of allergic reaction scenarios were selected from a total of 32 vignettes providing a wide spectrum of reaction severity (Table S8). The survey was administered by an independent market Research company (ResearchNow, UK), with potential respondents contacted through the EAACI, Spanish Society of Allergy and Clinical Immunology (SEAIC), British Society for Allergy & Clinical Immunology (BSACI), and World Allergy Organization (WAO). Responses were voluntary, anonymous, and confidential (detailed information in Stafford et al ).

Statistical analysis

Descriptive statistics included frequency and percent for qualitative variables, and minimum, maximum, median, first, and third quartiles (Q1, Q3) for numerical variables. The Cochran‐Armitage test for trend was used to analyze the frequency of outcomes across the oFASS levels.

In the evaluation of FASS, logistic regression models were generated for all outcomes. Models are presented with reference value, odds ratios (ORs) with their 95% confidence intervals (95% CI), and p‐value for the Wald's test. FASS performance was assessed by examining discrimination, classification, and calibration. Discrimination was quantified by calculating the area under the curve (AUC) of the receiver operating characteristic (ROC) curve and its 95%CI. Classification measures included sensitivity (Se), specificity (Sp), positive and negative predictive values (PPV, NPV). For the calibration, predicted and observed (real) probabilities of the outcomes were calculated. The Hosmer‐Lemeshow test for agreement was not applied because it is sensitive to grouping (thus, non‐applicable to oFASS) and sample size (large N in evaluation cohort).

Internal validation was performed by bootstrap with 100 replicates applied over ROC‐AUC. Bias and standard errors (SE) were estimated. External validation was done analyzing the predictive performance of FASS models in the external validation cohorts by logistic regression. ROC‐AUC and 95%CI were calculated.

The responses to the MaxDiff survey were modeled and a “preference” score (representing severity) determined for each scenario. These scores were compared to nFASS using Spearman's R correlation.

All statistical analyses were performed using R 4.0.3 and Python 3.8.5. Significant level was set at p = .05.

Ethical considerations

The Institutional Review Board of HCSC (Madrid, Spain) confirmed that ethical approval was not required to the development and validation of FASS. The online survey of allergy healthcare professionals did not require ethical approval, but was approved by EAACI, SEAIC, BSACI, and WAO. ResearchNow follows the UK Market Research Society's Code of Conduct.

RESULTS

Evaluation of FASS

The FASS scores were implemented in the 8232 food reactions of the EuroPrevall outpatient clinic dataset (Tables S3 and S4). The frequency of the different symptoms is presented in Figure S1, and the severity distribution in Table 3. According to oFASS‐3, 35.8% of reactions were mild, 34.5% moderate, and 29.7% severe. The majority (78.8%) of moderate reactions were of Grade 2 (one organ/system affected) accounting for 27.2% of total reactions. The most frequent severe reactions (81.1%) were of Grade 4 (Lx/BR) and represented 24.1% of total reactions. The nFASS values ranged from 1.07 to 7.75. All mild‐grade 1 reactions had an nFASS value of 1.07 because they were all collected under one category (oropharyngeal symptoms) in the EuroPrevall record forms. The nFASS values of moderate and severe reactions ranged from 2.01 to 3.98 and from 4.07 to 7.75, respectively. The correct mapping of nFASS onto oFASS (Table 3) allows to understand the meaning of a given numerical score.

TABLE 3

Severity of food reactions in the evaluation and external validation cohorts

Cohort	oFASS‐3		oFASS‐5		nFASS
	Level	N (%)	Level	N (%)	Min	Max	Median (Q1, Q3)
Evaluation cohort
EuroPrevall Outpatient clinic cohort N = 8232	Mild	2946 (35.8%)	Grade 1	2946 (35.8%)	1.07	1.07	1.07 (1.07,1.07)
	Moderate	2839 (34.5%)	Grade 2	2236 (27.2%)	2.01	2.80	2.23 (2.11, 2.66)
			Grade 3	603 (7.3%)	3.03	3.98	3.38 (3.12, 3.44)
	Severe	2447 (29.7%)	Grade 4	1984 (24.1%)	4.07	5.68	4.58 (4.39, 5.01)
			Grade 5	463 (5.6%)	6.07	7.75	6.78 (6.46, 7.26)
External validation cohorts
NORA N = 2020	Mild	0	Grade 1	0	–	–	–
	Moderate	210 (10.4%)	Grade 2	68 (3.36%)	2.01	2.27	2.12 (2.09,2.2)
			Grade 3	142 (7.02%)	3.04	3.74	3.12 (3.09,3.16)
	Severe	1810 (89.6%)	Grade 4	905 (44.8%)	4.08	5.68	4.69 (4.44,5.27)
			Grade 5	905 (44.8%)	6.11	7.64	6.57 (6.47,7.15)
HCSC Infant Cohort N = 129	Mild	15 (11.6%)	Grade 1	15 (11.6%)	1.07	1.07	1.07 (1.07,1.07)
	Moderate	101 (78.3%)	Grade 2	66 (51.2%)	2.07	2.74	2.65 (2.11,2.68)
			Grade 3	35 (27.1%)	3.07	3.44	3.12 (3.09,3.4)
	Severe	13 (10.1%)	Grade 4	10 (7.75%)	4.09	4.89	4.48 (4.42,4.64)
			Grade 5	3 (2.32%)	6.2	7.35	6.78 (6.49,7.06)
EuroPrevall general population N = 635	Mild	291 (45.8%)	Grade 1	291 (45.8%)	1.07	1.07	1.07 (1.07,1.07)
	Moderate	196 (30.9%)	Grade 2	170 (26.8%)	2.01	2.74	2.65 (2.18, 2.67)
			Grade 3	26 (4.1%)	3.03	3.43	3.37 (3.37,3.41)
	Severe	148 (23.3%)	Grade 4	138 (21.7%)	4.07	5.48	4.53 (4.24,5.17)
			Grade 5	10 (1.57%)	6.11	6.8	6.75 (6.3,6.76)
SAFE N = 385	Mild	230 (59.7%)	Grade 1	230(59.7%)	1.07	1.07	1.07 (1.07,1.07)
	Moderate	120 (31.2%)	Grade 2	103 (26.8%)	2.07	2.68	2.65 (2.65,2.65)
			Grade 3	17 (4.41%)	3.09	3.4	3.39 (3.39,3.39)
	Severe	35 (9.09%)	Grade 4	34 (8.83%)	4.26	4.79	4.54 (4.26,4.54)
			Grade 5	1 (0.2%)	6.46	6.46	6.46 (6.46, 6.46)
iFAAM N = 453	Mild	63 (13.9%)	Grade 1	63 (13.9%)	1.07	1.07	1.07 (1.07,1.07)
	Moderate	210 (46.4%)	Grade 2	159 (35.1%)	2.01	2.75	2.13 (2.11,2.59)
			Grade 3	51 (11.3%)	3.04	3.96	3.13 (3.09,3.38)
	Severe	180 (39.7%)	Grade 4	111 (24.5%)	4.07	5.63	4.52 (4.23, 4.69)
			Grade 5	69 (15.2%)	6.07	7.55	6.56 (6.39,6.81)

Abbreviations: Max, maximum; Min, minimum; Q1, first quartile; Q3, third quartile.

Logistic regression models were computed for oFASS‐3, oFASS‐5, and nFASS with any medication, adrenaline, corticosteroids, antihistamine, CMT, and request of emergency care as outcomes. The frequency of the severity indicators increased progressively (Cochran‐Armitage test for trend, p < .01) as the severity of reactions increased with oFASS‐3 and oFASS‐5 (Table S9). A positive association with all the severity indicators was found for the 3 versions of FASS, implying that a higher grade of oFASS‐3 and oFASS‐5, and an increment in the nFASS value (OR given per 1 point increment) was associated with higher probability of use of medications and request of emergency care to treat the reaction (Table 4). As presented in Table 4 and Figure 1A, all models had ROC‐AUC>0.70 which is a requirement in model development. There is a trend for a progressive increase of the ROC‐AUC from oFASS‐3 to oFASS‐5 and nFASS. Furthermore, the ROC‐AUC of the models with adrenaline (0.83–0.90), CMT (0.83–0.91), and emergency care (0.78–0.84) tended to be higher than those of either any medication (0.73–0.76), corticosteroids (0.76–0.80) or antihistamine (0.71–0.73).

TABLE 4

Model evaluation in the EuroPrevall outpatient clinic cohort: discrimination of logistic regression models with oFASS‐3, oFASS‐5, and nFASS

Severity indicator (outcome)	oFASS‐3			oFASS‐5			nFASS*
	Level	OR (95% CI)	ROC‐AUC (95% CI)	Level	OR (95% CI)	ROC‐AUC (95% CI)	OR (95% CI)	ROC‐AUC (95% CI)
Any medication N = 3309 (40.5%)#	Moderate:Mild	6.7 (5.9, 7.6)	0.73 (0.72,0.75)	Grade 2: Grade 1	5.8 (5.1, 6.6)	0.75 (0.74,0.76)	1.7 (1.7, 1.8)	0.76 (0.75, 0.77)
				Grade 3: Grade 1	11.3 (9.3, 13.8)
	Severe:Mild	11.5 (10, 13.2)		Grade 4: Grade 1	9.6 (8.4, 11.1)
				Grade 5: Grade 1	29.6 (22.8, 38.4)
Adrenaline N = 278 (3.37%)#	Moderate: Mild	2.4 (1, 5.8)	0.83 (0.80,0.85)	Grade 2: Grade 1	2.3 (0.9, 5.8)	0.88 (0.85,0.90)	2.8 (2.6, 3.1)	0.9 (0.88, 0.93)
				Grade 3: Grade 1	2.8 (0.8, 9.7)
	Severe:Mild	49.2 (23.2, 104.5)		Grade 4: Grade 1	25.7 (12, 55.3)
				Grade 5: Grade 1	188.9 (87.7, 407.1)
Corticosteroids N = 993 (12.2%)#	Moderate:Mild	10.6 (7.4, 15.1)	0.76 (0.74,0.78)	Grade 2: Grade 1	9.4 (6.5, 13.5)	0.79 (0.78,0.80)	1.9 (1.8, 2)	0.8 (0.78, 0.81)
				Grade 3: Grade 1	15.4 (10.2, 23)
	Severe:Mild	31.2 (22, 44.3)		Grade 4: Grade 1	21.8 (15.3, 31.1)
				Grade 5: Grade 1	103.6 (70.4, 152.3)
Antihistamine N = 3012 (36.9%)#	Moderate:Mild	6.3 (5.5, 7.2)	0.71 (0.70,0.72)	Grade 2: Grade 1	5.4 (4.7, 6.3)	0.73 (0.72,0.74)	1.6 (1.6, 1.7)	0.73 (0.72, 0.75)
				Grade 3: Grade 1	10.5 (8.6, 12.8)
	Severe:Mild	9.0 (7.9, 10.3)		Grade 4: Grade 1	7.5 (6.5, 8.7)
				Grade 5: Grade 1	21.7 (17.1, 27.6)
CMT N = 331 (4.05%)#	Moderate:Mild	39.2 (5.4, 285.9)	0.83 (0.81,0.85)	Grade 2: Grade 1	26.8 (3.6, 199.6)	0.89 (0.87,0.91)	2.9 (2.6, 3.1)	0.91 (0.9, 0.93)
				Grade 3: Grade 1	86.4 (11.5, 650.2)
	Severe: Mild	403.9 (56.7,2878.3)		Grade 4: Grade 1	190.5 (26.6,1364.2)
				Grade 5: Grade 1	1811 (252.8, 12979)
Emergency Care (request of) N = 928 (11.3%)#	Moderate:Mild	21.9 (12.7, 37.5)	0.78 (0.76,0.80)	Grade 2: Grade 1	17.6 (10.2, 30.4)	0.81 (0.80,0.83)	2.8 (2.6, 3.1)	0.84 (0.82, 0.85)
				Grade 3: Grade 1	39.2 (22.2, 69.3)
	Severe:Mild	75.4 (44.3, 128.4)		Grade 4: Grade 1	49.3 (28.8, 84.3)
				Grade 5: Grade 1	292 (167.3, 509.6)

N, number of food reactions of the evaluation cohort in which the severity indicator is present; #, percent of food reactions excluding those in which the severity indicator is missing (missing values shown in Table S6); CMT, critical medical treatment; OR, odds ratio; ROC‐AUC, receiver operating characteristic area under the curve.

Mild in oFASS‐3 and Grade 1 in oFASS‐5 were used as reference categories.

*OR in nFASS corresponds to the relative increment of probability per 1 point increment in the nFASS.

All ORs are significant (Wald's test p <.05), except for adrenaline use and grades 2 and 3 of oFASS‐5

FIGURE 1

A, Model evaluation: ROC curves for oFASS‐3, oFASS‐5, and nFASS in the EuroPrevall outpatient clinic cohort. B, External validation: ROC curves for the severity indicators with nFASS. TPR, true‐positive rate; FPR, false‐positive rate; ROC, receiver operating characteristic, CMT, critical medical treatment

Classification measures of the models were calculated (Table S10). With the outcomes adrenaline and CMT, Se ranged from 0.87 to 0.92 and Sp from 0.73 to 0.78. With emergency care, Se was around 0.70 and Sp 0.75. The PPVs ranged from 0.10 to 0.63 depending on the prevalence of the severity indicators in the cohort (with higher outcome prevalence, higher PPV).

Predicted and observed (real) probabilities of the severity indicators show that oFASS‐3, oFASS‐5, and nFASS are well calibrated (Table S11).

Internal validation

As a mechanism to account for overfitting of the FASS models, a bootstrap internal validation over the ROC‐AUC was undertaken. The bias and SE results for all the outcomes (Table S12) show negligible overestimation biases (1.0 × 10−6–1.23 × 10−3).

External validation

The FASS scores were implemented in the food reactions of the external validation cohorts and the severity distributions are presented in Table 3. In the NORA cohort close to 90% were grade 4 and 5 reactions in oFASS‐5, and 7.02% had systemic reactions with involvement of 2 or more organs (grade 3), which is consistent with the inclusion criteria in NORA. , In contrast, in the SAFE cohort on apple allergic patients, close to 60% of patients had exclusively mild‐grade 1 oropharyngeal symptoms, and severe reactions only appeared in 9% of subjects. It can be seen that nFASS mapped correctly onto oFASS for the 5 external validation cohorts.

In the external validation, we used the best 3 severity indicators of the evaluation in terms of discrimination, classification, and calibration: adrenaline, CMT, and emergency care. The frequency of these outcomes increased significantly in all the cohorts as severity increased classified with oFASS (Table S13). The predictive performance of FASS was evaluated using logistic regression. The ROC‐AUC with their 95%CI are presented in Table 5 and those of nFASS in Figure 1B. Overall, there was a trend for lower ROC‐AUC for oFASS‐3 and higher for nFASS in all the cohorts. The ROC‐AUC for the 3 FASS models were lower in NORA compared with the other cohorts.

TABLE 5

External validation: logistic regression models with oFASS‐3, oFASS‐5, and nFASS

Severity indicator (outcome)		Adrenaline ROC‐AUC (95% CI)	CMT ROC‐AUC (95% CI)	Emergency care ROC‐AUC (95% CI)
NORA cohort N = 2020	N(%)#	400 (22.35%)	226 (12.6%)	1411 (69.88%)
	oFASS‐3	0.53 (0.49, 0.58)	0.54 (0.48, 0.6)	0.53 (0.48, 0.57)
	oFASS‐5	0.59 (0.55, 0.63)	0.65 (0.6, 0.69)	0.57 (0.54, 0.61)
	nFASS	0.62 (0.59, 0.65)	0.7 (0.66, 0.73)	0.58 (0.56, 0.61)
HCSC Cohort N = 129	N(%)#	4 (4.87%)	3 (2.3%)	37 (28.68%)
	oFASS‐3	0.96 (0.91, 1.0)	0.96 (0.89, 1.0)	0.63 (0.46, 0.8)
	oFASS‐5	0.98 (0.93, 1.0)	0.98 (0.94, 1.0)	0.65 (0.52, 0.79)
	nFASS	0.97 (0.94, 1.0)	0.98 (0.94, 1.0)	0.68 (0.59, 0.78)
Europrevall general population N = 635	N(%)#	1 (0.15%)	0	15 (2.37%)
	oFASS‐3	0.89 (0.66, 1.0)	Not estimable	0.74 (0.61, 0.87)
	oFASS‐5	0.89 (0.68, 1.0)		0.79 (0.65, 0.93)
	nFASS	0.94 (0.91, 0.96)		0.77 (0.64, 0.89)
SAFE cohort N = 385	N(%)#	Information not collected		19 (5.02%)
	oFASS‐3			0.81 (0.69,0.92)
	oFASS‐5			0.82 (0.71,0.94)
	nFASS			0.84 (0.74,0.95)
iFAAM cohort N = 453	N(%)#	52 (11.76%)	49 (11.08%)	35 (7.91%)
	oFASS‐3	0.66 (0.60, 0.72)	0.75 (0.69, 0.80)	0.67 (0.60, 0.74)
	oFASS‐5	0.70 (0.64, 0.77)	0.76 (0.70, 0.82)	0.72 (0.64, 0.80)
	nFASS	0.72 (0.65, 0.78)	0.78 (0.72, 0.84)	0.74 (0.66, 0.82)

#, percent of food reactions excluding those in which the severity indicator is missing (missing values shown in Table S7).

ROC‐AUC, receiver operating characteristic area under the curve; 95% CI, 95% confidence interval.

Comparison of nFASS to the BWS ranking of allergy healthcare professionals

The responses of 334 allergy healthcare professionals were modeled following BWS methodology, and a BWS score was assigned to each of the 32 food reaction scenarios. The reaction scenarios include information on the symptoms/signs and on the treatment given (or not), and the final outcome of the reaction. The severity of these reactions was also scored with FASS (Table S8). A strong correlation (Spearman R 0.89) was found between BWS and nFASS values (Figure S2), even if FASS severity scoring was derived solely on the basis of symptoms.

DISCUSSION

We have developed and validated a severity score of food allergic reactions with ordinal (oFASS‐3 and oFASS‐5) and numerical (nFASS) formats that map consistently. They have different granularity and are intended to be used by different stakeholders in different settings. After a comprehensive validation following the TRIPOD statement (Table S14), FASS has proven to be a reliable and well calibrated method to describe severity of food allergic reactions. This was shown in the EuroPrevall outpatient clinic cohort used in the evaluation and was confirmed by bootstrap internal validation, and by external validation in 5 different cohorts that cover the whole spectrum of FA. Additionally, nFASS is strongly correlated to the classification of severity of food reaction scenarios done by expert allergy clinicians, further supporting the good performance of FASS.

For the validation, we selected as severity indicators (outcomes) the request of emergency care or the medications given to control the reactions, since they reflected management decisions taken by the patients themselves or by the clinicians providing medical care. The severity indicators were those collected in the forms of the evaluation cohort, with the CMT outcome built to identify the most severe ones. The variability in the management of allergic reactions and anaphylaxis and the underuse of adrenaline in relation to the recommendations ,  might lead to a limitation of the selected outcomes. We have shown that the frequency of the outcomes significantly rose with increasing severity of reactions graded with oFASS in the evaluation cohort (Table S9). The discrimination, classification, and calibration of FASS were better for the outcomes of use of adrenaline, CMT and request of emergency care selected for the external validation (Figure 1A, Table 4, Tables S10 and S11), suggesting that they are more appropriate severity indicators than use of any medication, antihistamine, and corticosteroids. Also in the external validation cohorts, the 3 selected outcomes significantly increased as severity increased by oFASS (Table S13). We thus believe our results support the adequacy of adrenaline, CMT and request of emergency care as severity indicators. Additionally, several studies , , reporting that adrenaline administration by health professionals increases with increasing severity of the allergic reactions support the selection of adrenaline as an outcome. This has been shown in the pre‐hospital management of anaphylaxis by ambulance crew in Manchester, UK, in two pediatric emergency departments of Marseille, France, and in the management of severe allergic reactions done by health professionals of 10 European countries of the NORA network. With the external validation cohorts, we found an overall good predictive performance, especially of nFASS, with some limitations in the NORA cohort. Due to the fact that almost 90% of the reactions included in NORA were severe/grade 4–5 (Table 3), the discrimination capacity of oFASS is limited (ROC‐AUC<0.7) (Table 5). The best performance is found with nFASS for CMT (ROC‐AUC 0.7) due to its higher granularity, and because CMT may be a more adequate outcome than adrenaline or request of emergency care for these anaphylaxis patients. Further work is needed to assess the performance of FASS with other outcomes for anaphylaxis (ie, number of adrenaline doses, intravenous adrenaline, intensive care admission), but since this information was not collected in the evaluation cohort, we could not analyze them in the external validation.

Like other scoring systems of allergic reactions, FASS has an anatomical basis, , , , , , , , , , , , , , , , , , , , but includes some novel approaches in scoring severity, present already in the (more conventional) oFASS.

Isolated oral symptoms (OAS) are included separately in oFASS‐3 (mild) and oFASS‐5 (grade 1), because they are a local reaction, and frequently the only clinical manifestation of FA as can be seen in Tables 2 and 4. Actually, 30% of all the food reactions (3545 out of 11854) of the evaluation and validation cohorts would not have been properly classified without including such category. In the currently available grading systems applied in FA, OAS is either absent , , , or included with symptoms of other organs under the same grade. , , ,

In oFASS when the number of organ/systems involved increase, the severity increases, but reactions will only be considered severe (Table 1) if Lx, BR, CV, and/or NS are involved, even if isolated. When reactions involve the eye/nose, skin, GI tract, uterus, they are considered moderate in oFASS‐3. The oFASS‐5 provides more detail by dividing moderate reactions into grades 2 (1 target organ) and 3 (>1 organ), and severe reactions into grades 4 (Lx/BR) and 5 (CV, NS). So with oFASS‐5, it can be seen in the evaluation and validation cohorts (Table 3) that the most frequent reactions among the moderate ones are those with only one organ/system affected (grade 2), and among the severe ones those with respiratory involvement (grade 4). The severe grade in oFASS‐3 and grades 4 and 5 in oFASS‐5 capture all the potentially life‐threatening anaphylaxis. ,

The anatomical approach to classify severity of oFASS‐3 is a simple, easy‐to‐remember system to identify the potentially life‐threatening anaphylaxis and prompt the early use of adrenaline. It can thus be used to educate patients and their careers, healthcare professionals, and even non‐healthcare professionals who may face food allergic reactions at work (eg, restaurants, schools). At a population level, it may be useful for education and raising awareness of food anaphylaxis, and thus of interest for public health authorities. oFASS‐5 may be more informative for allergy healthcare professionals who need to document reactions with more detail, but even so, it is still a simple system easy‐to‐remember and use. In summary, oFASS‐3 may be of highest value for patients and non‐healthcare professionals, and both oFASS‐3 and oFASS‐5 for all healthcare professionals in their clinical practice. Furthermore, oFASS‐3 and oFASS‐5 have shown to be reliable in measuring severity, with a trend for a better performance of oFASS‐5 (Tables 4 and 5) related to its higher level of detail.

In contrast to other ordinal systems, we have not considered in oFASS the intensity of the symptoms/signs of the organ/system affected, or whether they are subjective or objective. This was done for the sake of simplicity and the intended use aforementioned. When more detail is required nFASS can be applied. As an example, a reaction with “skin involvement” will be classified as moderate with oFASS‐3 and grade 2 with oFASS‐5, regardless of whether the skin involvement is “mild pruritus” or “intense pruritus with generalized erythema, urticaria and angioedema.” However, the nFASS will score 2.36 for the later and 2.01 for mild pruritus, and it is thus able to differentiate between these two reactions.

nFASS seems suitable for scoring reaction severity in clinical research settings and could help in the risk management of FA by including beside reference doses the severity parameter. We are currently exploring its usefulness in observational and intervention studies, analyzing the effect of cofactors and food allergen immunotherapy on the severity of food allergic reactions.

Further validation of FASS in other retrospective and, more importantly, prospective cohorts in both clinical and research settings, is needed to refine this tool and confirm its reliability. FASS is based on the symptoms and organs affected which are not specific of food allergic reactions. Similar symptomatology and organ involvement can be observed in allergic reactions elicited by insect stings and drugs, and we are currently working in the adaptation and validation of FASS to allergy triggers other than foods.

In order to facilitate the use of FASS, we have developed a software tool written in R language that allows to implement oFASS‐3, oFASS‐5, and nFASS in any dataset of food allergic reactions. The FASS tool is able to read any file in table format from Excel, SPSS, and STATA. The software tool, a tutorial and an example are available at Zenodo http://doi.org/10.5281/zenodo.4836276.

Through the FASS dissemination, widespread use, and further refinement, we can achieve the goal of improving education and immediate care decisions of patients and clinicians, help advance research, and guide the food industry and the health authorities.

CONFLICT OF INTEREST

Dr Fernández‐Rivas reports research grant to her institution from EU FP7 for the iFAAM project related to the work reported in this manuscript; research grants to her institution from Spanish Government (MINECO, ISCIII), and Aimmune Therapeutics and Diater, consultancy fees from Aimmune Therapeutics, Novartis and SPRIM, fees for participation in DSMB from DBV, lecture fees from Aimmune Therapeutics, ALK, Allergy Therapeutics, Diater, GSK, HAL Allergy and Thermo Fisher Scientific, outside the submitted work. Dr. Beyer reports grants/research supports from Aimmune Therapeutics, Danone/Nutricia/Milupa, DBV, Hipp, Hycor, Infectopharm and honoraria or consultation fees from Aimmune Therapeutics, Bencard, Danone/Nutricia/Milupa, DBV, Hipp, Hycor, Infectopharm, Jenapharma, Mylan/Meda, Nestle, Novartis, Thermo Fisher outside of the submitted work. Dr de Blay reports advisory board membership of Aimmune Therapeutics, Stallergènes Greer, Mundipharma, Novartis, Regeneron, DVB, Sanofi, ALK, Boehringer, and AstraZeneca, outside the submitted work. Dr Hourihane reports research funding from Aimmune Therapeutics, DBV Technologies, Johnson&Johnson, Dublin Skin and Cancer Hospital Charity, National Children's Research Centre, Dublin, Ireland, Temple St Hospital Foundation Charity and Clemens von Pirquet Foundation; advisory board membership of Aimmune Therapeutics; speaker bureaux of Aimmune Therapeutics, DBV Technologies and Danone‐Nutricia, outside the submitted work. Dr Papadopoulos reports research Support from Gerolymatos Int, Capricare, Nutricia, Vian; Speaker/Chairperson fees from HAL, Menarini/Faes Farma, MSD, Biomay, Novartis, Nutricia, Sanofi, Boehringer Ingelheim, Mylan/Meda, Asit Biotech; Advisory board fees from HAL, Menarini/Faes Farma, Novartis, Nutricia, GSK, AstraZeneca, Mylan/Meda, outside the submitted work. Prof. Worm declares the receipt of honoraria or consultation fees by the following companies: ALK‐Abelló Arzneimittel GmbH, Mylan Germany GmbH, Leo Pharma GmbH, Sanofi‐Aventis Deutschland GmbH, Regeneron Pharmaceuticals, DBV Technologies S.A, Stallergenes GmbH, HAL Allergie GmbH, Allergopharma GmbH & Co.KG, Bencard Allergie GmbH, Aimmune Therapeutics UK Limited, Actelion Pharmaceuticals Deutschland GmbH, Novartis AG, Biotest AG., AbbVie Deutschland GmbH & Co. KG and Lilly Deutschland GmbH, outside the submitted work. Prof. Roberts reports funding to his institution by the European Commission and the UK Food Standards Agency to undertake research as part of iFAAM, for the reported work in this manuscript. Dr van Ree reports research grant from EU FP7 for the iFAAM Project for the reported work in this manuscript; research grants from EU FP7, Dutch Science Foundation, and Health Holland, a research contract from Agany Inc., consulting fees from HAL Allergy BV, Citeq BV, Angany Inc., lecture fees from HAL Allergy BV, ALK and Thermo Fisher Scientific, support for attending meetings from HAL Allergy BV, advisory board fees from HAL Allergy BV and Agany Inc., stock options from Agany Inc., outside the submitted work. Dr Turner reports grants from European Commission H2020 iFAAM collaboration for the reported work in this manuscript; grants from UK Medical Research Council, NIHR/Imperial BRC, and UK Food Standards Agency; advisory board fees from ILSI Europe, Aimmune Therapeutics, and AllerGenis, outside the submitted work. Dr. Mills owns founder shares in Reacta Biotech Ltd, a company developing oral food challenge diagnostics for food allergy. Funding for research and consultancy is received from the company through the University of Manchester. All other authors declare no conflicts of interest.

AUTHOR CONTRIBUTIONS

MFR lead the score development and validation, and wrote the manuscript; MFR, ENCM, AM, and GR lead the iFAAM‐EAACI collaboration. MFR, GMM, BBW, KB, MC, MRD, KECG, KHS, JOBH, LKP, AM, MW, GR, RVR, PJT, and ENCM were involved in the expert consensus for the score development. IGG developed nFASS. IGG and AGF performed the statistical analysis under the guidance of MFF, CFP, and MFR. IGG developed the software tool for scoring severity. AGF revised and refined it. MFR, SDB, GMM, BBW, RA, SB, KB, FB, MC, MRD, RD, KECG, KHS, JOBH, MJC, ACK, TK, TML, NGP, TAP, LKP, AP, SLS, AS, AS, MT, SVC, RNVB, MW, GR, RVR, CFP, PJT, and ENCM contributed to the acquisition of data of the different cohorts. PJT lead the MaxDiff survey of health professionals. All the authors critically revised the manuscript and approved the final version.

Appendix S1

Click here for additional data file.

Appendix S2

Click here for additional data file.