The reliability and validity of outcome measures for atopic dermatitis in patients with pigmented skin: A grey area.

BACKGROUND: Outcome measures for atopic dermatitis (AD) patients with pigmented skin have neither been developed nor validated.
OBJECTIVE: To compare the reliability and validity of four common AD outcome measures in patients with various levels of skin darkness.
METHOD: The inter- and intra-rater reliability and construct validity of the EASI (Eczema Area and Severity Index), objective-SCORing Atopic Dermatitis (oSCORAD), Three Items Severity index (TIS) and Six Areas, Six Sites Atopic Dermatitis (SASSAD) were evaluated in 18 patients of various levels of skin darkness, using their full body photographs, by five trained clinicians.
RESULTS: The inter-rater reliability intraclass coefficient (ICCs) and 95% confidence intervals were poor for highly pigmented patients: EASI -.054(-.200 to .657), oSCORAD -.089(-.206 to .598), TIS -.21(-.24 to .147), SASSAD -.071(-.200 to .631); fair for mildly pigmented patients: EASI .464(.140-.839), oSCORAD .588(.265-.89), TIS.524(.200-.865), SASSAD .41(.045-.775); and fair to good for non-pigmented patients: EASI .64(.330-.908), oSCORAD .586(.263-.889), TIS .403(.09-.809), SASSAD .667(.358-.916). Erythema likely contributed to the inter-rater variability. Construct validity had significant correlations across all measures in non-pigmented patients, but no correlations in highly pigmented patients.
CONCLUSION: AD outcome measures have poor reliability and validity in highly pigmented patients, with variations in erythema perception being a contributor.

Introduction

Skin color contains vital diagnostic clues in dermatology, as it reflects the underlying pathological process. Patients with pigmented skin are a complex population in dermatology. Inflammatory conditions such as atopic dermatitis (AD) and psoriasis are more difficult to assess in these patients, putting them at risk for misdiagnosis or mistreatment. Many times, patients with skin of color have been excluded from clinical trials in dermatology due to the difficulties of assessing disease severity.

There are many reasons to account for the difficulties in assessing pigmented patients with AD. Phenotypic variations secondary to genetic differences is one reason; for example, filaggrin-2 mutation variations in AD have been found in African-American patients, and have been associated with a more persistent disease course (Margolis et al., 2014, Torrelo, 2014). Another example is that pigmented skin has been shown to be less likely to develop erythema when exposed to irritants (Berardesca and Maibach, 2003). Also, cultural and environmental factors can change how the skin is cared for, leading to further heterogeneity in manifestation. In addition, the clinician’s perception of color may be distorted by the background skin pigmentation or be mistaken for post-inflammatory hyperpigmentation (Ahmad Fadzil et al., 2009). Furthermore, clinical experience with managing patients with skin of color is a contributing factor. As a result, considerable intrarater and interrater variations in assessing the patient can occur, while the validity is compromised by the clinical heterogeneity.

AD is a common dermatological condition that can affect patients of all ethnicities and skin types. In the pediatric population, AD has comparably high prevalence of 17% in the United States, 14% in England, 24% in Japan, 17% in Korea, 17% in South Africa (mixed Caucasians and Blacks), 20% in Kenya, and 32% in Melbourne, Australia (Esamai et al., 2002, Oh et al., 2004, Robertson et al., 2004, Shaw et al., 2011, Simpson et al., 2009, Sugiura et al., 1998, Zar et al., 2007). A systematic review in 2012 has also found that in Africa, Eastern Asia, Western Europe and parts of Northern Europe, trends in AD were mainly increasing (Deckers et al., 2012).

The course of AD is typically chronic, requiring ongoing monitoring and an accurate assessment instrument. Also, many clinical trials have being conducted to evaluate interventions for AD, calling for a uniform outcome measure. Significant international efforts have been made to facilitate the standardization and validation of AD outcome measures. The Harmonising Core Outcome Measures (HOME) for eczema initiative has had several meetings (Chalmers et al., 2014, Schmitt and Williams, 2010, Schmitt et al., 2012), and confirmed that excoriation, erythema, edema, or papulation and lichenification are four essential components for the assessment of AD severity. A recent systematic review indicated that out of the 16 proposed outcome measures used in clinical trials, only the EASI (Eczema Area and Severity Index) and the SCORing Atopic Dermatitis (SCORAD) have received adequate validation (Schmitt et al., 2013). Meanwhile, a recent HOME consensus recommended the EASI alone as the optimal outcome measure (Chalmers et al., 2014).

Despite all of the above progress, the assessment of AD in pigmented-skin patients is still a grey area requiring attention. In fact, a study has found the underreporting of patient’s skin type in clinical trials, with only 59.5% of the clinical trials published in the United States between 2000 and 2009 reporting the patient’s race or ethnicity (Hirano et al., 2012). Another systematic review showed that there is a dearth of studies demonstrating efficacy of systemic AD therapy in different racial and ethnic patient subsets in the United States (Bhattacharya and Silverberg, 2014).

The aim of this study was to contribute to the expanding work in the standardization of AD outcome measures by addressing the issue of disease assessment in patients with pigmented skin. A prospective study was conducted to compare the interrater and intrarater reliability, as well as convergent construct validity, of the four most commonly used atopic dermatitis outcome measures in patients with various levels of skin darkness. This study also aimed to explore the underlying factors contributing to the variations, such as erythema.

Materials and methods

This prospective study was granted ethical approval from the South Eastern Sydney Health District Human Research Ethics Committee Northern Sector (reference: HREC/12/POWH/155).

Outcome measures tested

The outcome measures evaluated in this study were chosen as these have been most frequently validated as per a systemic review published in 2013 (Schmitt et al., 2013). These include the EASI, SCORAD, of which has a clinician-reported only version called the objective SCORAD (oSCORAD), Three Items Severity index (TIS) and Six Areas, Six Sites Atopic Dermatitis (SASSAD; Berth-Jones, 1996, Hanifin et al., 2001, Stalder and Taieb, 1993, Wolkerstorfer et al., 1999).

Participants and assessors

The full-body photographs of 20 patients with AD were obtained from dermatology outpatient clinics from Sydney. Two patients were later excluded as they had more than two body parts missing from their full-body photographs.

Five assessors participated in the scoring process (D.F.M, M.J.D, A.G.H, S.V.J, and K.L). All assessors were either qualified dermatologists or have been doing full-time dermatology research, and hence had been familiar with atopic dermatitis. Two (M.J.D and K.L) had trained in the Philippines and were used to darker-skinned patients. One had trained in North Carolina (D.F.M), where approximately one third of patients were African-American. All assessors were required to attend a training lecture on the use of the EASI, oSCORAD, TIS, and SASSAD. Also, the assessors were required to attend a debriefing session prior to each scoring session to raise queries regarding the administration of these scoring systems. The assessors were completely blinded to the identity of the patients chosen.

Scoring process

The assessments were performed for 2-hour sessions, over 4 days. Each session was limited to 2 hours in length, to avoid assessor fatigue. Full-body photographs of the 18 patients were presented on a screen of at least 1.5 m by 1.5 m, with an example demonstrated in Supplementary Fig. 1. Three patients with various levels of skin pigmentations, whose identities were unknown to the assessors, were also arranged by a separate investigator to have their photographs repeatedly shown at the end of the 18 patients for intrarater reliability testing. For each of the patients scored, the assessors were given four color-coded scoring sheets including the four measures. The assessors were given the time to view the photographs until they were satisfied with their scores. Each assessor was neither allowed to look at their own scores from the other outcome measures, nor another assessor’s scores. When any patients had minor body parts missing, which five patients did, all assessors were asked not to assess the particular missing body part across all scores.

Data input

All outcome measure scores were calculated by two separate study investigators. Data input was performed by one investigator, then separately double-checked by another investigator. The five patients with minor body parts missing from their photographs had their EASI, SASSAD, and oSCORAD’s total denominators reduced to reflect the exclusion of the corresponding body parts.

Categorization of skin pigmentation levels

Each patient’s skin pigmentation was scored by all assessors on a numerical scale of 0 to 10, ranging from 0 representing no pigmentation, to 10 representing the darkest level of pigmentation. The average of each patient’s pigmentation score across the five assessors was then used to categorize patients into three groups: nonpigmented (score range 0-3), mildly pigmented (score range 3.1-7) and highly pigmented (score range 7.1-10). These ranges were chosen as they divide into three approximately equal categories.

Statistical analysis

All statistical analyses were performed using SPSS Version 22.0 (Armonk, NY; IBM Corp.). A professor of statistics (M.G.L) from the Kirby Institute of University of New South Wales provided help with choosing the most appropriate statistical tests.

For reliability testing, both interrater and intrarater reliabilities were assessed by the Intraclass coefficient (ICC) with 95% confidence interval (CI), using a one-way random analysis variance model. When an ICC is below .40, the clinical correlation is poor; when it is between .40 and .59, the level of correlation is fair; when it is between .60 and .74, the level of correlation is good; and when it is between .75 and 1, the level of clinical significance is excellent (Cicchetti, 1994). Scatterplots were constructed to illustrate interrater differences across all outcome measures and skin types.

To determine whether the erythema components contributed to the variability in reliability, the erythema component and the “total minus erythema component” of each outcome measure were separately inputted. The ICCs and coefficient of variations (CV) means of the erythema component and the “total minus erythema component” were then calculated. The null hypothesis is that the ICC for the erythema component would be bigger than ICC for the “total minus erythema component,” while the CV means across all patients should be smaller. This would indicate that erythema did not contribute to the variability of the scores. To determine the significance of the difference in the ICCs, the CVs of the erythema components and the “total minus erythema components” were correlated using the paired T-test.

For convergent construct validity, all outcome measures were correlated with each other, and the respective Spearman rho correlation coefficients were determined.

Additional assessments by an overseas dermatologist

Additional convergent construct validity testing was performed by a South African dermatologist with expertise in assessing atopic dermatitis in pigmented skin patients (N.C.D). The results were compared to the Australian dermatologists’ results to determine whether experience would improve the validity of outcome measures in highly pigmented-skin patients.

Results

Demographics

Altogether, 18 patients were included in the final analysis for interrater reliability and convergent construct validity (Table 1). Of these, three were of Asian background, 11 were of Caucasian background, two were of African background, and four were of Indian background. Three patients were also used for intrarater reliability analysis (Table 2).

Table 1

Characteristics of the Total Patient Cohort (n = 18).

	Highly pigmented⁎	Mildly pigmented	Nonpigmented	Overall
N (%)	4 (22.2)	7 (38.9)	7 (38.9)	18 (100)
Age range (years)	7-40	2-65	1-65	1-65
Mean age (years)	26.8	29.1	32.3	29.8
Male (%)	2 (50)	5 (71.4)	3 (42.9)	10 (55.6)
Female (%)	2 (50)	2 (28.6)	4 (57.1)	8 (44.4)

The pigmentation groups were derived from the mean pigmentation given by the assessors: nonpigmented (score range 0-3), mildly pigmented (score range 3.1-7), and highly pigmented (score range 7.1-10).

Table 2

Characteristics of the Patients for Intra-rater Reliability Testing (n = 3).

	Patient 1	Patient 2	Patient 3
Age⁎	60	22	25
Sex	Female	Female	Female
Ethnicity	Caucasian	Asian	African
Average pigmentation score	2.0	5.4	7.6

When the investigator was unable to ascertain what the patient’s age was when his/her photo was taken, an estimate of age was made by two separate investigators.

Inter-rater reliability

The interrater reliability ICCs (and their 95% CIs) for the highly pigmented patients were: TIS, -.21 (-.24 to .147); SASSAD, -.071 (-.2 to .631); EASI, -.054 (CI: -.2 to .657); and oSCORAD, -.089 (-.206 to .598), indicating very poor interrater reliabilities. The ICCs for the mildly pigmented patients were: TIS, .524 (.2-.865); SASSAD, .341 (.045-.775); EASI, .464 (.14-.839); and oSCORAD, .588 (.265-.890), indicating fair interrater reliability. The ICCs for the nonpigmented patients were: TIS, .403 (.09-.809); SASSAD, .667 (.358-.916); EASI, .64 (.33-.908); and oSCORAD, .586 (.263-.889), indicating fair interrater reliability for the TIS and oSCORAD, and good interrater reliability for the EASI and SASSAD.

Interrater scatterplots showed that, in highly pigmented skin patients, all scores have poor interrater reliability regardless of disease severity. In mildly pigmented patients, all scores had poorer interrater reliability with increased disease severity. In nonpigmented patients, interrater reliability does not appear to be influenced by disease severity (Supplementary Fig. 1 and Table 3).

Table 3

Inter-rater Reliability ICCs of the EASI, OSCORAD, TIS, and SASSAD Total scores; the Erythema Component; and the “Total Minus Erythema Component” by Pigmentation Level.

	EASI	oSCORAD	TIS	SASSAD
Highly pigmented group ICCs (95% CI; n = 4)
Total score	.509 (.300-.731)	-.089 (-.206 to .598)	-.210 (-.240 to .147)	-.071 (-.201 to .631)
Erythema	-.171 (-.200 to .363)	-.230 (-.245 to .015)	-.055 (-.200 to .656)	-.088 (-.206 to .599)
Total minus erythema	-.072 (-.201 to .628)	-.062 (-.198 to .645)	-.229 (-.308 to .399)	-.144 (-.222 to .462)
Mildly pigmented group ICCs (95% CI; n = 7)
Total score	.464 (.140-.839)	.588 (.265-.890)	.524 (.199-.865)	.341 (.045-.775)
Erythema	.607 (.290-.896)	.636 (.320-.906)	.603 (.281-.895)	.601 (.280-.895)
Total minus erythema	.358 (.570-.785)	.500 (.176-.855)	.413 (.100-.814)	.193 (-.051 to .670)
Nonpigmented group ICCs (95% CI; n = 7)
Total score	.640 (.330-.908)	.586 (.263-.889)	.403 (.092-.809)	.667 (.358-.916)
Erythema	.749 (.470-.941)	.423 (.108-.820)	.409 (.097-.812)	.783 (.526-.950)
Total minus erythema	.533 (.209-.869)	.560 (.235-.879)	.303 (.180-.751)	.459 (.139-.837)
Overall ICCs (95% CI; n = 18)
Total score	.431 (.223-.673)	.621 (.423-.805)	.480 (.270-.710)	.504 (.294-.747)
Erythema	.609 (.409-.798)	.509 (.300-.731)	.568 (.363-.772)	.701 (.522-.853)
Total minus erythema	.509 (.300-.731)	.597 (.396-.790)	.436 (.228-.677)	.340 (.141-.598)

Erythema’s contribution to variability

The inter-rater reliability ICCs and coefficient of variations (CV) means of the erythema components and “total minus erythema components” were compared against each another. For the highly pigmented patients, the EASI and oSCORAD had slight, but clinically significant, superior reliabilities when the erythema components were excluded (EASI: -.171 vs. -.072, p = .034; oSCORAD: -.230 vs. -.062, p = .04; Table 3). This superior reliability was not present in the TIS or SASSAD, or in any other pigmentation groups. These results suggest that the erythema components have likely contributed to the variability of EASI and oSCORAD in highly pigmented skin patients. However, given the poor ICCs of the “total minus erythema components,” other factors were likely to have also contributed to the variability.

When comparing the CV means (an indicator of variability) in highly pigmented patients, there were higher CV means in the erythema components than the “total minus erythema components”, across all outcome measures (Table 4). The higher CV mean of the erythema component was not present in any other outcome measure or pigmentation groups, except for SASSAD in mildly pigmented patients. These results were also evident of erythema’s contribution to the variability of the outcome measures in highly pigmented patients.

Table 4

Coefficient of Variation (CV) Means of the Erythema Components Versus the “Total Minus Erythema Components” of the Four Outcome Measures by Pigmentation Level.

	EASI	oSCORAD	TIS	SASSAD
Highly pigmented group CVs (n = 4)
Erythema	.987	.961	1.01	.852
Total minus erythema	.525	.331	.499	.427
Mildly pigmented group CVs (n = 7)
Erythema	.359	.323	.249	.323
Total minus erythema	.659	.283	.377	.471
Nonpigmented group CVs (n = 7)
Erythema	.578	.284	.201	.387
Total minus erythema	.746	.526	.807	.647

Intrarater reliability

For the patients with highly pigmented skin, the intrarater reliability ICCs were poor in all scores except for the SASSAD (Table 5). For the patients with mildly pigmented skin, the intrarater reliability ICCs were poor in the EASI and oSCORAD, and fair for TIS and SASSAD. For the patients with nonpigmented skin, the intra-rater reliabilities of the oSCORAD and TIS were poor, but for EASI and SASSAD were good. Given the wide 95% CIs of the results, the findings were limited by poor power. However, these results may suggest that in patients with highly pigmented skin, intrarater unreliability is more likely to be unreliable.

Table 5

Intra-rater Reliability ICCs of the EASI, oSCORAD, TIS and SASSAD by Pigmentation Level.

	EASI	oSCORAD	TIS	SASSAD
Highly pigmented	.391 (-.528 to .911)	.075 (-.728 to .832)	-.434 (-.899 to .574)	.787 (.064-.975)
Mildly pigmented	.037 (-.746 to .819)	-.537 (-.922 to .476)	.429 (-.494 to .918)	.458 (-.467 to .924)
Nonpigmented	.830 (.184-.980)	-.151 (-.818 to .747)	-.760 (-.922 to .122)	.699 (-.134 to .963)

Convergent construct validity

In highly pigmented patients, none of the scoring instruments significantly correlated with each other. In mildly pigmented patients, SASSAD is not statistically significantly correlated with any of the other scoring instruments. The correlations of the other three outcome instruments were EASI with TIS: .829 (p = .021), EASI with oSCORAD: .857 (p = .014), and oSCORAD with TIS: .919 (p = .003). In nonpigmented patients, the correlations of all four outcome measures were mostly statistically significant: EASI with TIS: .919 (p = .003), EASI with oSCORAD: 1.000 (p < .01), EASI with SASSAD: .786 (p = .036), oSCORAD with TIS: .919 (p = .003), oSCORAD with SASSAD: .786 (p = .036), and TIS with SASSAD: .793 (p = .033). These results suggest the poorer convergent construct validity of all outcome measures in highly pigmented patients.

The same 18 patients were assessed by the South African dermatologist (N.C.D). In highly pigmented patients, none of the scoring instruments significantly correlated, except for EASI with SASSAD (p = .051). In mildly pigmented patients, all outcome measures correlated well, but were limited by low power, as there were only two patients in this group (as per pigmentation score out of 10 rated by the South African dermatologist). In nonpigmented patients, the correlations of all four outcome measures were mostly statistically significant: EASI with TIS: .645 (p = .023), EASI with oSCORAD: .867 (p < .01), EASI with SASSAD: .967 (p < .01), oSCORAD with TIS: 1.000 (p < .01), oSCORAD with SASSAD: .872 (p < .01), and TIS with SASSAD: .648 (p = .023). These results were comparable to the Australian dermatologists, confirming the poorer convergent construct validity of all outcome measures in highly pigmented patients.

Discussion

To the authors’ knowledge, this is the first study to evaluate the commonest AD outcome measures in patients with pigmented skin. Overall, all AD outcome measures—the EASI, the oSCORAD, the TIS, and the SASSAD—may have poor reliability and validity in patients with very dark skin. All four measures had very poor interrater reliability when used in patients of highly pigmented skin with ICCs of < .4, regardless of disease severity. Thus, we would warn against the performance of AD severity assessments by two different doctors, at different time points, in dark-skinned patients. All four measures also produced poorer results when used in mildly pigmented patients as compared to nonpigmented patients, with the level of variation increasing with disease severity. Also, all measures had poor intra-rater reliability in highly pigmented patients, although the power of the results was limited by the small sample size.

The study suggested that erythema is a contributor to the inter-rater variations, with higher CV means of erythema components compared to the “total minus erythema components.” This finding echoed a previous study by Ben-Gashir and Hay (2002), which investigated the effect of including and excluding erythema assessment on the assessment of AD severity; when the erythema component of the oSCORAD was excluded and adjusted for the total, disease was found to be severer in Black children. Our study went into more breadth to evaluate this issue. Given that, in our study, the interrater ICCs of the “total minus erythema components” were also poor in the highly pigmented patients, other factors are likely have also contributed to the variability.

The inferior reliability of all outcome measures in nonpigmented patients suggested by this study when compared to other studies is likely due to the fact that patients were virtually scored instead of in a routine clinic setting. Three-dimensional assessments for lichenification and papulation were likely compromised, as was the feeling of heat via palpation to distinguish between active inflammation and postinflammatory hyperpigmentation. Given the recent growth of teledermatology worldwide, this study highlighted that virtual scoring may be suboptimal for a thorough and accurate assessment of AD. The authors also acknowledge the limited number of patients used for this study, given the difficulties to obtain high-definition, full body photos of very dark patients. This was compensated for by using at least five trained clinicians to perform the assessments. Another limitation was that being a single-center study, the poorer reliability and validity may not be demonstrated in all other settings and locations. To compensate for this, a South African dermatologist experienced with > 95% of her patients being Fitzpatrick skin types V-VI from Durban, South Africa, was invited to also score the same patients for convergent construct validity analyses.

Regarding construct validity, all four measures were again inferior in highly pigmented patients, even when scored by a South African dermatologist with experience in assessing dark skin. This is suggestive of underlying heterogenicity of clinical signs in highly pigmented patients, which may not have been included in the four common AD outcome measures.

From this study, a new outcome measure may be needed to take into account the manifestation and perception of erythema, as well as other clinical heterogeneity found in dark AD patients such as ichthyosis or prurigo nodularis (Vachiramon et al., 2012). A grey scale may more accurately reflect the erythema shown in ethnic skin, while a heat or temperature scale may help to distinguish between inflammation and post-inflammatory hyperpigmentation. The development of the new outcome measure should be followed by a multicenter validation study in a routine clinical setting. In the field of dermatology, this study also has implications for the validation of psoriasis outcome measures, as well as patch test reading protocols, in patients with very dark skin.

In conclusion, the study suggests that AD outcome measures have poor reliability and validity in patients with very dark skin. A new outcome measure may be needed to take into account the manifestation and perception of erythema and other clinical signs in these patients.

The following is the supplementary data related to this article.

Supplementary Fig. 1

Inter-rater scatterplots demonstrating the spread of scores for all four outcome measures in patients of various pigmentation levels. EASI scores for (a) highly pigmented patients, (b) mildly pigmented patients, and (c) nonpigmented patients; oSCORAD scores for (d) highly pigmented patients, (e) mildly pigmented patients, and (f) nonpigmented patients; SASSAD scores for (g) highly pigmented patients, (h) mildly pigmented patients, and (i) nonpigmented patients; and TIS scores for (j) highly pigmented patients, (k) mildly pigmented patients, and (l) nonpigmented patients. These scores demonstrate that in patients with highly pigmented skin, all scores and all disease severities have poor inter-rater reliability. In patients with mildly pigmented skin, across all scores interrater reliability decreases as disease severity increases. In patients with nonpigmented skin, interrater reliability generally is not influenced by disease severity.