Examination of the Validity of Skin Ultrasound to Quantitate Skin Involvement for Multicenter Clinical Trials in Patients with Systemic Sclerosis (SSc): A Systematic Literature Review (SLR).

OBJECTIVE: The study objective was to conduct a systematic literature review (SLR) of ultrasound of the skin in patients with systemic sclerosis (SSc) to establish the degree to which ultrasound of the skin has been validated, using the Outcome Measures in Rheumatology (OMERACT) Filter.
METHODS: We conducted an SLR of publications between 1950 and 2018, using PubMed and Cochrane library, to examine ultrasound validity to quantitate SSc skin involvement. Inclusion criteria were as follows: (1) in English; (2) used the 1980 or 2013 classification criteria for SSc criteria; (3) either a randomized controlled trial, an observational study, or a case study including more than 15 patients; (4) subjects 18 years of age or older; (5) for mixed patient populations, SSc results were separable; and (6) the ultrasound machine was clearly described. Exclusion criteria were as follows: (1) not in English; (2) data did not record at least one of the validation criteria; (3) subjects aged less than 18 years; (4) subjects had disease other than SSc (eg, localized scleroderma or scleroderma-like disease); (5) a letter to the editor or an editorial; and (6) involved a modified Rodnan skin score of less than 2. Descriptive statistics were generated for each criterion.
RESULTS: From an initial 292 citations, 14 articles (1,055 patients) met inclusion and exclusion criteria. The status of validation for ultrasound was evaluated by using the OMERACT criteria of truth, discrimination, and feasibility (in turn divided into nine different criteria). Face, criterion, content, construct, reliability, and responsiveness criteria were met, and the feasibility criterion was partially met, whereas discrimination and reproducibility criteria were not met.
CONCLUSION: Based on an SLR through December 31, 2018, ultrasound of the skin met some but not all validation criteria for use in clinical trials.

No systematic literature review examining the validity of the skin ultrasound in systemic sclerosis (SSc) has been published.

Ultrasound for SSc skin involvement has not been fully validated.

Of the nine validation criteria, face, criterion, content, construct, reliability, responsiveness, and (partially) feasibility were met.

Discrimination and reproducibility criteria were not met.

INTRODUCTION

Systemic sclerosis (SSc; scleroderma) is an uncommon autoimmune disorder of unknown etiology. A hallmark of this disease is the degree of cutaneous involvement, whereby the skin thickness varies extensively from a barely palpable and localized increase in thickness to a diffuse dermal sclerosis (1, 2). The validated gold standard for cutaneous involvement, the modified Rodnan skin score (mRSS), has been used to assess the magnitude and extent of cutaneous involvement in patients with SSc (2, 3). However, there are limitations intrinsic to the skin‐scoring method, such as problems with both intra‐ and interobserver variability (2, 3).

With this background, ultrasound has been introduced and used as a more objective measure of skin thickening (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15). In that context, it has been assumed that the ultrasound modality has been fully validated for rheumatic diseases such as SSc.

We wished to undertake a systematic literature review (SLR) of skin ultrasound with the goal of examining the degree to which ultrasound has been validated to measure skin thickening in multicenter clinical trials of SSc.

MATERIALS AND METHODS

The protocol followed was the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses for completion of the SLR. A complete literature search using two medical databases (PubMed and the Cochrane library) was conducted to identify studies between the time frame of 1950 and December 2018 that examined the validity of ultrasound to quantitate skin involvement of patients with SSc. All 292 articles reporting on SSc and ultrasound were found using specific medical subject heading (MeSH) terms: (((((((((("scleroderma, Systemic"[MeSh] OR scleroderma[tw] OR "systemic sclerosis"[tw] OR morphea[tw] OR "mixed connective"[tw] OR "overlap disease"[tw]))) AND (("ultrasonography"[MeSh] OR "ultra sound"[tw] OR ultrasound[tw] OR ultrason*[tw] OR sonograph*[tw]))) AND (("skin"[MeSh] OR skin[tw] OR epidermis[tw] OR dermis[tw])))) NOT ((editorial[pt] OR letter[pt])))) NOT ((animals[mh] NOT humans[mh])))) AND (1960/01:2018/06 [dp]).

Inclusion criteria were as follows: (1) in English; (2) the 1980 or 2013 classifications criteria were met; (3) either a randomized controlled trial, an observational study, or a case study involving more than 15 patients; (4) subjects aged 18 years or older; (5) if the article contained a mixed patient population, the patients with SSc and results were separable; or (6) the ultrasound machine was clearly described.

Exclusion criteria were as follows: (1) not in English; (2) data did not record at least one of the validation criteria; (3) subjects aged less than 18 years; (4) subjects had disease other than SSc (eg, localized scleroderma or scleroderma‐like disease); (5) letter to the editor or editorial; or (6) or mRSS score of less than 2.

In our literature review, we wanted to be sure that the articles did in fact refer to patients with skin involvement and scleroderma. With a skin score of only 1, we were not sure that the literature actually was referring to patients who had sufficient skin involvement for ultrasound evaluation.

All data were double extracted. Study selection and data extraction were performed independently by two reviewers (OA and KK). Disagreements were resolved by consensus or resolved by a third reviewer (DEF).

Outcome data in each study consisted of group size and the number of patients in each group who had an event for each outcome. Collected study characteristics included important inclusion and exclusion criteria. Quality assurance of the articles included within the study was performed using the Effective Public Health Practice Project (EPHPP) quality assessment tool (Supplemental Table 3), which showed that all 14 articles were moderate in rating (16). Two reviewers (OA and KK) independently assessed whether the biases to the studies' internal validity were adequately reported.

The Outcome Measures in Rheumatology (OMERACT) filter consists of the following three concepts: truth, discrimination, and feasibility, which in turn are divided into nine different validity criteria (17). The status of validation of ultrasound was evaluated by using the OMERACT criteria. The description of the objective criteria as well as the evidence and reasoning to define “met” or “not met” are provided in Table 1.

Table 1

OMERACT criteria: truth, discrimination, feasibility (17)

OMERACT Criteria	Outcome	Description of Objective Criteria	Evidence and Reasoning
Face	+	US method appears effective in terms of its stated aims to examine the skin thickness of patients with SSc.	Consensus exists in the literature that the US measured skin thickness in patients with SSc (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14).
Criterion	+	US of the skin correlates with a “gold standard”– skin histology	Criterion validity for US visualization of skin layers was reported in a study by Suliman et al as noted in the discussion (15).
			Chen et al supported the criterion validity by Suliman for histological skin thickness and local mRSS (18).
Content	+	To what extent the US method can assess the skin thickness over a large range of patients.	The studies included 1,055 subjects, 43% lcSSc, 24% dcSSc, with a large range of ages, disease durations, disease severity, and visceral involvement (see above and Supplemental Table 1) (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14).
Construct	+	US of the skin has a positive correlation with mRSS.	US measurements correlated with mRSS both at baseline and at 1‐year follow‐up (correlation coefficient: 0.48, P < 0.001) (10).
			US measurements correlated with the local mRSS from the corresponding anatomical region and also with the total mRSS (n = 88; correlation coefficient = 0.55; P = 0.001) (7).
			US measurements correlate with the mRSS as well as the severity score of the disease (correlation coefficient = 0.470, P = 0.002) (9).
			Subclinical dermal involvement was detected by US even in the skin areas in patients with lcSSc who had a normal local mRSS (correlation coefficient = 0.37, P = 0.04) (12).
			Patients with dcSSc skin thickness increased as echogenicity changed on the order of isoechoic, hypoechoic, and hyperechoic (P < 0.001) (11).
Discrimination	−	US of the skin can discriminate between treatments.	Data on treatments were not available. No treatments have been shown to be effective.
Reproducibility	−	US of the skin can be reproduced over a duration of time.	Data on reproducibility were not available.
Reliability	+	US of the skin can be reproduced over a short period of time (by obtaining the same measurement twice) with minimal error and with accuracy.	ICC (intra‐observer) at 5 sites: range of 0.92‐0.98 ICC (interobserver) at five sites: range of 0.83‐0.88 (4). Two separate occasions for control subjects demonstrated a good reproducibility (SD 0.06 mm) (2).
			Four occasions for one subject (CV of 2.7%; r = 0.98; P < 0.001) (13).
			Seventeen different sites (ICC range: 0.65‐0.94 and 0.55‐0.96) (3).
			“Skin thickness measurement determined by US was highly reproducible and there was little variability between observers.” No specific data given (14).
			Low variability except for the phalanx (ICC: 0.66) (7).
			Intra‐operator reproducibility was 96% (95% CI, 0.94‐0.97) (12).
Responsiveness	+	US of the skin can change over time.	Baseline: 8.53 (7.94‐9.03); 1‐year follow‐up: 8.28 (7.47‐8.94); change: −0.22 (−0.79 to 0.30); P value: 0.011; result: significant decrease in TST over 1 year; *TST: median (IQR) mm (10).
			Dermal echo intensity after photo chemotherapy (33.51 ± 9.34) significantly increased (IMPROVED) versus before therapy (21.23 ± 6.00, P < 0.01). Also, dermal thickness (1.20 ± 0.20) significantly decreased versus before therapy (1.38 ± 0.18, P < 0.05) (5).
Feasibility	+/−	US of the skin can be easily or conveniently done.	All 14 centers within the articles successfully demonstrated the ability to conduct US studies of the skin. Furthermore, 9 different US machines, 8 different probes, and 9 locations/combinations were used (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14). *See Supplemental Table 2.

+, met the validity criterion; −, did not meet the validity criterion; +/−, partially met validity criterion.

Abbreviations: CI, confidence interval; CV, coefficient of variance; dcSSc, diffuse cutaneous systemic sclerosis; ICC, intra/interclass correlation coefficient; IQR, interquartile range; lcSSc, limited cutaneous systemic sclerosis; mRSS, modified Rodnan skin score; OMERACT, Outcome Measures in Rheumatology; SD, standard deviation; TST, total skin thickness; US, ultrasound.

Figure 1 demonstrates the disposition of citations to derive the final articles used in the SLR. Of the initial 292 articles found, 278 were excluded according to the exclusion criteria. A total of 14 articles remained to be entirely extracted using a consistent case report form. The standardized case report form included all relevant extracted data for the purpose of the SLR. All data were double extracted using two reviewers (OA and KK). The following data were extracted if mentioned within the studies: journal information, funding sources, trial design, population demographics, SSc classification (limited cutaneous SSc [lcSSc], diffuse cutaneous SSC [dcSSc], control), anatomical regions measured, skin thickness measurements using ultrasound, mRSS values, correlations of ultrasound and mRSS, and ultrasound descriptions (model, mode, frequencies of transducers).

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses flow diagram of study selection method. ARFI, acoustic radiation force impulse; Echo, echogenicity; MRI, magnetic resonance imaging; mRSS, modified Rodnan skin score; SWE, shear wave elastography; US, ultrasound.

RESULTS

In total, there were 14 studies and 1,055 subjects. The subjects were divided as 43% limited SSc, 24% diffuse SSc, and 33% control. The sum of the total men (150) plus the total women (746) was not equal to the total of subjects (1,055). This was because, for some studies, the information by gender was not reported. The total mean age was 39.9 (30.59) years. The mean age of the dcSSc group (56.6 [37.51] years) was higher than the lcSSc group (37.3 [28.65] years) and the control group (30.7 [27.22] years). The total mean disease duration was 4.3 (8.55) years. The mean disease duration of the dcSSc group (13.3 [15.22]) years and was higher than the lcSSc group (3.8 [8.14]) years and the control group (0.9 [0.50]) years.

Table 1 portrays the nine different OMERACT Filter criteria tested, including whether the validity criterion was met (outcome), a brief description of the criterion, and the evidence and reasoning for the outcome chosen, respectively column by column. Based on our analysis, six of the nine criteria were met (+): face, criterion, content, construct, reliability, responsiveness; one was partially met: feasibility. The remaining two criteria were not met (−): discrimination and reproducibility.

Face validity was met because there was agreement among the authors of the studies that the ultrasound was a logical method to be used in measuring the skin thickness of patients with SSc.

Criterion validity was met and demonstrated by Suliman et al in a visualization of skin layers (epidermis and dermis) using ultrasound (15). They biopsied the skin of three healthy volunteers and scanned the resulting wound/ulcer. The biopsied skin samples were then stained by Haematoxylin and Eosin staining, illustrating the layers (epidermis and dermis) of the skin. A General Electric Health (Systems Logic E9) ultra‐sound apparatus with a variable frequency linear probe (8‐18 MHz) was used. The grey scale was set for superficial structures. The probe was utilized to scan the skin on days 4 and 15 after the biopsy. Four days after the biopsy, the ultrasound scan demonstrated the distinct absence of epidermis and dermis corresponding to the same layers present on H&E sections. By day 14, a scab was well demarcated from regenerated skin beneath, and both the epidermis and dermis were demonstrated by the ultrasound (see Figure 2). In addition, Chen et al supported the criterion validity by Suliman for histological skin thickness and local mRSS (18).

Figure 2

(A) Picture of biopsied skin covered by scab, (B) ultrasound (15 mHZ) picture of the scab covering the normal healed epidermis, (C) ultrasound picture (18 mHZ) of the scab covering the healed epidermis. Data from: Suliman et al (2018) (Reference 15).

Content validity was demonstrated in Supplemental Table 1, wherein demographic data of patients were included to show the large range of patients in whom the technique had been used.

Construct validity was demonstrated in the literature by showing a positive correlation between mRSS and ultrasound (correlation coefficient, 0.48, P < 0.001) (Table 1) (10). Hesselstrand et aldemonstrated that ultrasound measurements correlated with the local mRSS from the corresponding anatomical region and also with the total mRSS (n = 88; rS = 0.55; P = 0.001) (7). Sedky et al showed that ultrasound measurements correlated with the mRSS as well as the severity score of the disease (r = 0.470, P  =  0.002) (9). Furthermore, subclinical dermal involvement was detected by ultrasound even in the skin areas in patients with lcSSc who had a normal local mRSS (r  =  0.37, P  =  0.04) (12). Correlation was also shown in the skin of patients with dcSSc, where thickness increased as echogenicity changed from isoechoic to hypoechoic to hyperechoic (P < 0.001) (11).

Reliability by ultrasound can be validated by documenting the degree of reproducibility within a short period of time (Table 1). Åkeson et al demonstrated low variability at four different anatomical sites: hand, forearm, leg, and chest (intraclass correlation coefficient [ICC]: 0.83‐0.88). The phalanges were an exception, with a lower ICC of 0.66 (4). In addition, other studies showed reliability on two separate occasions for control subjects (standard deviation: 0.06 mm) (2), on four occasions for one subject (coefficient of variation = 2.7%; r = 0.98; P < 0.001) (13), and at 17 different sites (ICC: 0.65‐0.94 and 0.55‐0.96) (3). Ihn et al mentioned that “the skin thickness measurement determined by ultrasound was highly reproducible and one in which there was little variability between observers,” with no specific data given (14). Sulli et al found ultrasound of the skin to be reliable, finding that the intraoperator reproducibility was 96% (95% confidence interval, 0.94‐0.97) (12).

Hesselstrand et al demonstrated responsiveness with a significant decrease in total skin thickness after 1 year of observation (change: −0.22 [−0.79 to 0.30], P = 0.011) (10) (Table 1). Furthermore, Hashikabe et al showed responsiveness by dermal echo intensity after photo chemotherapy. Dermal echo intensity (33.51 ± 9.34) increased versus before therapy (21.23 ± 6.00 [before] to 33.51 ± 9.35 [after], P < 0.01). Also, dermal thickness (1.20 ± 0.20) significantly decreased versus before therapy (1.38 ± 0.18, P < 0.05) (5) (Table 1).

Within clinical trials, feasibility describes the ability to utilize the technique in various settings and with multiple individuals as has been shown among the 14 studies in this SLR. Furthermore, nine different ultrasound machines, eight different probes, and nine different skin‐scoring methods and combinations were used (Supplemental Table 2). Unfortunately, discrimination between treatments and reproducibility of ultrasound have not been tested.

DISCUSSION

This article's value is in our attempt to ascertain the status of validation of skin ultrasound for SSc. This has not been done previously. We found that only six of nine validity criteria have been met: face, content, criterion, construct, reliability, and responsiveness, whereas feasibility as a criterion was partially met. Reproducibility and discrimination criteria were not met (see Table 1).

It should also be noted that the quality of the data ranged between 2.2 and 2.6 (EPHPP), making these data of moderate quality (Supplemental Table 3).

Face validity examines the logic of the method being tested and is adequate, en face.

Content validity examines whether a large and sufficient range of patients were tested, and this aspect of validity was also met (Supplemental Table 1).

Criterion validity is frequently difficult to establish. No articles that detailed criterion validity surfaced in the SLR. However, Suliman et al showed that the biopsied skin of three healthy individuals demonstrated that ultrasound of the skin directly reflected histological changes in the skin (15). Furthermore, criterion validity was established using healthy or normal patients but could have been established in patients with SSc. However, the ability to show a change in the epithelium layer using ultrasound is independent of whether the patient is healthy or diseased. Therefore, criterion validity was fulfilled.

Construct validity was shown by Hesselstrand et al (7, 10) and Sedky et al (9). Furthermore, subclinical dermal involvement was detected by ultrasound even in skin areas in patients with lcSSc who had a normal local mRSS (r = 0.37, P = 0.04) (12). This was extended to patients with dcSSc, where skin thickness increased as echogenicity changed from isoechoic to hypoechoic to hyperechoic (P < 0.001) (11).

Tinazzi et al performed a study with shock wave therapy and seemed to show a relationship between extracorporeal shock wave therapy (ESWT) and mRSS (P < 0.001); however, no correlation coefficient was given, the appropriateness of the statistical analysis used (the Student t test) was unclear, and the mRSS change before and after ESWT of less than 1 (a change well below the Minimally Clinically Important Difference) led us to remove this article.

Akkesson et al provided reliability data for ultrasound measurements of the skin at five different sites (ICC: 0.92‐0.98) (4). In addition, reliability was shown on two separate occasions for control subjects (2) and in a large number of skin sites in one patient with SSc (3, 13). Sulli et al supported these results (12), as did Ihn (14), although the latter was in a statement not supported by data: “The skin thickness measurement determined by ultrasound was highly reproducible and there was little variability between observers” (14). Among the studies, the ICC values were different, so the reliability measurement varied, but the overall result was that these ICCs demonstrated reliability (Table 1).

Responsiveness was shown by Hesselstrand et al, showing that the skin can change over time using ultrasound of the skin (change: −0.22; P = 0.011). Furthermore, Hashikabe et al showed responsiveness with increasing dermal echo intensity and decreased dermal thickness after photo chemotherapy (5).

Feasibility generally refers to the ease and uniformity of the measurement, specifically documenting the time it takes to do the ultrasound measurement in a practical manner. It should include considerations of cost (ie, it should be inexpensive) and simplicity (ie, it should be easy to use). Although no article actually examined the time to conduct a study, there are data supporting the feasibility of ultrasound of the skin for SSc. Fourteen centers were successful in performing SSc skin ultrasound, demonstrating feasibility in a very practical sense. The specialized skills and type of machines, probes, and centers to conduct clinical trials are adequate. However, if this technique is to be widely used, feasibility must be examined in clinical practice, and the fact is that the published data used nine different machines, nine different probes, and eight different locations and combinations, clearly indicating a need to establish uniform technical criteria and showing that feasibility is not fully established in that context.

Unfortunately, no studies have used ultrasound of the skin to discriminate among treatments. Also, whether ultrasound of the skin can be reproduced over a short duration of time was not tested. These remain to be studied.

These data have some limitations. The inevitable limitations of an SLR include the following: (1) lack of individual patient data, (2) methodology changes over many years (30 years in this case), and (3) incomplete descriptions of methodology. More specific to ultrasound, there was heterogeneity among articles in definitions of skin thickness, machines, probes, anatomical sites, and its correlates. In fact, the heterogeneity (I 2) was greater than 0.98, so no meta‐analysis was justifiable. In addition, the studies did not specifically detail the stage of scleroderma in which the ultrasound was used. It is possible that the results during the early edematous phase may be different than in the later fibrotic or inflammatory phase. This is an area for future research.

Studies are needed to establish uniformity regarding the technical aspects of ultrasound (eg, probes to be used). Based on the frequency of the use of probes of up to 18 MHz and the fact that this is the range of MHz that usually comes with most standard probes (thus mitigating cost), we recommend 18‐ to 20‐MHz probes for ultrasound skin scans (Table 2). We realize that this is purely arbitrary, but it seems reasonable.

Table 2

Different studies assessing skin thickness used different probes and varying measurement method

Author, Year∧Reference Number	Definition of Measured Skin Thickness	Frequency (MHz)
Åkeson A, 1986∧2	Beginning echo skin surface and at beginning echo underlying bone	10
Myers SL, 1986∧13	Plane of the reticular dermis and subcutaneous pad interface	25
Ihn H, 1995∧14	Skin surface and skin‐fat interface; full skin (epidermis and dermis)	30
Hesselstrand R, 2002∧1	Interfaces between the epidermis, dermis, and subcutis	20
Moore TL, 2003∧3	Epidermis and dermis, separately	22
Åkeson A, 2004∧4	Epidermis plus dermis	20
Hashikabe M, 2005∧5	Dermo‐epidermal junction to the boundary of dermis and subcutaneous fatty tissue	20
Kissin E, 2006∧6	—	10
Hesselstrand R, 2008∧7	Interfaces between the epidermis, dermis, and subcutis	20
Tinazzi E, 2011∧8	—	12
Sedky MM, 2013∧9	Epidermis plus dermis	5‐12
Hesselstrand R, 2015∧10	Interfaces between the epidermis, dermis, and subcutis	20
Liu H, 2017∧11	Epidermis and dermis combined	4‐9
Sulli A, 2017∧12	The upper surface epidermis‐dermis and the lower layer dermis‐subcutis; dermal thickness	18

Abbreviation: —, not available.

Ultrasound machines should be as simple and inexpensive as possible, but we would urge the ability to record results and to be flexible enough for multiple uses (eg, skin, joints, lungs). We would also urge continued standardization of teaching and certification of the skills to perform ultrasound.

In conclusion, based on an SLR through December 31, 2018, ultrasound of the skin for examining skin thickness in SSc met six out of nine validation criteria for use in multicenter clinical trials. These include face, content, criterion, construct, reliability, and responsiveness. Feasibility was generally validated for clinical trials on a practical basis but is not formally validated and is not validated in clinical practice; thus, it is considered partially validated. Specifically, discrimination and reproducibility criteria were not met. The validation of ultrasound of the skin requires further research if it is to be used as a fully validated measure in clinical trials.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual contact, and all authors approved the final version to be published.

Study conception and design

Kafaja, Aly, Furst, Suliman.

Acquisition of data

Kafaja, Aly.

Analysis and interpretation of data

Kafaja, Aly, Furst, Suliman.

Supplemental Table 1 Demographic data of patients within studies (content validity)

Supplemental Table 2. Study design and technical aspects

Supplemental Table 3. EPHPP Quality assessment

Click here for additional data file.