Both Educational Lectures and Reference Photographs Are Necessary to Improve the Accuracy and Reliability of Psoriasis Area and Severity Index (PASI) Assessment: Results from Korean Nation-Wide PASI Educational Workshop.

BACKGROUND: Accurate assessment of the severity of psoriasis is important in daily practice and clinical studies. However, the assessment of psoriasis area and severity index (PASI) reflects the physician's experience, and thus evaluations by physicians are inherently subjective, with intra-rater and inter-rater variability.
OBJECTIVE: To elucidate the effectiveness of PASI educational lectures and the use of reference photographs on the improvement of accuracy and reliability in PASI assessments and to develop effective educational programs for PASI assessments.
METHODS: We performed a before-and-after comparison study during nation-wide PASI educational workshops. The participants were asked to assess the severity components of PASI (erythema, thickness, scale, and affected area) three times: in the test administered before an educational lecture, the test immediately after the lecture, and lastly the test with the use of reference photographs. The improvement of accuracy and reliability was analyzed by comparing the results of three tests.
RESULTS: Ninety-six board-certified dermatologists and residents participated and 72 participants completed all three tests. The accuracy and reliability of the assessment of severity components of PASI increased significantly after the educational lecture and the use of reference photographs. Use of reference photographs resulted in limited improvements when the recognition of three-dimensional structures was required, such as in the assessment of thickness or scale.
CONCLUSION: Our study confirmed that the combination of standardized educational training and reference photographs can improve the accuracy and reliability of PASI assessments. Understanding how to evaluate three-dimensional psoriatic lesions can help with proper assessment of the severity of psoriasis.

INTRODUCTION

Psoriasis is a chronic inflammatory disorder of the skin that frequently recurs after treatment. Psoriatic lesions in exposed skin can pose a significant psychological burden for patients, hence, the need for effective psoriasis treatment is imperative1. Various treatment modalities have been introduced and physicians have to choose the appropriate management modalities based on the status and severity of the disorder for each patient. Therefore, accurate assessment of the status or severity of psoriasis is important2. Moreover, the need for accurate evaluation has increased after the recent introduction of new treatment modalities to enable the objective and reliable assessment of different cases3. Among the various methods available for assessing psoriasis severity, the psoriasis area and severity index (PASI) is well validated and most commonly used24. However, the PASI method has some limitations in evaluating the four components of PASI (erythema, thickness, scale, and affected area), subjective assessments by evaluators can have intra-rater and inter-rater variability23.

To attain substantial accuracy and reliability in evaluating the disease severity, educational training for health-care providers and patients has been introduced2567. It was reported that educational training for laypersons can improve their ability to detect skin cancers67, and online PASI training improved the accuracy of PASI assessments by psoriatic patients and health-care providers2. However, a previous study also reported a limitation of educational training, namely that it is difficult to standardize the conventional educational program2. Moreover, it was reported that the conventional lecture consisting of an explanation of the ABCD algorithm for early detection of melanoma was less effective than photography-oriented cognitive education7.

In this study, we enrolled board-certified dermatologists and residents in Korea who attended a nation-wide PASI educational workshop to elucidate the effectiveness of PASI educational lectures and reference photographs on the accuracy and reliability of PASI assessments. On the basis of the results, we suggest some factors that should be taken into account to develop effective educational lectures and image-based training programs for accurate and reliable PASI assessments.

MATERIALS AND METHODS

Study design and enrolled subjects

PASI educational lectures and tests for assessing the severity components of PASI (erythema, thickness, scale, and affected area) were offered during a nation-wide PASI educational workshop. The nation-wide PASI educational workshop is held four times in South Korea. The board-certified dermatologists and residents who attended the workshop were enrolled. The study was approved by the institutional review board of Seoul National University Bundang Hospital (IRB no. X-1608/360-902).

PASI educational lecture and test description

The PASI educational lecture consisted of an explanation of the general algorithm, the four components (erythema, thickness, scale, and affected area) of the PASI system, and how to grade the severity of the four PASI components. Because previous studies reported a high degree of variability in determining the degree of the body affected58, the lecture introduced some practical methods for estimating the affected area. The participants were shown images of psoriasis on a part of patient's body and were asked to assess the severity components of PASI (erythema, thickness, scale, and affected area): the degree of severity (erythema, thickness, and scale) and the percentage of the area affected. The effect of the PASI educational lecture was evaluated by comparing the results of tests administered before (test A) and immediately after the lecture (test B). Lastly, after short instructions for reference photographs, participants assessed the degree of severity and the affected surface area while using reference photographs to evaluate the usefulness of reference photographs in the assessment of PASI scoring (test C: test with reference photographs).

The test was developed by a team of dermatologists through consultations with medical statisticians. The test consisted of five images of psoriatic lesions representing small or large plaque psoriasis with variable degrees of severity. In each test (test A, test B, and test C), five images were shown twice in a random sequence and the participants were asked to assess the severity components of PASI (erythema, thickness, scale, and affected area): the degree of severity (erythema, thickness, and scale) and the percentage of the area affected of the images. The scores assigned by PASI expert who participated in developing the test were regarded as the standard and were compared with those of the participants. When the assessment of erythema, thickness, and scale was the same as that of the expert, the assessment was regarded as correct. In the assessment of the affected area, we regarded the participants' assessments as correct when the difference between the scores assigned by the participants and those assigned by the expert was 5 or less. To ensure a uniform test environment, all the participants completed the tests using iPads (Apple Inc., Cupertino, CA, USA) with the same brightness and contrast.

Statistical analysis

To evaluate the effectiveness of the PASI educational lecture and the reference photographs on the accuracy and reliability of PASI scoring, we conducted a before-and-after comparison study by comparing the assessments of four components of the PASI system. The ability of participants to assess the severity of psoriatic lesions was evaluated three times: in a test before the PASI educational lecture (test A), a test immediately after the lecture (test B), and a test using reference photographs (test C). The effectiveness of the PASI educational lecture and the reference photographs was analyzed for two parameters: the improvement of accuracy and the reduction of intra-rater and inter-rater variability. To analyze the improvement of the accuracy of PASI assessment, we counted the number of correct assessments out of 10 assessments in each test: the number of correct assessments is between 0 and 10 and if the participant correctly assessed four out of ten assessments, the number of correct assessments is four. Then, we calculated the average of the number of correct assessments of each test and compared them using the Wilcoxon signed rank test. For the evaluation of intra-rater variability, the differences in the assessment of the affected area, erythema, thickness, and scale between the same images presented twice in each test was calculated, then the change in the intra-rater variability was analyzed by comparing the differences in the assessment of the same image using the generalized estimating equation (GEE) model to correct the repeated test during the PASI educational workshop. To analyze the inter-rater reliability, we defined the deviation by comparing individual assessments with the mean assessment for all participants in each test and calculated the deviation of each test as the absolute value of [the participant's assessment-the mean of all participants' assessments]. Then the deviations were compared using the Wilcoxon signed rank test. IBM SPSS Statistics ver. 22.0 (IBM Co., Armonk, NY, USA) was used to perform statistical analysis and a p-value <0.05 was considered statistically significant.

RESULTS

Characteristics of participants

A total of 96 board-certified dermatologists and residents participated in the study. Among the 96 participants, 38 (39.6%) were board-certified dermatologists and 55 (57.3%) were residents. Among them, 76 participants had previous experience with PASI assessment. After removing tests with missing data, 72 participants completed all three tests without missing any assessments.

Accuracy of PASI assessment improved by the PASI educational lecture and reference photographs

To analyze the effects of the PASI educational lecture and the use of reference photographs on the accuracy of PASI assessment, we counted the number of correct assessments in each test (Table 1). The PASI educational lecture increased the accuracy for all components of PASI; in the assessment of erythema, thickness, and scale, the accuracy significantly increased (p=0.001 for erythema, p<0.001 for thickness, and p=0.002 for scale). Moreover, further improvements were noted in the assessment of the affected area and erythema when the participants used the reference photographs (p=0.013 for the affected area and p<0.001 for erythema). With the PASI educational lecture and the use of reference photographs, the accuracy of assessment increased for all PASI components except the scale (p=0.010 for the affected area, p<0.001 for erythema, p=0.025 for thickness). In contrast, the accuracy of the assessment of scale decreased when the reference photographs was used (p<0.001).

Table 1

The accuracy in severity assessment analyzed by comparing the number of correct assessments

	Number of correct assessments (mean±standard deviation)			Difference of correct assessments between the tests (estimate, 95% confidence interval)			p-value
	Test A	Test B	Test C	Test A vs. test B	Test B vs. test C	Test A vs. test C	Test A vs. test B	Test B vs. test C	Test A vs. test C
Area	4.74±2.32	5.03±1.77	5.54±2.17	−0.292±2.58 (−0.897 to 0.313)	−0.514±1.71 (−0.916 to −0.112)	−0.806±2.60 (−1.42 to −0.195)	0.340	0.013	0.010
Erythema	3.54±1.65	4.35±1.65	5.17±1.91	−0.806±2.03 (−1.28 to −0.328)	−0.819±1.60 (−1.20 to −0.443)	−1.63±2.20 (−2.14 to −1.11)	0.001	<0.001	<0.001
Thickness	4.71±1.96	5.71±2.26	5.38±2.15	−1.00±2.31 (−1.54 to −0.456)	0.333±1.99 (−0.135 to 0.802)	−0.667±2.48 (−1.25 to −0.084)	<0.001	0.160	0.025
Scale	6.31±2.04	7.18±2.40	6.28±2.78	−0.875±2.28 (−1.14 to −0.339)	0.903±1.67 (0.510 to 1.30)	0.028±2.63 (−0.591 to 0.646)	0.002	<0.001	0.929

Test A: test before psoriasis area and severity index educational lecture, test B: test immediately after lecture, test C: test using reference photographs. We counted the number of correct assessments out of 10 assessments in each test and calculated the average of correct assessments of the participants and compared them using the Wilcoxon signed rank test.

Intra-rater and inter-rater variability in PASI assessment decreased after PASI educational lecture and reference photographs

We analyzed the effect of the PASI educational lecture and the reference photographs on the intra-rater reliability of PASI assessments using the GEE model (Table 2). Analysis of the GEE model, adjusted for the five images used in the test, revealed that the difference in the assessments of the same two images in each test was reduced after the PASI educational lecture and after viewing the reference photographs (p=0.001 for erythema and scale; p<0.001 for the affected area and thickness). After the PASI educational lecture, we observed less difference in the assessment of thickness and scale (p=0.018 for thickness and p=0.001 for scale). Use of the reference photographs reduced the intra-rater variability only in the assessment of the affected area (p<0.001).

Table 2

The intra-rater variability in psoriasis area and severity index (PASI) assessment analyzed based on the difference between the assessment of the same images presented twice in each test

	Test A-test B		Test B-test C		Test A-test C
	Difference in assessments	p-value	Difference in assessments	p-value	Difference in assessments	p-value
Area	−0.125	>0.999	−1.728	<0.001	−1.853	<0.001
Erythema	−0.067	0.183	−0.050	0.447	−0.117	0.001
Thickness	−0.100	0.018	−0.058	0.099	−0.158	<0.001
Scale	−0.122	0.001	<0.001	>0.999	−0.122	0.001

Values are presented as number only. Test A: test before PASI educational lecture, test B: test immediately after lecture, test C: test using reference photographs.

Both the PASI educational lecture and the reference photographs reduced the inter-rater variability of PASI assessments. In the assessment of the affected area and erythema, the Wilcoxon signed rank test showed that the inter-rater variability reduced in test B and test C than in test A (Fig. 1). In the assessment of the affected area, an improvement in the inter-rater variability was observed after the PASI educational lecture and while using the reference photographs (p=0.024 for image1, p=0.003 for image4, and p=0.001 for image5). In addition, the PASI educational lecture reduced the inter-rater variability for image4 (p<0.001) while the use of reference photographs reduced the inter-rater variability for image5 (p=0.013). In the assessment of erythema, the PASI educational lecture and the use of reference photographs reduced the inter-rater variability (p<0.001 for image2, p=0.034 for image3, and p<0.001 for image4). We also found fewer deviations after the lecture (p=0.002 for image2 and p=0.007 for image4) and while using the reference photographs (p<0.001 for image1, p=0.027 for image2, and p<0.001 for image5). However, in the assessment of thickness and scale, there was no uniform change after participation in the PASI educational lecture or while using the reference photographs.

Fig. 1

The inter-rater variability in the assessment of the affected (A) area, (B) erythema, (C) thickness, and (D) scale. The inter-rater variability, determined based on the deviation from the mean assessment for all participants, decreased in the assessment of the affected area and erythema after participation in the psoriasis area and severity index (PASI) educational lecture and while using the reference photographs. Test A: test before PASI educational lecture, test B: test immediately after lecture, test C: test using reference photographs.

*p<0.05.

DISCUSSION

Education and training can improve accuracy and reliability in the assessment of PASI25. In a previous study, we evaluated the effectiveness of PASI training on the accuracy and variability of PASI assessments and found that an in-person training course for PASI and the use of reference photographs increased the accuracy and inter-rater reliability5. However, the number of participants and the images used in the test were small. Therefore, we were unable to elucidate the effectiveness of PASI training and the use of reference photographs or to develop an effective PASI training program and proper reference photographs that can be used in daily practice and clinical studies5. In the present study, we enrolled 96 board-certified dermatologists and residents and developed a new test with the help of medical statisticians to analyze the effectiveness of a PASI training lecture and the use of reference photographs for two parameters: the improvement of accuracy and the reduction of intra-rater and inter-rater variability.

In this study, we determined that a PASI educational lecture and the use of reference photographs improved the accuracy of PASI assessment. The PASI educational lecture, which consisted of an explanation of the general algorithm for the PASI system and a description of how to grade the severity, improved the accuracy of all four components of PASI. With the exception of the assessment of the affected area, the improvements in the other three components were statistically significant. The results support the effectiveness of conventional educational lectures for improving the accuracy of PASI assessments. In contrast, the use of reference photographs improved the accuracy of the assessment of the affected area and erythema. However, there were no significant improvements in the assessments of thickness or scale. In a previous study, we reported that the effectiveness of reference photograph usage was not identical for the four components of PASI, as the improvement in the assessment of erythema was larger than that of thickness or scale5. On the basis of the results, we speculated that the PASI evaluator needs to recognize the three-dimensional structure of psoriatic lesions when assessing thickness or scale (in the psoriatic lesions, scale is attached to skin). Two-dimensional reference photographs provide significant benefits during the assessment of the affected area and erythema, but have limited effectiveness in improving the assessment of thickness and scale.

The recent introduction of new treatment modalities has highlighted the need for reliable methods to compare the effectiveness of treatment234. However, it was reported that the PASI score, subjectively assessed by evaluators, showed substantial variability24. In this study, we developed a test comprised of five images shown twice and found that participation in a PASI educational lecture and the use of reference photographs reduced the intra-rater and inter-rater variability. The intra-rater variability, analyzed by comparing the differences in the assessments of the same images shown twice, decreased after participation in the PASI educational lecture and while using refer ence photographs. The PASI educational lecture improved the reliability of the assessment of thickness and scale, while the use of reference photographs improved the reliability of the assessment of the affected area. In addition, the inter-rater reliability in the assessment of the affected area and erythema was also improved after the lecture and while using the reference photographs. The results confirm the effectiveness of the PASI educational lecture and the use of reference photographs in improving the reliability of PASI assessment.

The results of the study highlighted some factors that require improvement in the development of PASI educational lectures and reference photographs for use in daily practice and clinical studies. The effects of the PASI educational lecture and the use of reference photographs on the four components of PASI were not identical. Both the accuracy and reliability of thickness and scale assessments improved by the PASI educational lecture, while the assessment of the affected area improved by the use of reference photographs. This suggests that the combination of conventional education and image-oriented education can improve the accuracy and reliability of all four components of PASI. In addition, we observed limited effectiveness with two-dimensional reference photographs in the assessment of thickness. This result suggests that education for PASI assessments should include information on how to assess the three-dimensional structure of psoriatic lesions and suggests that three-dimensional photographs with embossing may be more helpful for use in daily practice and clinical studies in assessing the three-dimensional structure of psoriatic lesions.

In this study, we elucidated the effectiveness of an educational lecture and the use of reference photographs for improving the accuracy and reliability of PASI assessments. However, the effectiveness of educational lecture and the use of reference photographs was not identical for the four components of PASI, as the improvement in the assessment of thickness and scale was limited. To improve assessments in daily practice and clinical studies where the assessment of disease severity is essential, a combination of standardized educational training with well-established criteria for assessment and three-dimensional reference photographs with embossing will help to reduce the innate drawbacks of PASI by increasing the accuracy and decreasing subjectivity and variability.