Efficacy Endpoints in Clinical Trials in Actinic Keratosis.

INTRODUCTION: Actinic keratosis is regarded as a chronic disease of the skin and, although fluctuating, is chronically progressive. Approval of new products for the treatment of actinic keratosis requires the use of a standard methodology in clinical trials which emphasize complete clearance of all actinic keratoses in a treatment field in a defined time span and the evaluation of long-term efficacy in terms of recurrence rate among completely cleared patients.
METHODS: Analysis of data from six previously published clinical trials in patients with actinic keratosis.
RESULTS: There was poor agreement over a period of 1 month in the complete clearance endpoint. This variation in assessment renders recurrence in cleared patients invalid as the estimate of long-term efficacy. Furthermore, complete clearance was shown to depend heavily on the number of baseline actinic keratoses.
CONCLUSION: The main endpoints presently in use for the assessment of short- and long-term efficacy of actinic keratosis field-directed therapy, namely, complete clearance and recurrence rate, are obsolete and should be replaced by the percentage reduction in actinic keratosis count or the absolute actinic keratosis count. FUNDING: LEO Pharma A/S.

Introduction

Actinic keratosis (AK) is regarded as a chronic disease of the skin and, although fluctuating, is chronically progressive [1, 2]. Indeed, although single AK clearance is common, field clearance is not [1, 2]. This is clearly illustrated in Criscione et al.’s study [3] in which 1960 AKs present at baseline were individually tracked in 169 study participants. Although the patients had a total of 7784 AKs during the 5-year study period, the proportion of baseline AKs that were still present at study year 5 had dropped to 30%—without treatment. Among the 30% of baseline AKs found at the year 5 medical examination, 92% had been absent at one or more examinations during the previous 4 years. Werner et al. have summarized other literature [1] on spontaneous regression rates of single AK lesions and found that these range from 15 to 65% after 1 year.

As a result of the strict requirements for the approval of new products for the treatment of AK, a standard methodology has evolved for conducting well-controlled studies in AK which emphasize the complete clearance of all AKs in a treatment field within a defined time span. This efficacy endpoint is, however, not well suited for a disease with a fluctuating course, whereas other endpoints such as the percentage reduction and absolute AK count are more appropriate. In this article we use data from published clinical trials to substantiate this point.

Methods

Data from the following trials were used:

Study 1: A seamless phase 1/2 trial with ingenol mebutate gel in patients with AK on the face, chest, or scalp [4]. A total of 313 patients were randomized 1:1:1:1:1 to treatment with ingenol mebutate gel 0.018% or 0.037% for 2 or 3 days or to vehicle. Each patient had 5–20 clinically typical, visible, and discrete AKs in the treatment area. The treatment area was full face or full balding scalp or a 250-cm2 area on the chest. AKs were counted 4 and 8 weeks after start of treatment. For further details see Hanke et al. [4].

Study 2: A phase 3 trial in which 729 patients, with the same characteristics as in Study 1, were randomized 3:1 to ingenol mebutate gel 0.027% or to vehicle. For further details see Hanke et al. [5].

Studies 3–6: Four phase 3 trials in which 547 patients were randomized 1:1 for ingenol mebutate gel 0.015% or to vehicle. Each patient had four to eight AKs in a treatment area of 25-cm2 on the face or scalp. AKs were counted 8 weeks after start of treatment. For details see Lebwohl et al. [6].

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.

Using the data from these studies we were able to analyze the fluctuation of efficacy endpoints over time and the dependency of efficacy endpoints on the baseline AK count.

Data are presented descriptively. The mean percentage reduction (hereafter % reduction) was computed by first computing the individual patient’s percentage reduction and then taking the mean over patients in the treatment group.

Complete clearance (defined as no clinically visible AKs in the treatment area) at week 4 and week 8 was analyzed in a 2 × 2 table with the following format:

The following agreement statistics were computed [7]:

The 95% confidence intervals (CIs) for the agreement statistics were bootstrapped.

Results

The Effect of Baseline AK Counts on Efficacy Endpoints over Time

Actinic keratoses were counted at day 1 (baseline) and at weeks 4 and 8 in pooled Studies 1 and 2. The histograms of the AK counts at these visits are shown in Fig. 1. From day 1 to week 4, the AK count in the vehicle group can be seen to shift to the left and the mean number of AKs to fall by 16.2%. This “vehicle effect” is probably a regression to the mean effect: patients with fewer than five AKs were excluded from inclusion in the study. If the disease varies, the variation in itself will lead to a flattening of the histogram. This interpretation is supported by the observation that the histogram does not change from week 4 to week 8; the mean is unchanged and the distribution has the same shape. Thus, in the course of 1 month there is no measurable overall disease progression in the vehicle group.

Fig. 1

Histograms of actinic keratosis (AK) counts before treatment (day 1) and at weeks 4 and 8 in pooled Studies 1 and 2. CI Confidence interval. Part of bars in black indicates percentage of patients who were completely cleared at week 4

Although in the ingenol mebutate gel population efficacy as assessed by % reduction in the AK count is around 74% at weeks 4 and 8, the proportion of patients who have no AKs, i.e., complete clearance, is around 24%. The histograms clearly show that completely cleared patients (the 0 column) are not the only ones who benefit from the treatment: the whole distribution is shifted to the left, indicating a reduction in the AK counts.

Furthermore, the efficacy endpoint of around 74% reduction in AK count in the ingenol mebutate treatment group at weeks 4 and 8 is obviously not simply due to “flattening” of the distribution, which only resulted in an approximately 16% reduction in the vehicle group.

In Fig. 1 the net changes in % reduction and complete clearance in the AK counts between weeks 4 and 8 are small. However, even though there is no net change, patients can move between the categories. The shaded areas in Fig. 1 illustrate this shifting: the percentage of patients who were completely cleared at week 4 are shown in black in all figure parts. Around 60% of the ingenol mebutate-treated patients who were completely clear at week 4 (black 0 column at week 4) are to be found in the 0 column at week 8. The remaining 40% are spread in the non-completely cleared columns at week 8. The black part of the 0-column at week 8 represents patients who had a zero count at both week 4 and week 8.

The complete clearance data for the ingenol mebutate group for weeks 4 and 8 are shown in Table 1. The overall agreement is 0.80 (95% CI 0.78–0.84), the positive agreement is 0.59 (95% CI 0.55–0.67), and the negative agreement is 0.87 (95% CI 0.86–0.90).

Table 1

Complete clearance at weeks 4 and 8, ingenol mebutate gel group

		Complete clearance at week 8
		Yes	No
Complete clearance at week 4
Yes		116	77
No		70	520

A scatterplot of the counts at the two visits (weeks 4 and 8) is shown in Fig. 2. The size of the dots reflects the number of patients. It is clear from the figures that the paired counts are associated, but it is also clear that patients do not always stay in their category. If they did, all dots would be on the identity line. This applies to both vehicle and ingenol mebutate gel patients.

Fig. 2

Actinic keratosis count at week 4 by AK count at week 8. The size of the dots reflects the number of patients. Left panel Vehicle gel, right panel ingenol mebutate gel

Dependency of the Complete Clearance Rate on the Baseline AK Number

As has been reported by Szeimies et al. [8] and Dirschka et al. [9]. complete clearance depends among other things on the number of baseline AKs and the size of the treatment area. The aggregated data for ingenol mebutate treatment on the face and scalp on small and large areas is given in Table 2; the baseline AK counts range from 4 to 20. The % reduction in AK counts is virtually unaffected by the number of baseline AKs, the complete clearance rate falls with increasing baseline AK count, and the absolute difference in AK count increases. With four baseline AKs the complete clearance rate is around 45%, and the number of AKs removed by the treatment is approximately three. At the other end of the spectrum, 14 AKs are removed by the treatment and the complete clearance rate is around 5%. Thus, paradoxically, the clinical benefit for the patient increases as the complete clearance decreases. The same data broken down by each baseline count from 4 to 20 are shown in Fig. 3, and the findings are remarkably consistent with those given in Table 2.

Table 2

Percentage reduction in AK count since baseline, complete clearance, and absolute reduction in AK count since baseline

Number of baseline AKs	N	Mean percentage reduction at week 8a	Mean complete clearance at week 8a	Mean difference in AK count at week 8a
4–8 baseline AKs	532	74.3	40.9	4.7
9–12 baseline AKs	245	70.2	22.3	7.3
13–20 baseline AKs	311	73.9	14.8	12.2

AKs Actinic keratoses

aData presented in table are pooled data from Studies 1–6, ingenol mebutate treatment groups

Fig. 3

Percentage reduction in AK count since baseline, complete clearance, and absolute reduction in AK count since baseline by baseline AK count. Pooled data from Studies 1–6, ingenol mebutate treatment groups

The Recurrence Endpoint Is Inaccurate and Should Be Replaced with % Reduction from Baseline

The standard for estimating the long-term effect after a treatment is completed is based on patients who obtained complete clearance at the end of treatment. This means that only those who fail (get a recurrence) are taken into account, whereas those who were not clear at the end of the treatment but would have subsequently cleared are ignored. In the example investigated in this article (Fig. 1, Table 1), if the end of treatment were to have been at week 4, the patients who were cleared at week 8 but not at the assessment at the end of treatment at week 4 would not have been included, and a recurrence rate of 77/(116 + 77) = 40% would have been computed at week 8 when in reality the proportion of patients who are completely clear at the two visits is almost identical: (116 + 77)/(116 + 77 + 70 + 520) = 25% at week 4 and (116 + 70)/(116 + 77 + 70 + 520) = 24% at week 8.

In addition, with subsequent repeat assessments over time of only those who were completely cleared at the previous visit, the recurrence rate will be further inflated.

A more appropriate assessment of long-term efficacy would include AK counts for all patients at all visits, whether cleared or not at the end of treatment efficacy assessment, preferably presented as the percentage reduction relative to the baseline AK count.

Discussion

We have tabulated data from several clinical trials with ingenol mebutate with the aim to show that there is low agreement over a 1-month period in the endpoint complete clearance. This variation renders recurrence in cleared patients invalid as the estimate of long-term efficacy. Furthermore, complete clearance depends heavily on the number of baseline AKs.

In this evaluation, the starting point is that AK is a disease with a variable presentation. Looking at the variation in clearance between weeks 4 and 8, we suggest that an alternative interpretation would be that there is a carry-over effect of the treatment in some patients and recurrences of AK lesions in other patients, and that these two effects cancel out. This interpretation is, however, contradicted by the observation that there is no net increase in the number of AKs in the vehicle group. Thus, there are not many new AKs appearing during a 1-month period. The interpretation would rely on baseline AKs coming back very quickly after being completely cleared in some patients, while in other patients the clearing of AKs continues. The theory could be tested in a trial in which individual AKs are tracked over time without treatment in the observation period; this would enable recurrent and not-yet-cleared AK lesions to be distinguished.

What has led to the general acceptance of complete clearance as an endpoint in AK trials? No regulatory guidelines exist in which a rationale has been presented. One might speculate that the “progression to cancer” aspect of the disease has played a role. Regulatory agencies have taken the position that it could be the last AK remaining after a treatment that progressed to cancer and, therefore, that the relevant endpoint in a clinical trial would be the complete eradication of all AKs. Although this notion is compelling, it does not fit well with the chronic course of the disease. Even if all AKs are cleared on a given day, new AKs will develop over time. Furthermore, as shown in the present analysis, due to the fluctuating presentation of the disease, completely cleared patients are likely to be non-cleared even in the short term.

The risk of progression of AKs to squamous cell carcinoma (SCC) has been quantified in a series of studies by different authors. The most comprehensive of these was published by Marks et al. who showed that the “rate of malignant transformation of a solar keratosis to SCC within 1 year was 0.075%” [10]. Noting that some estimates of clinical progression rates have ranged up to 25%, Dodson et al. [11]. applied simple probability calculations to Marks et al.’s estimate of the per AK and year risk to hypothetical patients and showed that although the conversion risk for a single AK in a single year is low, the cumulative risk experienced by a patient with several AK lesions over several years may be substantial. Helfand et al. [12] further expanded on this work to show that the rates of conversion to SCC of 14% during 5 years of follow-up observed by Moon et al. [13] are compatible with Marks et al.’s estimate of the progression rate. Underlying all this work is the assumption that the risk of SCC increases with the number of AKs and with observation time; more specifically, that the risk of SCC progression for one AK is independent of other AKs in the same patient and that the risk is constant over time. Although these assumptions may seem crude, the data fit the assumptions well. Thus, there is reason to believe that the risk for progression to SCC is related to the number of AKs. To our knowledge, there are no data to support the assumption that it is the last AK left after a treatment that progresses to cancer.

The nature of the disease thus seems to be such that on the one hand a substantial proportion of AKs spontaneously regress if left untreated and that on the other hand there is a risk of progression to SCC if AKs are left untreated. This may seem paradoxical, but there is no logical contradiction between these two aspects of the disease, and indeed they have been reconciled in the theoretical framework of “field cancerization”. AK is now regarded as a chronic disease of the skin with a fluctuating but chronically progressive course. Although single AK clearance is common, field clearance is not [1, 2].

Limitations to the study

A major limitation/shortcoming of the current study is that our evaluation includes findings after treatment with only one topical product. Whereas there is prima fascia no reason to believe that the findings would be different after treatment with other topicals, this assumption should definitely be tested. Conversely, the use of data from more clinical trials with the same product (here ingenol mebutate) and the same duration of treatment is likely to increase the consistency and robustness of the findings.

The tabulations reported in this paper were not pre-specified in the protocols of the studies, and the findings were unexpected. In order to obtain as much precision as possible we pooled the data from Study 1 (313 patients) and Study 2 (729 patients); the findings were similar for each study separately.

Could observer variation explain the low agreement of complete clearance at the two assessments at week 4 and week 8? The two assessments were made by the same observer at both occasions, thereby reducing intra-observer variation (whether the observer was always consistent in the assessment of the presence or absence of any AK in the treated field). This would seem a manageable task. In principle, intra-observer variation can be assessed by having observers repeat their assessment of the same patient after a lag-time when the previous assessment is assumed to be forgotten. This is in effect what we did here—except that during the time between the assessments, the object of the assessment, the treatment field, and the presentation of the AKs changed. It is in fact impossible under these circumstances to isolate the intra-observer variation from the variation in the disease presentation.

Tracking of individual AKs throughout the follow-up period would have been better than just reporting the total AK count in the treatment area. It should be noted, however, that tracking a large number of individual AKs in a treatment area in itself may be subject to observer variation. A future study with meticulous tracking using state-of-the-art imaging methods may overcome this problem. Finally, the reported studies included only clinically typical, non-hypertrophic and non-hyperkeratotic AKs. It would be helpful to know whether the reported variation is the same in severe and less severe AK. If variation in presentation is more pronounced for low-grade AKs, one would see the paradoxical effect that studies of less severe AKs would show the highest recurrence rates.

Conclusions

Complete clearance as an endpoint is not useful: firstly, because it loses information compared to the AK count or the % reduction; secondly, because it is unsuited for any comparison of efficacy across trials with different baseline AK counts and different treatment areas (the % reduction is obviously much more robust for this purpose); thirdly, because it invites researchers to study long-term efficacy with a flawed endpoint, namely, recurrence after complete clearance. The main endpoints presently in use for the assessment of short- and long-term efficacy of AK field-directed therapy, i.e., complete clearance and recurrence rate, are obsolete and should be replaced by the % reduction in AK count or the absolute AK count.

	Complete clearance week 8
	Yes	No
Complete clearance week 4
Yes	a	b
No	c	d