Optimal designs for group randomized trials and group administered treatments with outcomes at the subject and group level.

With group randomized trials complete groups of subject are randomized to treatment conditions. Such grouping also occurs in individually randomized trials where treatment is administered in groups. Outcomes may be measured at the level of the subject, but also at the level of the group. The optimal design determines the number of groups and the number of subjects per group in the intervention and control conditions. It is found by taking a budgetary constraint into account, where costs are associated with implementing the intervention and control, and with taking measurements on subject and groups. The optimal design is found such that the effect of treatment is estimated with highest efficiency, and the total costs do not exceed the budget that is available. The design that is optimal for the outcome at the subject level is not necessarily optimal for the outcome at the group level. Multiple-objective optimal designs consider both outcomes simultaneously. Their aim is to find a design that has high efficiencies for both outcome measures. An Internet application for finding the multiple-objective optimal design is demonstrated on the basis of an example from smoking prevention in primary education, and another example on consultation time in primary care.

RCT Entities:   Population Interventions Outcomes

1 Introduction

With group randomized trials, complete groups, such as school classes, general practices or even neighborhoods are randomized to treatment conditions and all subjects within the same group receive the same treatment. Such designs are often chosen over individual randomization for political, administrative and financial reasons, and to avoid the risk of contamination.[1] This trial design is very common in the health and behavioral sciences, which is emphasized by textbooks[2-6] and special issues of statistical journals that have been devoted to it.[7-9] It also goes under the names cluster randomized trial, community intervention trial and place-based trial in other fields of science.

An important question in the design phase of such a trial is how many groups and how many subjects per group should be enrolled. This question has been addressed in many publications over the past two decades; most of these sought the optimal sample sizes to maximize the efficiency of the treatment effect estimator while taking into account the costs at the group and subject level. Such optimal designs are referred to as single-objective optimal designs since they consider one objective (i.e. maximum efficiency for a single outcome measure). The first publications focused on equal costs and variances over treatment conditions and derived balanced designs with equal number of groups and equal group sizes over the treatments.[10-12] The more realistic scenario with varying group level costs across treatment conditions was considered for trials with a fixed and common group size by Liu[13] and Candel and van Breukelen,[14] where the latter also considered heterogeneous variances. The most generic approach does not fix group sizes a priori and takes heterogeneous costs and variances into account.[15]

The focus of these papers is on a single outcome measure at the subject level; while in group randomized trials outcomes may also be measured at the group level, or at both.[16,17] For instance, in a cognitive behavioral intervention for disruptive classrooms, outcomes were not only measured at the level of the child (e.g. self-esteem, depressed mood and prosocial behavior) but also at the level of the classroom (classroom climate).[18,19] A study in primary care focused on the effect of treatments to improve asthma care. Asthma symptom days per year and the need for oral steroid bursts were measured at the subject level and medication use at the practice level.[20,21]

The optimal number of groups per treatment in the case of a group level outcome follows from Schouten.[22] The optimal design for a group level outcome does not spend any part of the budget on measuring outcomes at the level of the subject; hence the whole budget can be spent on implementing the treatments and taking measurements on group level outcomes. This implies that the optimal design for a group level outcome includes more groups than the optimal design for a subject level outcome. The aim of this paper is to illustrate the use of multiple-objective optimal designs[23] for group randomized trials to take into account outcomes at the subject and group level simultaneously. Two objectives are considered: the efficiencies of the treatment effect estimator for the subject and for the group level outcome. The multiple-objective optimal design provides the optimal number of groups and group sizes in each treatment condition such that the efficiency for the most important outcome is at a user-specified level and the efficiency for the other outcome is maximized.[24] The multiple-objective optimal design is derived analytically and is implemented in a free web application. The focus is on studies that compare two treatments: an intervention and a control, where the latter can be an old treatment or no treatment at all.

The methodology of this paper does not only apply to group randomized trials but also to individually randomized trials where treatment is offered in peer pressure or focus groups.[25-28] The group sizes in such group administered trials are often fixed in advance because such groups need to be small to promote dialogue among participants.[29] Hence, for such trials, the multiple-objective optimal design seeks the optimal number of groups per treatment condition, given fixed group sizes. Furthermore, the methodology can also be applied to trials where multiple subjects are treated by the same health professional;[30-33] here the number of clients that can be treated by a professional is often small and fixed a priori.

The contents of this paper are as follows. The next section specifies the regression models for the analysis of subject and group level outcomes, as well as the cost function that is used as a constraint for finding the optimal design. Section 3 describes two group randomized trials that are used to illustrate the optimal design methodology in the succeeding two sections. The focus of section 4 is on trials in which group sizes are fixed a priori. A summary of single-objective optimal designs from the literature is given and the multiple-objective optimal design is derived. Section 5 extents to trials in which group sizes are not fixed a priori. Conclusions and a discussion are given in the final section.

2 Specification of regression models and cost function

The first focus is on the model for a subject level outcome. Subjects are nested within groups and the dependency of outcomes within the same group must be taken into account while analyzing the data. A suitable model is the linear mixed model, which takes into account a fixed treatment effect and random effects for groups and subjects. The model that relates a quantitative outcome for subject i in group j to treatment condition is given by

Treatment condition is a binary group level predictor and takes on the value for the intervention and for the control. The model includes two fixed regression weights: is the intercept (i.e. the mean outcome in the control) and is the treatment effect size (i.e. the difference in mean outcomes between intervention and control). A treatment that affects the mean of an outcome variable may also be expected to affect its variance. The model allows for heterogeneous variances at the subject and group level and subscripts T and C are used for the random effects in the intervention and control condition and their related variances, respectively. At the group level, we have and and the two variances do not need to be equal. Similarly, the random effects ) and ) capture the residual variance at the subject level and again we do not assume homogeneity. The total variances in both conditions are the sum of the variance components at the group and subject level: and . The intraclass correlation coefficients quantify the proportions variance at the group level and are defined as for the intervention and for the control.

Outcomes may also be measured at the level of the group. The quantitative group level outcome is related to treatment by the following model

Note that this model includes just one level, namely the group level j, so the subject level identifier i does not appear in the subscripts. Again, treatment condition is coded for the intervention and for the control. The quantitative outcome is measured at the group level and it is indicated by an asterisk to distinguish it from the subject level outcome . For the same reason, an asterisk is also used for the intercept and treatment effect size . Again, we allow for heterogeneity across treatments: ) is the random effect for the intervention and ) is the random effect for the control.

The experimental designs that are derived in the next two sections allow for different sample sizes across treatment conditions. and are the number of groups in the intervention and control, and and are the common group sizes in these two conditions. The combination of sample sizes is called a design. As is obvious, a design becomes more efficient when sample sizes increase, but in practice they cannot increase without bounds. The optimal designs will be derived given a budgetary constraint

This constraint allows for different subject and group level costs across treatments: are the costs per group in the intervention and are the costs per group in the control. These consist of the costs for enrolling the intervention or control within a group and the costs to measure the group level outcome. These costs do not depend on the number of subjects within a group. Similarly, are the costs to measure the subject level outcome on one subject in the intervention condition and are the costs to measure the subject level outcome on one subject in the control. The total costs are given at the left side and should not exceed the budget B. These total costs are the sum of the costs in the intervention and the costs in the control and these may be different from each other. A special case of the cost constraint is the one where and . In that case, the total sample size is limited. This may be realistic when treatments for a rare disease or disorder are compared and costs are of less importance.

3 Illustrative examples

3.1 School-based smoking prevention intervention

A school-based smoking prevention intervention was conducted to study the effects of an in-school and tailored out-of-school intervention and their interaction that targeted elementary school children in eight grade[34] in the Netherlands.

Suppose a researcher wishes to evaluate the effects of the in-school intervention in his or her country. This program consists of a school-based social influence program that is offered in seven lessons in the classroom setting. It may be obvious that all pupils within a given class should be involved in the program. It is not feasible from a practical and ethical point of view to offer the program to only part of the pupils within a class and to refrain it from others. Furthermore, the aim of the intervention will be to influence the norms within the class, which is hard to achieve when only part of it receives the intervention.

This is an illustration of an intervention where group sizes are fixed a priori. Although classes may somewhat vary with respect to their size, we assume a common class size of in both the intervention and control. The costs are assumed to be the same as in the original study: , , , and . It is obvious the class-level costs in the program condition are much larger than those in the control. Such costs consist of incentives, teaching materials, the costs to train teachers to deliver the intervention and the costs to actually implement the intervention in the class setting. These costs do not depend on the class size. The pupil-level costs are the costs for taking and processing measurements. These are much lower than the class-level costs and do not vary across treatments.

As will be shown in the next section, the single- and multiple-objective optimal design depend on the total variance and intraclass correlation coefficients in both treatments, and prior estimates must be specified. The pupil level outcome in this illustration is the attitude towards the disadvantages of smoking, which is the sum score of 11 items that are measured on a five-point scale (range 11–55). Estimates are with for the intervention and with for the control. Here we observe heterogeneity at both the class at pupil level and a higher intraclass correlation in the control than in the intervention.

Although the original study did not include an outcome measure at the level of the class, the researcher plans to measure class climate with respect to tobacco use. Such a variable could take into account social norms and peer pressure with respect to smoking. Let us assume this variable is standardized to have variance equal to 1 in the control ( and higher outcome variance in the treatment ().

3.2 Consultation time in primary care

The average consultation time in primary care in the Netherlands is 10 min.[35] Such a short amount of time may not only result in incorrect diagnoses and unnecessary referrals to second line care, but also in patient low satisfaction and physician burnout. In 2017–2018 a pilot was conducted to evaluate the effects of longer consultation time. The seven general practices that participated were requested to plan fewer consults, which could then be 15–30 min. The pilot was funded by health insurance companies.[36]

Suppose the effects of longer consultation time are to be further studied in a large-scale group randomized trial. There would be two conditions: a control group that consists of general practices that use the standard consultation time of 10 min, and an intervention group of general practices that get incentives to allow for longer consultation time. Outcomes may be measured at the level of the patient, such as quality of the communication with the physician, trust and confidence in the physician, and satisfaction.[37] Outcomes at the level of the physician may be satisfaction, work pressure, stress and burnout.

In this study it is not necessary to measure all clients within a general practice, and optimal design methodology can be used to determine the optimal size of the sample that should be taken from each practice. In this illustration, the two outcomes are satisfaction at the level of the patient and stress at the level of the physician. Suppose both are a sum score with a range 0–100. The intervention is expected to reduce the physicians' mean stress level by five points. The anticipated variances at the level of the physician are , which implies that the difference in means is 0.5 standard deviations (i.e. a medium effect). Furthermore, suppose the intervention increases patients' mean satisfaction by 10 points. A priori estimates at the patient level are with . Here a large effect is anticipated.

The costs to measure satisfaction (and other outcomes) on one patient are independent of treatment condition: . Such costs could include the costs to send out a questionnaire and process the responses, but also incentives to increase response rate. The costs at the level of the physician are much higher. Suppose a physician in the intervention receives , which are incentives to participate, costs to allow for longer consultation time, and costs to measure the physician's stress level. The costs per physician in the control are much lower at 500, which are incentives to participate in the study and costs to measure the physician's stress level.

4 Group sizes fixed a priori

In this section, the group sizes and are assumed fixed, so the optimal design problem reduces to finding the optimal number of groups and in the intervention and control conditions: . Here an asterisk is used to indicate the optimal design and optimal sample sizes.

The single- and multiple-objective optimal designs can be found on the basis of a free web application at https://utrecht-university.shinyapps.io/CRT_fixed_cluster_sizes/.

4.1 Single-objective optimal designs

The treatment effect for the group level outcome is estimated by taking the difference in mean group level outcomes of the two conditions: , where and are the mean outcomes in the intervention and control conditions, respectively. As these means are independent from each other, the variance of the treatment effect estimator is simply the sum of the variances of the two means

This variance depends on the number of groups in both conditions, that is, it depends on the design ξ. The objective is to minimize this variance: subject to the cost constraint (equation (3)). The subscript of the objective indicates that the outcome is measured at the second (i.e. group) level.

The optimal design is available in the literature[22] and is usually expressed as a ratio of sample sizes

This ratio does not depend on the total budget B. The optimal design has a clear interpretation: one would allocate more groups to the intervention condition (at the expense of allocating groups to the control) when the outcome variance in the intervention increases and/or when the costs on the control increase.

The performance of any alternative design ξ as compared to that of the optimal design is expressed in terms of the efficiency

The optimal design has efficiency equal to 1 and the efficiencies of all other designs are lower. High efficiencies of 0.8 or 0.9 are generally desired.

The optimal design for the subject level outcome is derived in a similar way. Again, the effect of treatment is estimated by taking the mean difference in outcomes of the two treatments: , where and are the mean outcomes in the intervention and control conditions, respectively. The variance of this estimator also depends on the design ξ through the sample sizes

The objective is to minimize the variance of this estimator: . The optimal design is[14] and as for the group level outcome it does not depend on the total budget B. The number of groups in the intervention increases with the total variance and intraclass correlation coefficient in the intervention and with the group size and costs in the control.

4.2 An example: school-based smoking prevention intervention

The a priori estimates of the variances and intraclass correlation coefficients from Section 3 are used, as well as the cost specification in that section. The single-objective optimal design for the class level outcome is given by the optimal ratio . The optimal design can also be expressed in terms of the proportion groups in the intervention condition: , which implies almost 50:50 allocation. The efficiencies of all other designs are presented in the efficiency plot in panel A of Figure 1. It is obvious that lower efficiencies are achieved when the proportion is further away from the optimal proportion.

Figure 1.

Optimal designs for trials with a fixed group size. Top panels: efficiency plots for single-objective optimal designs for a group level outcome (panel A) and a subject level outcome (panel B). Bottom panels: multiple-objective optimal designs (Panel C: optimal allocation; panel D: efficiency plot).

The single-objective optimal design for the pupil level outcome is . The optimal proportion classes in the intervention condition is about a third: . The efficiency plot in panel B shows the efficiencies of all other designs.

4.3 Multiple-objective optimal designs

The single-objective optimal designs for the pupil and class level outcomes in the example are different from each other. A multiple-objective optimal design can be constructed to take both objectives into account simultaneously. Let us assume objective is the more important one. The multiple-objective optimal design is the design that has highest efficiency under the other objective given that the efficiency for is larger than a user-defined constant e

This is a so-called constrained optimal design and it is most often difficult to find. As an alternative, one may construct a compound optimal design to minimize

The weight assigns a degree of importance to both objectives and .

Under convexity and differentiability, the constrained and compound optimal designs are equivalent.[24] Thus, the desired constrained optimal design may be found by first forming a compound optimal design as a function of the weight λ. Then, an efficiency plot is drawn in which the relation between both efficiencies and is given as a function of λ. The constrained optimal design is the one with λ such that and is maximized.

The two objectives and are often divided by their minimal values so that the two components in equation (10) are of equal magnitude. The optimality criterion is then

This can be rewritten (see Appendix 1) as where and .

The optimal design is further derived in Appendix 1 and is equal to

Again, we note the optimal design does not depend on the total budget B.

4.4 Example (continued)

The two plots at the bottom of Figure 1 visualize the multiple-objective optimal design. Panel C shows the optimal proportion of classes in the intervention as a function of These proportions are the optimal proportions for and when and , respectively. Panel D shows the efficiencies of both objectives and as a function of For any λ, the two objectives have high efficiencies of 0.9 or higher.

For these efficiencies are equal to each other and as high as 0.98. The optimal proportion of groups in the intervention condition is then equal to 0.62.

In this example, the two objectives are compatible, which means high efficiencies can be achieved for both of them simultaneously. When this is not the case, the objectives are competitive.

5 Group sizes not fixed a priori

The focus of this section is on the case where group sizes and are not fixed a priori. The optimal size of the sample to be drawn from each group may vary across treatment conditions. Finding the optimal designs is more complicated than in the previous session as they depend on four rather than two sample sizes: . The web application is available at https://utrecht-university.shinyapps.io/CRT_nonfixed_cluster_sizes/.

5.1 Single-objective optimal designs

The total budget B can be split into two components: , where is the budget for the intervention and for the control. The optimal design determines the optimal split, along with the optimal number of groups and optimal group sizes in both conditions.[15]

The optimality criterion for the group level outcome is given by equation (4) and as is obvious decreases when the number of groups in both conditions increases. So, it is more efficient to spend the whole budget on enrolling groups, implementing the treatment conditions and measuring the group level outcome than to spend it on measuring the subject level outcome on any subject. So, we set and the cost function then becomes

It should be understood that the constraint does not imply the trial does not include any subjects. In the example on consultation time in primary care, the physicians do indeed meet their patients during consults; without meeting patients it would not be possible to measure physicians’ stress at all. The notation implies the patient level outcome satisfaction is not measured on any of their patients. The optimal design can be expressed in terms of the optimal ratio of the number of groups in both treatments

This equation follows from equation (5) by setting . Alternatively, the optimal design can be expressed as the optimal ratio of costs in both treatments

This ratio does not depend on the total budget B. A higher part of the budget should be assigned to the intervention when the variance in the intervention and/or costs in the intervention increase.

The derivation of the optimal design for the subject level outcome is more complicated because not only the number of groups in both conditions needs to be derived but also the optimal group sizes.

Given budget for the intervention condition, the optimal sample sizes for the intervention condition are

These optimal sample sizes are found by expressing as a function of and the costs using the budgetary constraint . This expression is then substituted into the objective which is minimized with respect to to derive the optimal . The optimal then follows from the budgetary constraint.[12]

It is obvious that the optimal group size increases when the within-group variability becomes higher and/or when the group level cost increase. The optimal group size does not depend on the budget , while the number of group does depend on this budget. The optimal number of groups increases with the budget and decreases with increasing group size. The mean outcome in the intervention condition is then estimated with variance[12]

Equations (17) and (18) also hold for the control condition when the subscript T is replaced by C.

The optimal design question is how large and should be given fixed B. The optimal design follows from equation (26) in Lemme et al.[15]

Again the budget assigned to the intervention condition increases with the variance and costs in the intervention.

Substitution of equation (19) into the equation gives the minimal variance for the treatment effect estimator

5.2 An example: consultation time in primary care

For the outcome at the practice level, the optimal ratio of budgets is , which implies that the budget allocated to the intervention is over six times as large as the budget to the control. This is not surprising given that the practice level costs in the intervention are much higher than those in the control. The optimal design can also be expressed in terms of the proportion of the total budget that is allocated to the intervention: . Since we fixed , the budget is solely spent on costs at the level of the practice. The efficiencies of all other designs are given in panel A of Figure 2. Using a lower proportion than the optimal one has a stronger impact on efficiency than using a higher proportion.

Figure 2.

Optimal designs for trials with a non-fixed group size. Top panels: efficiency plots for single objective optimal designs for a group level outcome (panel A) and a subject level outcome (panels B and C). Bottom panels: multiple-objective optimal designs (panel D: optimal allocation of budget; panel E: optimal group size, panel F: efficiency plot).

For the patient level outcome, the optimal ratio of budgets is , which is equal to a proportion of budget allocated to the intervention of . Again, this proportion is rather high because of the high practice level costs in the intervention. The efficiencies of all other designs are given in panel B. The budgets and are not only spent at the level of the practice, but also at the level of the patient. The optimal number of patients to be measured within each practice follows from equation (17). For the intervention condition, it is and for the control it is . These optimal number of patients per practice are so very different because the practice level costs vary so much over the two conditions. Panel C shows the efficiencies of all other number of patients per practice for both treatment groups.

5.3 Multiple-objective optimal design

The optimality criterion for the multiple-objective optimal design is given by equation (12), but now group sizes are not fixed a priori. The complete derivation of the multiple-objective is given in Appendix 1.

The first part of the optimal design question is how large and should be given fixed B. The optimal budget split is equal to where and .

The second part of the optimal design question is how large the group sizes and should be. These optimal group sizes do not depend on the budget split. For the intervention condition, we have

This simplifies to the optimal group size for the group level outcome when λ = 1 (i.e. when ), and to the optimal group size for the subject level outcome when λ = 0 (i.e. when ). The optimal number of groups in the control follows from the optimal group sizes through the budgetary constraint: . The same equations hold for the control condition (with subscript T replaced by C).

As before, the efficiencies of both objectives can be drawn in an efficiency plot as a function of the weight λ. The objective is compatible if both efficiencies are above a user selected value (most often 0.8 or 0.9). Otherwise, the two objectives are competitive and the selection of λ is explained as below equation (10).

5.4 Example (continued)

Panel D of Figure 2 shows the optimal proportion of the budget that is allocated to the intervention condition as a function of λ. For this is the optimal proportion for the patient level outcome; for it is the optimal proportion for the practice level outcome. This proportion only slightly varies with λ, which implies that the patient and practice level objectives and are compatible with respect to the budget split.

For the practice level objective , the budget is solely spent at the level of the practice and no measurements are taken at the patient level (). This is not the case for the objective for the patient level outcome. This is visualized in panel E, where the optimal number of patients per group strongly depends on λ, especially so for the intervention condition. This implies that the two objectives are competitive with respect to the optimal number of patients per group. For these are the optimal number of patients per practice for the patient level outcome and for for the practice level outcome.

Panel F shows the efficiencies of both objectives as a function of λ. Both are related to λ and to the strongest degree for the patient level outcome. For the efficiency for the patient level outcome is equal to zero because no measurements are taken at the patient level. In that case the effect of the intervention on the patient level outcome cannot be estimated. For both objectives have an efficiency of at least 0.8. This implies that both objectives are compatible if one aims for an efficiency of at least 0.8 for both of them. However, the objectives are competitive if one aims for efficiencies of at least 0.9 since such a high efficiency cannot be achieved for both objectives simultaneously. Consider the case where is the more important objective and an efficiency of 0.9 should be achieved. Then the multiple objective optimal design is found at and the efficiency of is 0.84. The latter efficiency is lower than the desired value 0.9 since the two objectives are competitive.

For the efficiencies are equal to each other and as high as 0.88. In that case, a proportion of the budget is allocated to the intervention condition. The optimal number of patients that is to be sampled from each practice is then in the intervention and in the control.

6 Discussion and conclusions

In group randomized trials, outcomes may not only be measured at the level of the subject but also at the level of the group. Thus far, the optimal design of group randomized trials with outcomes at both levels had not been studied. This paper proposed the use of multiple-objective optimal designs to take outcomes at the subject and group level into account simultaneously. The methodology was illustrated using two examples from smoking prevention and consultation time in primary care. Two free web applications were made available to find the single- and multiple-objective optimal designs and to evaluate the efficiency of all other designs. The R syntax that was used to build these web applications is available upon request.

This paper considered a general case with treatment-dependent costs and variances. The optimal design is locally optimal in the sense that it depends on the treatment-dependent variances of the outcome at the group level and treatment-dependent total variance and intraclass correlation coefficient of the subject level outcome. These model parameters are most often not known in the design phase of a group randomized trial and prior estimates may be obtained from expert knowledge or the literature. Table 11.1 in Moerbeek and Teerenstra[38] gives an overview of dozens of papers that published estimated of intraclass correlation coefficients in many research fields with various types of groups. Of course, there is no guarantee that estimates from the literature will hold in another year, country or setting. It is therefore suggested to use the free web applications to do a robustness analysis. The optimal design can be derived for various plausible values of the variances and intraclass correlation coefficients and the optimal design is robust if it hardly depends on the chosen values. If the design is not robust, then it is suggested to use robust optimal designs techniques, such as internal pilots or maximin optimal designs. This will be topic of future research.

This contribution restricted to quantitative outcomes at the subject and group level. The results are also applicable to binary logistic regression models if appropriate substitutions in the equations are made. For subject level outcomes, the variances and are replaced by and , where and are the response rates in the intervention and control conditions.[39] Similarly, for a group level outcome we replace and by and , where and are the response rates in the intervention and control conditions. It would also be interesting to study designs where the outcome at the one level is quantitative and the outcome at the other level is binary, and to extend to other types of outcomes in a generalized linear mixed model (i.e. ordinal and nominal outcomes).

The optimal designs allow the group sizes to vary between but not within treatments. In practice it is likely that group sizes also vary within treatments. For instance, there was some variation in the sizes of the school classes in the smoking prevention intervention example. In the example from general care, the same number of patients may be sampled from each general practice, but varying group sizes may still occur as a result of non-response. For group randomized trials with an outcome at the subject level, it has been advised to increase the number of groups by 11%.[40] Future research should verify if this is also the case with outcomes at multiple levels.