Clinically Meaningful Improvements in Early Morning and Late Afternoon/Evening Functional Impairment in Children with ADHD Treated with Delayed-Release and Extended-Release Methylphenidate.

OBJECTIVE: The Before School Functioning Questionnaire and Parent Rating of Evening and Morning Behavior-Revised assess early morning (BSFQ, PREMB-R AM subscale) and late afternoon/evening (PREMB-R PM subscale) functional impairment in children with ADHD. Clinically meaningful improvements were identified and applied to a trial of delayed-release and extended-release methylphenidate (DR/ER-MPH) in children with ADHD (NCT02520388) to determine if the statistically-determined improvements in functional impairment were also clinically meaningful.
METHOD: Clinically meaningful improvements in BSFQ/PREMB-R were established post hoc by receiver operating characteristics curves, using anchors of Clinical Global Impression-Improvement (CGI-I) = 1 and CGI-I ≤ 2. Percentages of participants achieving these thresholds were calculated.
RESULTS: Thresholds for CGI-I = 1/CGI-I ≤ 2, respectively, were 27/20 (BSFQ), 5/3 (PREMB-R AM), and 9/5 (PREMB-R PM)-point decreases. More children achieved clinically meaningful improvements with DR/ER-MPH versus placebo (all p < .05).
CONCLUSION: DR/ER-MPH increased proportions of children achieving clinically meaningful improvements in BSFQ and PREMB-R.

Introduction

ADHD is a common neurodevelopmental disorder that affects an estimated 11% of children and 4.4% of adults in the United States (Kessler et al., 2006; Visser et al., 2014). ADHD is characterized by symptoms of inattention, hyperactivity, and impulsivity (American Psychiatric Association, 2013). Although school-aged children with ADHD often manifest disruption in the early and later parts of their day (i.e., before and after school), little information is available on the corresponding functional impairment during, or the impacts of treatment on, these non-school times of day. Activities such as early morning organization, self-care, preparing for the school day, and transportation to school are rarely captured in treatment studies of children with ADHD, yet they have a profound impact on these children and their families (Barkley et al., 1979; Faraone, Schachar, et al., 2017; Sallee, 2015). After-school and early evening activities such as returning from school, homework, sports, and peer or family interactions are also often neglected in studies of ADHD (Faraone et al., 2020). Similarly, adults with ADHD also experience impairment during the early morning and late afternoon/evening times before and after work (Wigal et al., 2020). Hence, studies examining treatment outcomes would benefit from focus on these important non-school/work periods of the day for individuals with ADHD and their families.

Current treatment guidelines for ADHD include improved functioning as an optimal treatment goal (Canadian ADHD Resource Alliance, 2020; Pliszka & The AACAP Work Group on Quality Issues, 2007), yet there has only recently been a shift toward defining optimal clinical outcomes for individuals with ADHD that go beyond symptom reduction to include approaches that reliably and validly assess ADHD-related functioning (Rostain et al., 2015; Sasser et al., 2017; Wong et al., 2019). Indeed, the European ADHD Guidelines Group recently noted that clinical trials in ADHD continue to focus on symptom reduction as the primary measure of efficacy, and strongly recommended that a broader range of outcome measures be incorporated into both clinical trials and clinical practice (Wong et al., 2019). It should be noted that non-pharmacological treatments such as parent training are also available to complement the action of medications on behavioral outcomes (Daley et al., 2018). Nevertheless, as a result of the focus on symptom reduction in medication trials, there is a robust literature on the effects of stimulants on ADHD symptoms but a paucity of literature on the naturalistic effects of stimulants on functional impairment, particularly before or after school/work (Cortese et al., 2018). Analog (laboratory) classroom/workplace studies have been helpful in demonstrating the time course of treatment effects on ADHD symptoms or math problem completion—generally showing onset of treatment effect between 30 minutes and 2 hours after administration that lasts for 10 to 16 hours for extended-release agents—however, these studies are limited to a simulated classroom/workplace environment and do not capture the before- and after-school/work times of day in an ecologically valid manner (Brams et al., 2008; Childress, 2016; McGough et al., 2006; Swanson et al., 2003; Wigal et al., 2020). For example, the laboratory classroom paradigm evaluates children with ADHD in a classroom made up entirely of children with ADHD, which is not an ecologically valid scenario. Given that the before- and after-school/work time represents a substantial portion of the waking day, naturalistic studies that capture treatment efficacy in these periods of time will shed important light on the mitigation of functional impairment with treatment (Faraone, Schachar, et al., 2017; Sallee, 2015).

To date, two rating scales have been validated for assessing temporal functional impairment in clinical trials of children with ADHD. The Before School Functioning Questionnaire (BSFQ) is a parent- or investigator-rated scale that evaluates early morning functional impairment between the time of awakening and getting to school or other morning activities (i.e., between 6:00 and 9:00 AM) (Faraone, Hammerness, et al., 2018; Wilens et al., 2010). The Parent Rating of Evening and Morning Behavior Scale–Revised (PREMB-R) consists of two separately validated subscales that assess early morning functional impairment (PREMB-R AM) and late afternoon/evening functional impairment (PREMB-R PM) through a clinician-rated scale based on a parent interview (Faraone, Childress, et al., 2018; Faraone, DeSousa, et al., 2017; Sutton et al., 2003). The internal homogeneity, test-retest reliability, and concurrent validity of these scales has been previously established (Faraone, Childress, et al., 2018; Faraone, DeSousa, et al., 2017; Faraone, Hammerness, et al., 2018; Sutton et al., 2003). Studies with both stimulant and nonstimulant medications have shown changes in the BSFQ and PREMB-R (Michelson et al., 2002; Wilens et al., 2010, 2017). Evening-dosed HLD200, a delayed-release and extended-release methylphenidate (DR/ER-MPH; Jornay PM®, Ironshore Pharmaceuticals Inc.), is a stimulant that has been shown to be efficacious in treating ADHD throughout the day (Childress et al., 2020; Pliszka et al., 2017). In two randomized, double-blind, placebo-controlled trials in children with ADHD, treatment with DR/ER-MPH resulted in statistically significant improvements in functional impairment during the early morning and late afternoon/evening versus placebo, as measured by the BSFQ and PREMB-R (Childress et al., 2020; Pliszka et al., 2017). Data on the interpretation of BSFQ and PREMB-R scores (e.g., best cut points) have not been previously published.

While informative, statistically significant differences in a clinical trial outcome provide limited information to guide treatment in clinical practice. One accepted way to arrive at a clinically meaningful improvement in an outcome is to anchor that outcome to improvement on an established scale of clinically meaningful change, notably the Clinical Global Impression–Improvement (CGI-I) scale (Goodman et al., 2010; Weiss et al., 2019). As it relates to symptom improvement, Goodman et al. (2010) showed that a minimally detectable clinical change, as defined by a change in CGI-I score of at least one level (e.g., a difference between 3, “minimally improved” and 2, “much improved”) required at least a 25% to 30% change in ADHD Rating Scale IV (ADHD-RS-IV) score in children and adults with ADHD. Weiss et al. (2019) demonstrated a similar relationship in children with ADHD, with clinically meaningful improvement (defined by the authors as a CGI-I score ≤2, “much improved” or “very much improved”) being aligned with a ≥40% reduction in ADHD-RS-IV score. Clinicians may prefer to use scales that measure functional improvement over scales that measure symptom improvement, as complaints related to functional impairment are often what brings a patient to treatment (Epstein & Weiss, 2012). Therefore, knowing the amount of change on a functional scale that corresponds to improvement on the CGI-I, a scale that is a metric of clinically meaningful change in a specific domain such as ADHD, may be important information for guiding treatment.

The aim of this study was to determine thresholds for the BSFQ and PREMB-R that corresponded to established measures of clinically meaningful improvement (i.e., scores of 1 or ≤2 on the CGI-I), with the purpose of applying these thresholds to previously reported statistically significant improvements in early morning and late afternoon/evening functional impairment following 3 weeks of treatment with DR/ER-MPH versus placebo in a phase 3 trial (Pliszka et al., 2017) to determine if they were clinically meaningful.

Method

Data Source

This was an exploratory post hoc analysis based on data from a randomized, double-blind, multicenter, placebo-controlled, parallel-group, phase 3 trial (ClinicalTrials.gov identifier: NCT02520388) of DR/ER-MPH in children with ADHD (aged 6–12 years) (Pliszka et al., 2017). Key inclusion criteria included: (1) diagnosis of ADHD based on Diagnostic and Statistical Manual of Mental Disorders, 5th Edition criteria; (2) ADHD-RS-IV score ≥90th percentile for age and gender and ≥26 at baseline; (3) Clinical Global Impression–Severity score ≥4 and Conners’ Global Index–Parent score >10 at baseline; (4) at least a partial clinical response to MPH; and (5) early morning functional impairment and/or difficulties performing a morning routine of ≥30 minutes between 6:00 and 9:00 am. Key exclusion criteria included: (1) history of or current medical condition or laboratory result that could either jeopardize participant safety or interfere with study participation; (2) history of psychosis, bipolar disorder, anorexia nervosa, bulimia, or suicide attempt; (3) current depression, anxiety, conduct disorder, substance use disorder, or other psychiatric condition; (4) history of severe allergic reaction or intolerance to MPH; and (5) past use of psychotropic medication.

The trial was conducted in two phases: a screening phase of up to 2 weeks with a minimum 72-hour washout, and a 3-week treatment phase using a forced-dose titration schedule. At the start of the treatment phase, participants were randomized (1:1) to receive either DR/ER-MPH (40 mg/day) or placebo once daily each evening at 8:00 pm (±30 minutes) for 1 week, with scheduled titration, as medically indicated and tolerated, in 20-mg increments over the subsequent 2 weeks (i.e., 60 and 80 mg/day, respectively). The maximum allowable dose was 3.7 mg/kg/day, and one down-titration step (i.e., 20-mg decrement) was permitted for safety or tolerability reasons. Participants who were unable to tolerate a dose of at least 40 mg during Week 3 were discontinued from the study. Evening dosing time adjustments between 6:30 and 9:30 pm in 30- or 60-minute increments per week were permitted to achieve optimal morning control of ADHD symptoms.

Assessments

The primary efficacy endpoint was the ADHD-RS-IV total score following 3 weeks of treatment. Secondary efficacy measures included the investigator-rated BSFQ, PREMB-R AM, PREMB-R PM, and CGI-I following 3 weeks of treatment. The investigator-rated BSFQ is a 20-item scale based on a structured parent interview (Faraone, DeSousa, et al., 2017; Faraone, Hammerness, et al., 2018; Wilens et al., 2010). Each item is rated on a 4-point scale (0 = none; 1 = mild; 2 = moderate; and 3 = severe), with a possible total score between 0 (no functional impairment) and 60 (high functional impairment). The PREMB-R AM and PM are also clinician-rated scales based on a structured parent interview (Faraone, Childress, et al., 2018; Faraone, DeSousa, et al., 2017; Sutton et al., 2003). Each item is rated on a 4-point scale (0 = none; 1 = a little; 2 = a moderate amount; and 3 = a lot). The possible total score ranges from 0 to 9 for the 3-item PREMB-R AM and from 0 to 24 for the 8-item PREMB-R PM, with a higher total score indicating greater functional impairment. The CGI-I is an investigator-rated scale used to evaluate clinically relevant improvement in a participant’s condition (e.g., ADHD) from baseline to endpoint based on the query: “Compared to his/her condition at admission to the project, how much has he/she changed?” (Goodman et al., 2010; Guy, 1976). Responses are rated on a 7-point scale (1 = very much improved; 2 = much improved; 3 = minimally improved; 4 = no change from baseline; 5 = minimally worse; 6 = much worse; and 7 = very much worse).

Statistical Analyses

The ADHD-RS-IV and BSFQ total scores at Week 3 were analyzed by using a mixed model repeated-measures analysis with the participant’s intercept as a random effect; treatment, study center, visit, and visit-by-treatment interaction as fixed effects; and baseline score as a covariate. The PREMB-R AM, PREMB-R PM, and CGI-I total scores at Week 3 were assessed using an analysis of covariance model, with treatment as the main effect and study center and baseline score as the covariates.

Thresholds for clinically meaningful change were calculated by using a receiver operating characteristics (ROC) curve analysis to examine the relationship between changes in BSFQ, PREMB-R AM, and PREMB-R PM scores and the following predefined categorical anchors for clinically meaningful change: (1) CGI-I of 1 (very much improved), and (2) CGI-I of ≤2 (i.e., 1 or 2; very much or much improved) (Figure 1). ROC analysis evaluates the overall discriminatory performance of a test by plotting sensitivity and specificity rates for the entire range of scores and then determining a threshold that represents the highest overall level of sensitivity and specificity for the given anchor. True positive (TP), true negative (TN), false positive (FP), and false negative (FN) values were computed for each anchor and used to calculate sensitivity and specificity using the following formulas: Sensitivity = TP/(TP + FN); Specificity = TN/(TN + FP). ROC curves with calculation of the area under the curve (AUC) were used to estimate the thresholds by identifying the BSFQ, PREMB-R AM, and PREMB-R PM score changes that best described a change in CGI-I in terms of Youden’s Index (a commonly used statistic that reflects overall test performance, giving equal weight to sensitivity and specificity). Higher AUC values indicate better overall test sensitivity and specificity, with 1.0 representing perfect discrimination between true and false positives and 0.5 representing chance classification. SAS v9.3 was used to perform the ROC curve analyses.

Figure 1.

ROC analysis of change from baseline to Week 3 in BSFQ, PREMB-R AM, and PREMB-R PM scores in participants reaching CGI-I anchors.

Note. BSFQ = Before School Functioning Questionnaire; CGI-I = Clinical Global Impression–Improvement; PREMB-R AM = Parent Rating of Evening and Morning Behavior–Revised, Morning subscale; PREMB-R PM = Parent Rating of Evening and Morning Behavior–Revised, Evening subscale; ROC = receiver operating characteristics.

Effect sizes (ES) for each parameter were calculated by dividing the mean change by the standard deviation of the baseline scores. The percentage of participants achieving each threshold was calculated and differences between treatment groups were determined using Chi-square tests. Cumulative percentages of children with changes from baseline in BSFQ, PREMB-R AM, and PREMB-R PM anchored to CGI-I were visualized using cumulative distribution functions.

Results

As previously described (Pliszka et al., 2017), there were 163 children enrolled across 22 sites, of whom 161 were included in the intent-to-treat population (DR/ER-MPH, n = 81; placebo, n = 80). The study population was 70% male, the mean age was 9.3 years, and the mean ADHD-RS-IV score at baseline was 43.3. The mean DR/ER-MPH dose after 3 weeks of treatment was 68.1 mg, and no participants discontinued due to an inability to tolerate the mandated minimum dose of 40 mg. Mean BSFQ, PREMB-R AM, and PREMB-R PM scores at baseline were 44.2, 6.4, and 17.4, respectively in the DR/ER-MPH group and 44.9, 5.8, and 16.6, respectively in the placebo group. There were no significant differences between treatment groups at baseline for demographic and baseline characteristics. Following 3 weeks of treatment with DR/ER-MPH in children with ADHD, there were statistically significant improvements in ADHD symptoms, early morning and late afternoon/evening functional impairment, and global improvement versus placebo, as demonstrated by the ADHD-RS-IV (least-squares [LS] mean: 24.1 vs. 31.2; p = .002), BSFQ (LS mean: 18.7 vs. 28.4; p < .001), PREMB-R AM (LS mean: 2.1 vs. 3.6; p < .001), PREMB-R PM (LS mean: 9.4 vs. 12.2; p = .002), and CGI-I (LS mean: 2.3 vs. 3.1; p < .001) respectively.

The mean changes in BSFQ, PREMB-R AM, and PREMB-R PM scores in participants who achieved a CGI-I score of 1 were −37.06 (ES: −3.24), −5.13 (ES: −2.10), and −13.28 (ES: −2.81), respectively (Table 1). Among participants who achieved CGI-I scores of ≤2, mean changes in BSFQ, PREMB-R AM, and PREMB-R PM were −32.24 (ES: −3.24), −4.66 (ES: −1.97), and −9.68 (ES: −2.23), respectively (Table 1). The AUCs for all ROC curves were high (between 0.74 and 0.92), indicating that the BSFQ, PREMB-R AM, and PREMB-R PM scores could be used effectively to identify patients who achieved the anchor scores of CGI-I = 1 and CGI-I ≤ 2 (Table 1; Figure 1).

Table 1.

ROC Curve Statistics and Thresholds for the BSFQ, PREMB-R AM, and PREMB-R PM.

Parameter	CGI-I = 1 (very much improved)			CGI-I ≤ 2 (much/very much improved)
	BSFQ	PREMB-R AM	PREMB-R PM	BSFQ	PREMB-R AM	PREMB-R PM
Event (n)	32	32	32	79	79	79
Mean change	−37.06	−5.13	−13.28	−32.24	−4.66	−9.68
AUC	0.860	0.736	0.885	0.918	0.796	0.843
Threshold (points)	−27	−5	−9	−20	−3	−5
Mean improvement from baseline (%)	61	82	53	45	49	29
Effect size	−3.24	−2.10	−2.81	−3.24	−1.97	−2.23

Note. AUC = area under the curve; BSFQ = Before School Functioning Questionnaire; CGI-I = Clinical Global Impression–Improvement; PREMB-R AM = Parent Rating of Evening and Morning Behavior–Revised, Morning subscale; PREMB-R PM = Parent Rating of Evening and Morning Behavior–Revised, Evening subscale; ROC = receiver operating characteristics.

Using the CGI-I scores of 1 and ≤2 as anchors, reductions in BSFQ scores of 27 and 20 points, respectively, were identified as thresholds for clinically meaningful improvement. These corresponded to 61% and 45% improvements from the mean baseline BSFQ score, respectively (Table 1). For the PREMB-R AM, thresholds for clinically meaningful improvement were reductions of 5 points anchored to CGI-I = 1 and 3 points anchored to CGI-I ≤ 2, which corresponded to 82% and 49% improvements from the mean baseline PREMB-R AM score, respectively (Table 1). For the PREMB-R PM, thresholds for clinically meaningful improvement were reductions of 9 points anchored to CGI-I = 1 and 5 points anchored to CGI-I ≤ 2, corresponding to 53% and 29% improvements from the mean baseline PREMB-R PM score, respectively (Table 1).

For BSFQ scores, a higher proportion of children achieved a “much/very much improved” (CGI-I ≤ 2) change from baseline (i.e., reduction of ≥20 points) after 3 weeks of treatment with DR/ER-MPH versus placebo (65.4% vs. 43.3%; p = .012) (Figure 2). More children also achieved a “very much improved” (CGI-I = 1) change from baseline (i.e., reduction of ≥27 points) in BSFQ scores with DR/ER-MPH treatment versus placebo (51.3% vs. 23.9%; p = .001) (Figure 2). For the PREMB-R AM, higher proportions of children treated with DR/ER-MPH versus placebo achieved a “much/very much improved” change from baseline (CGI-I ≤ 2, i.e., reduction of ≥3 points: 73.4% vs. 41.3%; p < .001) and a “very much improved” change from baseline (CGI-I = 1, i.e., reduction of ≥5 points: 45.6% vs. 24.0%; p = .008) (Figure 3). Similar results were also observed in the PREMB-R PM, with a higher proportion of children treated for 3 weeks with DR/ER-MPH versus placebo demonstrating a “much/very much improved” change from baseline (CGI-I ≤ 2, i.e., reduction of ≥5 points: 64.6% vs. 42.7%, p = .01) and a “very much improved” change from baseline (CGI-I = 1, i.e., reduction of ≥9 points: 46.8% vs. 24.0%, p = .005) (Figure 4).

Figure 2.

Cumulative percentage of participants with specified changes from baseline in BSFQ score anchored to CGI-I after 3 weeks of treatment.a

Note. BSFQ = Before School Functioning Questionnaire; CGI-I = Clinical Global Impression–Improvement; CI = confidence interval; DR/ER-MPH = delayed-release and extended-release methylphenidate.

aNegative score change indicates improved early morning functional impairment, as measured by the BSFQ.

Figure 3.

Cumulative percentage of participants with specified changes from baseline in PREMB-R AM score anchored to CGI-I after 3 weeks of treatment.a

Note. CGI-I = Clinical Global Impression–Improvement; CI = confidence interval; DR/ER-MPH = delayed-release and extended-release methylphenidate; PREMB-R AM = Parent Rating of Evening and Morning Behavior–Revised, Morning subscale.

aNegative score change indicates improved early morning functional impairment, as measured by the PREMB-R AM.

Figure 4.

Cumulative percentages of participants with specified changes from baseline in PREMB-R PM score anchored to CGI-I after 3 weeks of treatment.a

Note. CGI-I = Clinical Global Impression–Improvement; CI = confidence interval; DR/ER-MPH = delayed-release and extended-release methylphenidate; PREMB-R PM = Parent Rating of Evening and Morning Behavior–Revised, Evening subscale.

aNegative score change indicates improved late afternoon/evening functional impairment, as measured by the PREMB-R PM.

Discussion

In these post hoc analyses, two validated measures of temporal functional impairment, the BSFQ and the two subscales of the PREMB-R, were linked to a clinically relevant assessment of improvement in a placebo-controlled clinical trial of DR/ER-MPH in children with ADHD. Application of these thresholds demonstrated that treatment with DR/ER-MPH led to clinically meaningful improvements in functional impairment compared to placebo. These data also provide guidance to clinicians on the use of these functional scales that extends beyond the symptom-based outcomes in clinical trials.

In this study, reductions (i.e., improvements) from baseline of 27 and 20 points on the BSFQ, 5 and 3 points on the PREMB-R AM, and 9 and 5 points on the PREMB-R PM were thresholds of improvement in early morning and late day function associated with clinically meaningful ratings of “very much improved” and “much/very much improved”, respectively, on the CGI-I. The determination of thresholds for clinically meaningful improvement in the BSFQ, PREMB-R AM, and PREMB-R PM adds to the utility of these scales for clinicians who treat individuals with ADHD, and helps to describe the functional and temporal characteristics of response to DR/ER-MPH. The BSFQ and PREMB-R have been previously validated to assess ADHD-associated functional impairment in the early morning or late afternoon/evening (Faraone, Childress, et al., 2018; Faraone, DeSousa, et al., 2017; Faraone, Hammerness, et al., 2018). Normative-referenced cut-off points were also recently determined for both the BSFQ and PREMB-R, which allow clinicians to use the scales to identify individuals with ADHD who are experiencing difficulties in the early morning and late afternoon/evening (Faraone et al., 2020). Now, determination of thresholds for clinical meaningfulness in the analyses presented here provides clinicians with the option to use the scales to guide treatment in clinical practice.

Our findings are of interest in future prospective trials of medications for ADHD. For example, these findings suggest that improvements in functional impairment from the early morning until evening following 3 weeks of treatment with DR/ER-MPH versus placebo have clinical relevance. Given the increasing appreciation for assessing functional impairment and improvement in ADHD coupled with the need for treatment of individuals with ADHD to extend throughout the day, future studies assessing ADHD outcomes should consider utilizing thresholds of temporal functional scales such as the BSFQ and PREMB-R, which may provide clinically meaningful differentiations beyond those of statistical significance in naturalistic settings.

The current analyses have several limitations. They were post hoc and hence, relied upon existing data from a randomized controlled trial. Although anchor-based methods are recommended as the primary method for establishing responder thresholds (Food and Drug Administration, 2009), the definition of clinically meaningful improvement utilized here only takes into account one scale for external anchoring, the CGI-I, and does not consider distribution-based definitions of clinically significant change that have been employed in psychotherapy (Jacobson et al., 1991). The analyses did not include a covariate accounting for the reduction in symptom severity over time, as measured by ADHD-RS-IV scores; this reduction may have impacted the association between the BSFQ/PREMB-R and the CGI-I. While the CGI-I is an investigator-rated measure of improvement, which in this case focused on ADHD, it remains unclear if ratings on the BSFQ or PREMB-R, which were investigator-rated but based on a parent interview, ultimately influenced the CGI-I. Clearly, prospective studies examining the BSFQ and PREMB-R compared to independently rated functional or global measures are necessary.

Despite these limitations, our findings show that threshold reductions in two validated scales of early morning and late afternoon/evening functioning can be used to gauge the clinical meaningfulness of improvements in functional impairment in children with ADHD. Application of these thresholds showed that in a phase 3 trial, a larger proportion of children experienced clinically meaningful benefit from DR/ER-MPH treatment versus placebo.