Clinical Trials to Gain FDA Approval for Computerized Cognitive Training: What Is the Ideal Control Condition?

The explosive growth of mobile technologies combined with the rapid rise of aging populations fearful of their risk for Alzheimer's disease has led to a number of marketed products aimed at enhancing cognitive health. However, an increasing number of product claims that are not substantiated has led regulatory agencies, such as the Federal Trade Commission (FTC), to issue warnings or penalties against some companies. Therefore, it is likely that a number of computerized cognitive training (CCT) companies will conduct clinical trials to prove their efficacy to gain Food and Drug Administration (FDA) clearance as a medical software/device. This raises a number of issues such as optimal trial design for establishing efficacy. The type of control condition is unique issue for CCT, given the variety of non-specific known to produce beneficial effects on cognition that are difficult to isolate from the content of the program. These include participant expectancy, engagement, motivation, novelty, and therapist interaction. We herein discuss the types of non-specific factors, desirable qualities of an active control condition, and the nuances that exist between previously used control conditions within the context of CCT for mild cognitive impairment.

Expectancy

One nonspecific factor is expectancy, which refers to the participant's anticipation of positive or negative treatment effects. Expectancies come in two major forms: outcome expectancy, which is the belief that the treatment itself will result in a particular outcome, and response expectancy, which is the participant's subjective response to the treatment. Expectancies of both kinds impact outcomes across modalities, including psychotherapy (Goossens et al., 2005; Smeets et al., 2008; Weinberger, 2014), pharmacological interventions (Rutherford et al., 2010), and neurosurgery (Freeman et al., 1999). Long known to produce clinically significant outcomes, the belief that one will symptomatically improve is met with high response rates. For example, a meta-analysis of randomized placebo-controlled trials of antidepressants found the placebo effect to be responsible for the majority of observed change, in depression symptoms, with a response rate of 50% for those receiving medications compared to 30% for groups assigned to the placebo condition (Walsh et al., 2002).

Further evidence for the contribution of expectancy to treatment effectiveness can be found in the different response rates across different trial designs. Greater response rates have been observed in comparator trials between two drugs than in placebo-control trials in late-life depression (Sneed et al., 2008). This discrepancy may be a consequence of participants in comparator trials knowing they will receive treatment, while participants in placebo-controlled trials are hoping they are assigned to the treatment arm (and hence have lower expectations). This finding has been extended in middle-aged and adolescent depression (Rutherford et al., 2011) as well as in schizophrenia (Rutherford et al., 2014). Although these groups are diverse in terms of presenting ailment, consistent among them is that degree of expected improvement coincides with magnitude of actual improvement.

Rutherford et al. (2010) elaborated on a model of expectancy effects driving the differences in response rates between types of trials in which the orbitofrontal cortex (OFC), rostral anterior cingulate cortex (rACC), and nucleus accumbens (NAC) generate and maintain expectancies (Rutherford et al., 2010). The OFC and rACC are active during anticipation of pain and unpleasant experiences (Petrovic et al., 2005). Placebos lower activation of these regions which coincides with reduced severity of experienced pain (Wager et al., 2004). This analgesic effect may be partially explained by heightened opioid activity in the OFC and rACC that follows placebo administration and correlates with reported pain alleviation (Wager et al., 2007). Activation of the NAC, on the other hand, occurs in anticipation of reward. Greater activation of the NAC occurs with expectancy of analgesia and is correlated with placebo-induced pain reduction (Scott et al., 2007). Taken together, heightened expectancy of relief attenuates appraisal of negative experiences to a clinically significant degree.

In addition to altering perception of negative symptoms, raising expectancy increases participant's belief of self-efficacy in their functioning, making them more confident in their own capability to perform tasks. Judgments of high self-efficacy are associated with greater exerted effort in the face of challenges, as well as lengthened persistence (Bandura, 1982). A systematic review of placebo responses found self-efficacy and locus of control to be significant predictors of symptom improvement (Horing et al., 2014). The impact of response expectancy on cognition is particularly important in trials of CCT, given the use of performance-based outcome measures. In trials of CCT, participants are told that improvement in their cognitive functions is possible as a benefit from their involvement in the study. This creates an expectation of better performance following training, which is known to create positive cognitive outcomes. Indeed, participants who enrolled in a trial of CCT via a flyer suggesting CCT will improve working memory and fluid intelligence had greater post-training performance than participants who enrolled via a non-suggestive flyer, despite participating in the same program (Foroughi et al., 2016). Similarly, in a trial of healthy adults randomly assigned to a placebo pill or no-pill condition, those who took the placebo pill described as a “cognitive enhancer” had greater performance on tests of attention and delayed recall, though no effect was found in five secondary outcome measures (Oken et al., 2008). Given the potential effect of expectancy on cognitive performance, and the wide use of neuropsychological tests as primary outcomes in CCT trials, treatment, and comparison conditions must be balanced in terms of anticipated improvement.

Engagement

How well a CCT program captivates and sustains attention will affect the capability of subjects to participate and focus during training. How engaging a task is depends on numerous factors, including usability, focused attention, positive affect, esthetics, endurability, richness, and control (Wiebe et al., 2014). These characteristics can be independent of the main program content. For example, a 2-back task necessitates working memory ability by nature of the design alone. By adding auditory feedback, sound effects, colorful animations, score tracking, and countdown timers, the task becomes more attractive to the participant. Such design elements create a nonspecific cognitive load, are not unique to CCT, yet make the CCT condition more interesting than the comparison group. When comparing an engaging CCT task to either waitlist or uninteresting control activities, it becomes impossible to isolate the central treatment effect.

Motivation

Whereas engagement may be understood as emotional or attentional investment during a task, motivation is a global personal orientation to the activity (Wiebe et al., 2014). A participant may believe that cognitive improvement is contingent upon active involvement in their training. Numerous cognitive training platforms track then share performance data with participants. Following conclusion of a module, participants are shown their score, typically alongside a report of previous attempts. Congratulatory messages for surpassing earlier records are common. This positive feedback and potential for goal-setting behavior creates an environment where the participant is motivated to perform well.

Participant's beliefs about the malleability of their intelligence will affect their performance as well. In a study evaluating performance on tests of general knowledge, participants who held the believe that intelligence is malleable corrected more errors in their responses during retesting than participants who believed that intelligence is fixed (Mangels et al., 2006). When given negative feedback following errors, participants who believed in fixed intelligence demonstrated reduced memory-encoding activity in the temporal lobe. This suggests that one's attitude toward learning will affect the amount of effort placed in training tasks. This is particularly important for older adults with MCI, whose motivation toward the task may be increased by the prospect of preventing further cognitive decline.

Novelty

The novelty effect is the tendency for improved performance due to interest in a fresh experience, rather than the content. A review of educational research found that when not controlled, novelty effects create an increase in test scores. On average, performance increases due to novelty by 50% of a standard deviation for the first 4 weeks, and by 20% of a standard deviation after 8 weeks (Clark and Sugrue, 1991). Novel experiences may also simply be remembered better than familiar ones. On tasks of explicit recognition, subjects show higher accuracy in identifying novel words from a studied list than familiar words viewed multiple times (Tulving and Kroll, 1995; Habib et al., 2003). Given that CCT consists of tasks not encountered in everyday life, and prior familiarity with training protocols is grounds for excluding participants, there is ample opportunity for sheer novelty of the task to constitute a considerable portion of any measured effects.

Active controls

To address these issues, studies have employed active controls, where the comparison condition completes an activity designed to account for non-specific factors. This type of control condition aims to determine whether the benefits of such mental exercise are unique to CCT or can be obtained by any stimulating activity. Commonly used active controls include crossword puzzles, word searches, newspapers, and questionnaires. Active controls alleviate ethical issues of placebo-controlled treatments, as both the treatment and comparison condition may expect improvement. Although active controls are methodologically superior to waitlist conditions, differences in effect sizes between active control conditions and passive control conditions are not consistently observed (Karbach and Verhaeghen, 2014; Au et al., 2015), and the nature of the active control task may not account for all non-specific effects. Given the diversity of CCT paradigms and quantity of non-specific factors, researchers have argued that there is no universally applicable active control condition for all design cases (Boot et al., 2013). Instead, the control condition must be reasonably related to the tasks of the CCT condition such that participants expect improvement in the same cognitive domain regardless of their group assignment.

Table 1 contains a list of studies of CCT in patients with mild cognitive impairment (MCI). Most studies use waitlist control conditions (Rozzini et al., 2007; Finn and McDonald, 2011) or control conditions that do not account for engagement and motivation in the task (Galante et al., 2007; Talassi et al., 2007; Gagnon and Belleville, 2012; Herrera et al., 2012; Carretti et al., 2013; Gaitan et al., 2013). In such designs, the treatment condition is at an unfair advantage because patients assigned to CCT have a greater chance of improvement simply because the tasks are engaging and motivating whereas those assigned to waitlist control lose interest and motivation. Even when active control conditions have been used in previous studies of CCT in MCI, they are not consistently computerized (Talassi et al., 2007; Herrera et al., 2012; Carretti et al., 2013; Gaitan et al., 2013) and consist of CCT tasks of invariable complexity (Gagnon and Belleville, 2012; Gaitan et al., 2013). Comparing CCT that scales in difficulty with participant performance to CCT that remains at a fixed difficulty does not determine whether effects are specific to CCT or can be obtained with any engaging computerized game. Further, trials of scaling difficulty fail to take into account the novelty of experience they introduce, beyond just harder problems.

Table 1

Studies of CCT for MCI.

Study name	Treatment condition	Control condition	Control Weaknesses
Barnes et al., 2009	Processing speed and auditory processing CCT (n = 22)	Audio books, reading online newspapers, visuospatial computer games (n = 25)	Heterogeneous control tasks, not matched for engagement
Carretti et al., 2013	Working memory CCT (n = 10)	Memory discussion and questionnaires (n = 10)	Control group not computerized, unequal socialization opportunities
Finn and McDonald, 2011	Attention, processing speed, visual memory, and cognitive control CCT(n = 12)	Waitlist (n = 13)	Non-active control condition
Gaitan et al., 2013	CCT + pen and paper cognitive training (n = 37)	Pen and paper cognitive training (n = 23)	Control group not computerized, not matched for engagement
Galante et al., 2007	Attention, memory, language, abstract reasoning, and visuo-spatial CCT (n = 7)	Life history and current event discussion (n = 4)	Heterogeneous control tasks, not matched for engagement, not computerized, small sample
Gagnon and Belleville, 2012	Variable priority divided attention tasks (n = 13)	Fixed priority variable attention tasks (n = 13)	Does not account for engagement, novelty
Gooding et al., 2016	Milieu-based attention, memory, and executive function CCT (n = 23) Attention, memory, and executive function CCT (n = 31)	BrainAge, Sudoku, crosswords (n = 20)	Heterogeneous control activities, different CCT vs. control interface
Herrera et al., 2012	Short-term memory CCT (n = 11)	Fact searching, crossing out letters, sentence construction (n = 11)	Heterogeneous control tasks, not entirely computerized, not matched for engagement
Rosen et al., 2011	Processing speed and auditory processing CCT (n = 6)	Audio books, reading online newspapers, visuospatial computer games (n = 6)	Small sample size, heterogeneous control tasks, not matched for engagement
Rozzini et al., 2007	Attention, memory, language, abstract reasoning, and visuo-spatial CCT + cholinesterase inhibitors (n = 15)	Cholinesterase inhibitors only (n = 22) And Waitlist (n = 22)	Non-active control condition
Talassi et al., 2007	CCT, occupational therapy, and behavioral training (n = 54)	Physical rehabilitation, occupational therapy, and behavioral training (n = 13)	Heterogeneous control tasks, not matched for engagement, not computerized

The CCT conditions themselves vary considerably, both by targeted cognitive domains and whether they are administered alone (Galante et al., 2007; Barnes et al., 2009; Finn and McDonald, 2011; Rosen et al., 2011; Gagnon and Belleville, 2012; Herrera et al., 2012; Carretti et al., 2013; Gooding et al., 2016), or as part of a wider intervention (Rozzini et al., 2007; Talassi et al., 2007; Gaitan et al., 2013). If CCT is administered alongside additional therapies, the magnitude of participant's expectancy, engagement, and motivation may be greater than if they were treated with CCT alone. Such nuances in design will impact the magnitude of the observed differences between the training and control groups, and the measured effect sizes across trials. One needs to consider the nonspecific factors present in both the CCT and control conditions in order to accurately interpret the results.

Model control conditions

An adequate comparison condition must be matched for engagement, motivation, training time, computer interface, and novelty of stimuli. Preferably, the control group should account for nonspecific factors without introducing neurocognitive demands. If it does create specific effects, their impact on cognitive functioning should be known before comparing it to a CCT group. By using a CCT group, an active control group, and a passive control group, the relative contributions of each nonspecific factor can be estimated (Greenwood and Parasuraman, 2016). There is no “one size fits all” control condition. Instead, the selection of a control condition must be made in view of the content of the CCT platform and the specific research question.

An example of a well-balanced active control condition is a CCT program targeting domains of cognition that are different from those being targeted in the CCT program of the intervention condition. Should one group find greater improvement in cognitive domains of interest that transfer to untrained domains and quality of life, this can be taken as evidence that the content of CCT matters more than the non-specific factors. Participants in the active control group must expect the same types of cognitive benefits as those in the training group, principally by the control group and training group having identical descriptions of anticipated effects.

Another active control condition is one that incorporates non-adaptive versions of CCT. These programs use the same types of tasks as those in the experimental training condition, though they do not scale in difficulty with participant performance. When compared with adaptive versions of the same procedure, the tasks are balanced on participant expectancy and motivation, and practice effects can be ruled out. However, scaling difficulty introduces novelty, both in terms of activities encountered and strategies necessary for completing the task. Further, participants who complete the same task at unchanging difficulty may become less engaged as they reach their peak performance early. Controlling for practice effects is particularly important in older adults with cognitive impairment, given their propensity toward greater practice effects than their cognitively-stable peers (Suchy et al., 2011).

Remedial skills training has also been used as an active control condition. In these groups, participants discuss top-down strategies aimed at compensating with cognitive deficits. Such designs make it difficult to disentangle the plasticity of representations (knowledge, skills) from plasticity of processes (cognitive ability, brain function, brain structure, Fissler et al., 2015). It is possible that improvement following CCT is due to familiarity with neuropsychological tests and not due to improvement in underlying cognitive capacities. For example, learning to chunk numbers may improve digit span score, but may reflect one's ability to apply knowledge of cognitive strategies rather than one's actual attentional capacity. Comparisons between cognitive remediation and CCT are useful for balancing expectancy, motivation, and novelty, but offer little insight into the precise mechanism of any observed cognitive improvement.

Looking forward

Future studies would benefit from inclusion of measures to evaluate non-specific factors as covariates. For example, the User Engagement Scale measures aspects of engagement, usability, and satisfaction on a 5-point Likert scale and comprises both negative (“I felt annoyed when on this site,” “the game was confusing”) and positive (“I really had fun,” “It was really worthwhile”) items (Wiebe et al., 2014). The Immersive Experience Questionnaire provides ratings of temporal distortion, challenge, emotional involvement, enjoyment, and attentional involvement in the task (Jennett et al., 2008). Expectancy can be evaluated using the Credibility and Expectancy Scale (Devilly and Borkovec, 2000). The CES asks participants to report how logical the therapy seems, how successful they expect the treatment to be, and whether they would recommend the treatment to a friend. Two factors, credibility, and expectancy, have been found within the CES, with expectancy ratings successfully predicting treatment outcome in a randomized controlled trial of cognitive therapies for generalized anxiety disorders.

Until CCT can be found to improve cognitive and everyday functioning after accounting for each non-specific factor, its future as a treatment remains uncertain. What is more certain is the usefulness of being generally mentally active. Indeed, high levels of mental activity may be associated not only with higher cognitive performance, but reduced risk of dementia. A systematic review of 22 population studies found mental exercises may reduce overall incident dementia risk by 46% (Valenzuela and Sachdev, 2006). Whether CCT is the same as any other form of mental activity or represents a unique method for augmenting cognitive processes shall be a continuing topic of investigation.

Author contributions

JM: Manuscript conceptualization and design, writing of article. DD: Manuscript conceptualization and design, critical revision. PD: Manuscript conceptualization and design, critical revision. JS: Manuscript conceptualization and design, writing of article, critical revision. All authors have contributed to and have approved the final manuscript.

Conflict of interest statement

PD has received grants and advisory/speaking fees from several pharmaceutical and technology companies including antidepressant manufacturers. He owns stock in several companies whose products are not discussed here. DD has received fees for scientific advisory boards from AbbVie, Lundbeck, and Intracellular Therapeutics. The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.