Development and assessment of a brief screening tool for psychosis in dementia.

INTRODUCTION: Hallucinations and delusions (H+D) are common in dementia, but screening for these symptoms-especially in busy clinical practices-is challenging.
METHODS: Six subject matter experts developed the DRP3™ screen, a novel valid tool to detect H+D in dementia, assessed its content validity through alignment with DRP reference assessments (Scale for the Assessment of Positive Symptoms-Hallucinations + Delusions, Neuropsychiatric Inventory-Questionnaire, International Psychogeriatric Association Criteria), and retrospectively investigated its ability to detect H+D in HARMONY trial (NCT03325556) enrollees.
RESULTS: All items from three reference assessments demonstrated significant agreement with the DRP3 screen among raters (P < .0001). Retrospectively applying the DRP3 screen to HARMONY identified all (N = 392) trial enrollees. DISCUSSION: The DRP3 screen, comprising three yes/no questions, is a content-valid tool for detecting H+D in dementia that aligned with current reference assessments and successfully identified trial participants when retrospectively applied to a completed trial. Within busy practice constraints, the DRP3 screen provides a brief tool for sensitive detection of H+D in patients with dementia.

BACKGROUND

Hallucinations and delusions are common neuropsychiatric symptoms in persons with dementia and can be due to underlying disease (referred to as dementia‐related psychosis [DRP]) or other superimposed medical conditions, such as delirium; yet they may go unrecognized and untreated, leading to cognitive, functional, and behavioral decline, and higher burden for patients and care partners . Although the rates of DRP vary, it is prevalent across dementia types. Overall, approximately 30% of people diagnosed with dementia in the United States have DRP and experience hallucinations and delusions , , , , , , , , , , , , , , . These symptoms may persist and recur over time , , , and patients who experience hallucinations or delusions are at greater risk of faster cognitive decline, institutionalization, and mortality , , , , . These symptoms have been linked to physical or verbal aggression, as well as significant distress, depression, and burden among care partners , . Hallucinations and delusions compromise quality of life for patients and care partners , , , and delusions are associated with greater functional impairment in Alzheimer's disease (AD).

The American Psychiatric Association guidelines recommend early and routine screening of patients with dementia for the presence of hallucinations and delusions associated with dementia as the foundation for managing these symptoms and their impact . Unfortunately, these symptoms may go unrecognized by clinicians, persons with dementia, and care partners due to low awareness of DRP; challenges associated with obtaining an accurate description of these symptoms from patients and care partners; limited time to elicit these symptoms; and lack of brief, sensitive screening instruments suited for busy practices.

The International Psychogeriatric Association (IPA) Criteria were developed to define psychosis in patients with major or mild neurocognitive disorders for clinical, epidemiologic, and research applications . Established tools for assessing hallucinations and delusions include the Scale for the Assessment of Positive Symptoms–Hallucinations + Delusions (SAPS–H+D), a subscale of the SAPS designed to help clinical investigators measure positive psychotic symptoms in individuals with schizophrenia or other psychotic disorders; the Neuropsychiatric Inventory (NPI) , and the Neuropsychiatric Inventory‐Questionnaire (NPI‐Q) , which covers a wide range of neuropsychiatric symptoms and is based on care partner reports , , . Although these tools and criteria have established a firm foundation for assessing patients who exhibit various neuropsychiatric symptoms, they do not provide a simple way for practicing clinicians to screen for the presence of hallucinations and delusions in dementia patients in a clinical setting.

The high prevalence and potentially serious adverse consequences of hallucinations and delusions associated with dementia underscore the need for their accurate, timely identification in a rapidly growing dementia population . To address this need, a consensus panel meeting with six experts in the fields of neurology, psychiatry, and geriatric medicine was convened in August of 2020 by Acadia Pharmaceuticals, Inc. The panel sought to develop a novel, practical screening tool that sensitively and efficiently detects hallucinations and delusions in patients with dementia; and is usable across a variety of care settings, including primary and long‐term care. Given the time constraints in a busy practice, the panel pursued the development of a brief tool in simple language (three yes/no questions) that could be administered by clinicians or by family care partners/professional caregivers to help clinicians detect these symptoms.

HIGHLIGHTS

The DRP3™ screen efficiently detects hallucinations and delusions in dementia.

The DRP3™ screen's content validity aligned with current reference assessments.

Retrospectively applied, the DRP3™ screen identified 100% of NCT03325556 enrollees.

RESEARCH IN CONTEXT

Systematic review: The authors developed and examined content validity of the DRP3™ screen through alignment with established reference tools and criteria (International Psychogeriatric Association [IPA] Criteria; the Scale for the Assessment of Positive Symptoms–Hallucinations + Delusions [SAPS–H+D]; and the Neuropsychiatric Inventory Questionnaire [NPI‐Q]) used to assess hallucinations and delusions in patients with dementia‐related neuropsychiatric changes. Inter‐rater reliability was evaluated using Fleiss’ kappa statistic.

Interpretation: Existing tools and criteria have established a firm foundation for assessment of patients exhibiting neuropsychiatric symptoms; however, they do not provide the efficiency needed by practicing clinicians to detect the presence of hallucinations and delusions in persons with dementia.

Future directions: The content alignment of the DRP3 screen with three reference assessments and its retrospective application to sensitively detect hallucinations and delusions in an existing trial population are strengths. The DRP3 screen may facilitate sensitive detection of psychosis in persons with dementia in busy medical practices.

In this report, we describe: (1) the development of the DRP3™ Screen, an innovative screening tool for detection of psychosis in patients with dementia in a clinical setting; (2) findings from a content validation alignment exercise designed to determine content alignment of the DRP3 screen questions with established DRP reference assessments (SAPS–H+D, NPI‐Q, and IPA Criteria); and (3) findings from a retrospective application of the DRP3 screen to detect the patient population that was identified for inclusion in a phase 3 clinical trial evaluating treatment for DRP (HARMONY, NCT03325556).

METHODS

This study does not involve human subjects. No informed consent was required.

DRP3 screen development

The DRP3 screen was developed by Acadia Pharmaceuticals, Inc. in collaboration with an international team of six subject matter experts (manuscript authors: JLC, ZI, BCD, CB, GG, AA) in the fields of neurology, psychiatry, and geriatric medicine. In August 2020, a consensus panel meeting on Improving Case Finding for Dementia‐Related Hallucinations and Delusions was convened. The subject matter experts were selected for the consensus panel (and the content validation alignment exercise) based on their expertise in diagnosis and management of psychosis in patients with dementia. The group engaged in a collaborative, iterative process to develop a novel, practical, simple screening tool to sensitively detect hallucinations and delusions in patients with dementia. The panel discussed the need for the tool to be face‐valid and usable across a variety of settings, including primary and long‐term care. Family and professional care partners often bridge the communication gap between patients with dementia and their clinicians. Thus, the panel sought to create a user‐friendly tool that allows an informant to help the clinician sensitively and efficiently detect hallucinations and delusions and to facilitate communication with health‐care professionals. The panel developed face‐valid preliminary draft questions for the screening tool, then engaged in an iterative process to refine the screening questions and ultimately determine a method to assess the potential for clinical use of this tool.

Content validation alignment exercise

To determine content validity—if, and to what extent, the symptoms itemized in currently used reference assessments or criteria (SAPS–H+D, NPI‐Q, IPA Criteria) would be captured by one or more of the DRP3 screen questions—the subject matter experts completed the content validation alignment exercise, in which each rater independently rated the level of alignment of each of the DRP3 screen questions with each of the following:

Eighteen items (H1–H6 and D1–D12) from the SAPS–H+D questionnaire,

Two questions from the NPI‐Q domains related to hallucinations and delusions, and

Two items from the IPA Criteria for Psychosis in Major or Mild Neurocognitive Disorder related to hallucinations and delusions.

Raters were instructed to use the following scale to indicate the extent to which the reference assessment would be captured by the DRP3 screen question:

Reference assessment items and questions from these assessments are provided in Supplement A in supporting information. The full set of instructions provided to each rater is provided in Supplement B in supporting information.

For each reference assessment item, ratings of the DRP3 screen questions were summarized using descriptive statistics; the mean rating (point estimate) with corresponding 95% confidence intervals (CI) was used to develop forest plots.

A conservative and discerning approach was a priori devised to demonstrate content alignment. For an item on a reference assessment or criteria to be considered content‐aligned, it was not sufficient that the rating's point estimate achieve a simple threshold of “YES” (i.e., ≥ 1.0) by a DRP3 screen question in capturing the reference item. It was necessary that the point estimate achieve a high threshold of at least “YES, most circumstances” as well as that the lower limit for 95% CI in this estimate be substantially greater than “YES, some circumstances.” The process was formalized as follows. The DRP3 screen was considered content‐aligned with a reference assessment item if: (1) the mean rating was ≥ 2.0 for at least one of the DRP3 screen questions; and (2) for the same pairing, the lower limit of the 95% CI was ≥ 1.3. If the standard error (SE) was 0 (i.e., all raters provided the same rating), the DRP3 screen was considered aligned with the reference assessment item if the mean rating was ≥ 2.0. Furthermore, to ensure that rater agreements were not due to chance, inter‐rater reliability of the content validation alignment exercise was assessed using the unweighted kappa statistic based on methodology described by Fleiss . To assess agreement of the six raters, the MKAPPA SAS Macro was used to determine kappa, SE, and P‐value across the three DRP3 screen questions for each of the three reference assessments (SAPS–H+D, NPI‐Q, and IPA Criteria). Observed agreement—that is, greater‐than‐chance agreement—is indicated by a kappa coefficient of > 0; if there is complete agreement between raters, the kappa coefficient is equal to +1 . A one‐sided P‐value for the test of the null hypothesis that kappa is 0 (against the alternative that kappa > 0) was computed.

Retrospective assessment of the DRP3 screen applied to the HARMONY trial population

HARMONY trial population

The HARMONY trial (NCT03325556) was a phase 3, double‐blind, placebo‐controlled randomized discontinuation trial that evaluated a treatment for hallucinations and delusions associated with DRP across a population of 392 patients. Patients with moderate to severe psychosis secondary to one or more of the following conditions were enrolled: AD, dementia with Lewy bodies, Parkinson's disease dementia, vascular dementia, and frontotemporal dementia spectrum disorders . The primary measure of psychosis in the trial was the SAPS–H+D.

Retrospective assessment of the DRP3 screen

Based on findings from the content validation alignment exercise with SAPS–H+D, a retrospective assessment of the ability of the DRP3 screen to detect the patient population included in the HARMONY trial was conducted.

If a patient in the HARMONY trial population had a positive response (baseline value of ≥ 2) to a SAPS–H+D item that was content‐aligned with the DRP3 screen (mean rater rating ≥ 2.0 for at least one DRP3 screen question, and for the same pairing, if the SE was not 0, a lower limit of the 95% CI of ≥ 1.3 in the content validation alignment exercise), the patient would have been identified by the DRP3 screen.

RESULTS

DRP3 screen

The DRP3 screen consists of three yes/no questions to help assess people with dementia for the presence of hallucinations and delusions:

Is the person, or has the person become, suspicious or fearful?

Is the person seeing, hearing, or sensing things that are false, unreal, or strange?

Is the person feeling/thinking or believing things that are false, unreal, or strange?

If the answer to at least one of these questions is “yes,” further evaluation of the patient is required to characterize the symptoms and determine whether they are due to dementia or other causes.

Alignment across reference assessments

All six raters completed the content validation alignment exercise.

Alignment with SAPS–H+D

Per the predefined criteria, alignment was observed between the DRP3 screen and 17 of the 18 SAPS–H+D items assessed (Figure 1, Table S1 in supporting information). For 11 SAPS–H+D items (H1, H2, H3, H4, H6, D3, D4, D7, D8, D9, and D10), the mean (SE) rating was 3.0 (0.00) for at least one of the DRP3 screen questions, indicating that all six raters gave the same rating of 3. For six SAPS–H+D items (H5, D1, D2, D5, D11, D12), there was a mean rating of ≥ 2.0 and the lower limit of the 95% CI was ≥ 1.3 for at least one of the DRP3 screen questions. For SAPS–H+D item D6 (somatic delusions), the mean rating was < 2 for two of the DRP3 questions and the lower limit of the 95% CI was < 1.3 for all DRP3 screen questions, indicating that item D6 was not considered content‐aligned with the DRP3 screen. As indicated by a kappa value of 0.37 and a P‐value of < .0001, the agreement on alignment between the SAPS–H+D items and the DRP3 screen among the six raters was both greater than chance and statistically significant (Table 1).

FIGURE 1

Summary of SAPS–H+D ratings for alignment with DRP3 screen. The DRP3 screen was considered content‐aligned with a reference item if: (1) the mean rating was ≥ 2.0 for at least one of the DRP3 screen questions; and (2) for the same pairing, the lower limit of the 95% CI was ≥ 1.3, as indicated by the gray‐shaded area. N = 6 raters. CI, confidence interval; SAPS‐H+D, Scale for Assessment of Positive Symptoms–Hallucinations + Delusions

TABLE 1

Summary of inter‐rater agreement of content alignment of reference assessments with the DRP3 screen; N = 6 raters

Reference assessment	kappa	SE	P *
SAPS‐H+D Ratings	0.37	0.02	<.0001
NPI‐Q Ratings	0.26	0.06	<.0001
IPA Criteria Ratings	0.45	0.07	<.0001

Abbreviations: IPA Criteria, International Psychogeriatric Association Criteria; NPI‐Q, Neuropsychiatric Inventory Questionnaire; SAPS–H+D, Scale for Assessment of Positive Symptoms–Hallucinations + Delusions; SE, standard error.

P‐value represents a test of the null hypothesis that kappa is 0 against the alternative that kappa > 0.

Alignment with NPI‐Q

Per the predefined criteria, alignment was observed between the DRP3 screen and both NPI‐Q questions (Figure 2A, Table S2 in supporting information). For NPI‐Q question 1 (delusions), there was a mean rating of ≥ 2.0 and the lower limit of the 95% CI was ≥ 1.3 for two DRP3 screen questions. For NPI‐Q question 2 (hallucinations), the mean (SE) rating was 3.0 (0.00) for one of the DRP3 screen questions, indicating that all six raters gave the same rating of 3. As indicated by a kappa value of 0.26 and a P‐value of < .0001, the agreement on alignment between the NPI‐Q questions and the DRP3 screen among the six raters was both greater than chance and statistically significant (Table 1).

FIGURE 2

Summary of (A) NPI‐Q and (B) IPA Criteria ratings for alignment with DRP3 screen. The DRP3 screen was considered content‐aligned with a reference item if: (1) the mean rating was ≥ 2.0 for at least one of the DRP3 screen questions; and (2) for the same pairing, the lower limit of the 95% CI was ≥ 1.3, as indicated by the gray‐shaded area. N = 6 raters. CI, confidence interval; D, delusions; H, hallucinations; IPA, International Psychogeriatric Association Revised Criteria for Psychosis in Major or Mild Neurocognitive Disorder; NPI‐Q, Neuropsychiatric Inventory Questionnaire

Alignment with IPA criteria

Per the predefined criteria, alignment was observed between the DRP3 screen and both IPA Criteria for defining psychosis in major or mild neurocognitive disorders (Figure 2B, Table S2 in supporting information). For the IPA Criteria for defining hallucinations, the mean (SE) rating was 3.0 (0.00) for one of the DRP3 questions, indicating that all six raters gave the same rating of 3. For the IPA Criteria for defining delusions, there was a mean rating of ≥ 2.0 and the lower limit of the 95% CI was ≥ 1.3 for two DRP3 questions. As indicated by a kappa value of 0.45 and a P‐value of < .0001, the agreement on alignment between the IPA Criteria and the DRP3 screen among the six raters was both greater than chance and statistically significant (Table 1).

HARMONY retrospective assessment

Retrospective application of the DRP3 screen to the HARMONY trial population resulted in a positive screening result for all (392/392, 100%) enrolled patients. Though there was no alignment between SAPS–H+D item D6 and the DRP3 screen, it did not impact the results of the retrospective analysis because no patients in the HARMONY trial population responded positively to SAPS–H+D item D6 only.

Use of the DRP3 screen

No permission is required to reproduce, translate, display, or distribute the DRP3 screen.

DISCUSSION

Dementia is becoming more common as the global population ages, and with this demographic shift, the number of patients with dementia‐related behavioral changes such as DRP will increase . DRP has adverse effects on patient and care partner quality of life , indicating that prevention, detection, and management of DRP are important avenues to mitigate the burdens of DRP both for persons who may manifest DRP and for those caring for them.

Psychosis is under‐recognized, and treatment is often delayed, creating unnecessary stress and burden for care partners. Scales are available to detect and characterize DRP, and are often used in clinical research; however, they are not necessarily designed for routine clinical practice and take more time and administration than is typically available to busy practitioners , . The DRP3 screen was created to address this care gap, and was developed for use by clinicians, family care partners, and professional caregivers in residential facilities.

We implemented four approaches to assess the validity of the DRP3 screen. First, we investigated the content alignment between the SAPS–H+D and the DRP3 screen. This revealed good alignment, with only one item (somatic delusions) falling below our high, and predefined, threshold for detection by the DRP3 screen. Second, we assessed the content alignment of the NPI‐Q, a neuropsychiatric assessment widely used in research and some subspecialty dementia practice settings, and the DRP3 screen . We interrogated the alignment between the DRP3 screen with the NPI‐Q question for delusions and the NPI‐Q question for hallucinations and found that both questions aligned with the DRP3 screen, as per our predefined criteria for detection. Third, we assessed the content alignment between the IPA Criteria, which focus on the presence of hallucinations and delusions, and the DRP3 screen . We found alignment between the Criteria and the DRP3 screen. Finally, we examined the ability of the DRP3 screen to identify patients with DRP who had been enrolled into a previous clinical trial of DRP (defined in that trial using the SAPS–H+D). We demonstrated that 100% of the patients who met entry criteria for that trial would have been identified as having DRP by the DRP3 screen based on the content validation alignment exercise. This final evaluation is important, as it applies the DRP3 screen to “real world” patients with DRP who were identified for a clinical trial. Collectively, these four exercises establish the potential for clinical application of the DRP3 screen.

Throughout this exercise, we took a conservative approach to demonstrating content alignment that did not simply dichotomize ratings into yes/no binary categories, but allowed greater discernment through four categories (No = 0; Yes, some circumstances = 1; Yes, most circumstances = 2, Yes, nearly/all circumstances = 3), and required a high mean rating threshold of at least “Yes, most circumstances” as well for the 95% CI lower limit of the mean rating to be substantially greater than “Yes, some circumstances.” Inter‐rater agreements were not due to chance, as statistically significant agreement (P < .0001, nominally) among expert raters was demonstrated using Fleiss’ kappa statistic for each reference assessment. The fair to moderate values (according to Landis and Koch ) observed for Fleiss’ kappa, ranging from 0.26 to 0.45 (Table 1), reflect our conservative and multi‐level ratings approach. For example, we used unweighted kappas—as opposed to weighted (e.g., linear or quadratic) kappas—that result in lower kappas when there are multiple ordinal categories (we had four); and when ratings are not equally probable (because of the face‐valid nature of our item‐question pairings a priori).

The DRP3 screen is not specific for the etiology of the psychosis present in patients with dementia. Instead, the DRP3 screen alerts the clinician or care partner to the presence of symptoms suggestive of psychosis. The symptoms then need to be evaluated with additional history, examination, and appropriate workup, which may include laboratory tests or neuroimaging to characterize the origin of the psychotic symptoms and determine whether they can be attributed to dementia or may have another etiology. Use of the DRP3 screen is not expected to be confounded by false positives. Once other causes of psychosis have been excluded, a patient's hallucinations or delusions may be ascribed to the dementia and appropriate management instituted.

The DRP3 screen does not generate a score. The three questions are answered as either “yes” or “no.” There is no benefit in summing the answers of the three questions. The content validation alignment exercise revealed that Question 1 from the DRP3 screen (Is the person, or has the person become, suspicious or fearful?) was never crucial on its own; however, we felt that it may be useful in certain clinical contexts (e.g., when frank hallucinations and delusions are more subtle and/or less obvious to care partners). Patients who meet the IPA Criteria for psychosis in a neurocognitive disorder will be detected by the DRP3 screen, as supported by our studies of the alignment of the DRP3 screen with the IPA definition.

Our goal in developing the DRP3 screen was to create an instrument that was sufficiently brief and straightforward to be realistically administered in busy clinical practices. Most clinicians in primary and specialty practice have little time for extensive assessments, so having effective, succinct tools is critical to their practical use . By developing questions that could be used by family members and residential caregivers, we sought to design an instrument that could assist others involved in the care of patients with dementia to aid in detecting DRP. After using the DRP3 screen, these non‐clinician observers may be better positioned to be cognizant of manifestations of DRP and to subsequently discuss DRP with the patient's clinicians. Similar tools have been shown to improve patient–clinician communication—without prolonging care visits .

Quality of life is poorer in patients with dementia and psychosis compared to patients with dementia who do not experience psychosis , . Patients with dementia and psychosis have increased health‐care use and higher care costs than those without psychosis . Effective identification of psychosis may facilitate pharmacologic and nonpharmacologic management to reduce patient and care partner distress, improve quality of life, delay use of residential facilities, and avoid excess disability associated with untreated neuropsychiatric symptoms . This study of the DRP3 screen has important limitations. The ratings were performed by dementia subspecialists in cognitive neurology, geriatric psychiatry, and geriatrics. These individuals are intimately familiar with the concepts, definitions, and terminology associated with psychosis in dementia; their performance in the content validation alignment exercises may not be representative of non‐subspecialist clinicians. We have not demonstrated the validity of the DRP3 screen when performed by care partners or professional caregivers. Why somatic delusions were not detected by the DRP3 screen is uncertain. Somatic delusions are uncommon and may occur in combination with other delusional or hallucinatory experiences; our observations suggest that these would be detected by the DRP3 screen. This is an area for further investigation. The content validity of the DRP3 screen with the SAPS–H+D, NPI‐Q, and IPA Criteria definition is a strength; however, these alignments were not explored prospectively in real‐world clinic populations, which include a broad range of psychosis severity levels. This reinforces the importance of the findings in the retrospective validity study of the DRP3 screen with the SAPS–H+D—in 392 patients who were assessed as having moderate to severe psychosis for a clinical trial involving treatment of DRP—in which the DRP3 screen demonstrated 100% sensitivity for detecting hallucinations and delusions.

In summary, we have developed a DRP screening tool that comprises three questions that assess the presence of hallucinations and delusions. Potential for clinical application of the DRP3 screen has been established through association with current reference assessments, with statistically significant agreement among raters, and through successful retrospective application for detecting DRP in an existing trial population. The DRP3 screen is intended to help clinicians sensitively and efficiently detect the presence of DRP; to encourage assessment of hallucinations and delusions in patients with dementia; and ultimately, to provide more proactive and better care for patients with dementia.

CONFLICTS OF INTEREST

Dr. Atri has received honoraria for consulting; participating in independent data safety monitoring boards; providing educational lectures, programs, and materials; or serving on advisory boards for AbbVie, Acadia, Allergan, the Alzheimer's Association, Axovant, AZ Therapies, Biogen, Grifols, Harvard Medical School Graduate Continuing Education, JOMDD, Lundbeck, Merck, Roche/Genentech, Novo Nordisk, Sunovion, Suven, and Synexus. He receives book royalties from Oxford University Press. Dr. Ballard has provided consultation to Acadia, Lundbeck, Otsuka, Janssen, Lilly, Orion, Biohaven, Roche, Enterin, Addex, NovoNordisk, and AARP. Dr. Cummings has provided consultation to Acadia, Alkahest, AriBio, Avanir, Axsome, Behren Therapeutics, Biogen, Cassava, Cerecin, Cortexyme, EIP Pharma, Eisai, GemVax, Genentech, Green Valley, Grifols, Janssen, Merck, Novo Nordisk, Ono, Otsuka, ReMYND, Resverlogix, Roche, Signant Health, United Neuroscience, and Unlearn AI pharmaceutical and assessment companies. Dr. Cummings owns the copyright of the Neuropsychiatric Inventory. Dr. Dickerson has provided consultation to Acadia, Alector, Arkuda, Biogen, Denali, Lilly, Merck, Novartis, Takeda, and Wave Lifesciences. He receives royalties from Cambridge University Press, Elsevier, and Oxford University Press. Dr. Foff was an employee of Acadia Pharmaceuticals, Inc., at the time of this study and during manuscript preparation. Dr. Grossberg has provided consultation to Acadia, Alkahest, Avanir, Axovant, Axsome, Biogen, BioXcel, Genentech, Karuna, Lundbeck, Otsuka, Roche, and Takeda. He has provided research support for Lilly, Roche, and the National Institute on Aging. He has served on a Speaker's Bureau for Acadia and Biogen, and has served on Safety Monitoring Committees for Anavex, EryDel, IntracellularTherapies, Merck, and Newron. Dr. Ismail has received personal fees from Lundbeck and Otsuka. His institution has received funds from Acadia, Biogen, Roche, and Sunovion. Dr. McEvoy was an employee of Acadia Pharmaceuticals, Inc., at the time of this study and during manuscript preparation.

Supporting information

Click here for additional data file.

DRP3TM screen are no circumstances in which a YES/+ on the reference item would result in a YES on the DRP3 question

In some circumstances, a YES/+ on the reference item would result in a YES on the DRP3 question

In most circumstances, a YES/+ on the reference item would result in a YES on the DRP3 question

In all or nearly all circumstances, a YES/+ on the reference item would result in a YES on the DRP3 question