A best-worst scaling experiment to identify patient-centered claims-based outcomes for evaluation of pediatric antipsychotic monitoring programs.

OBJECTIVE: This article employs a best-worst scaling (BWS) experiment to identify the claims-based outcomes that matter most to patients and other relevant parties when evaluating pediatric antipsychotic monitoring programs in the United States. DATA SOURCES: Patients and relevant parties, with pediatric antipsychotic oversight and treatment experience, completed a BWS experiment, including policymakers (n = 31), foster care alumni (n = 28), caseworkers (n = 23), prescribing clinicians (n = 32), and caregivers (n = 18). STUDY
DESIGN: Respondents received surveys with a scenario on antipsychotic monitoring programs and ranked 11 candidate claims-based outcomes as most and least important for program evaluation. DATA ANALYSIS: Stratified by respondent group, best-worst scores were calculated to identify the relative importance of the claims-based outcomes. A conditional logit examined whether candidate outcomes for safety, quality, and unintended consequences were preferred over reduction in antipsychotic treatment, the outcome used most often to evaluate antipsychotic monitoring programs. PRINCIPAL
FINDINGS: Safety indicators (eg, antipsychotic co-pharmacy, cross-class polypharmacy, higher than recommended doses) ranked among the top three candidate outcomes across respondent groups and were an important complement to antipsychotic treatment reduction. Foster care alumni prioritized "antipsychotic treatment reduction" and "increased psychosocial treatment." Caseworkers, prescribers, and caregivers prioritized "increased follow-up after treatment initiation." Potential unintended consequences of an antipsychotic monitoring program ranked lowest, including increased use of other psychotropic medication classes (as a substitute), increased psychiatric hospital stays, and increased emergency room utilization. Results of the conditional logit model found only caregivers significantly preferred other indicators over antipsychotic treatment reduction, preferring improvements in follow-up care (5.78) and psychosocial treatment (4.53) and reduction in prescriptions of higher than recommended doses (3.64).
CONCLUSIONS: The BWS experiment supported rank ordering of candidate claims-based outcomes demonstrating the opportunity for future studies to align outcomes used in antipsychotic monitoring program evaluations with community preferences, specifically by diversifying metrics to include safety and quality indicators.

What is known on this topic

While the importance of patient‐centered outcomes in health services research is increasingly recognized, a limited number of methods are available to identify patient‐centered outcomes in evaluating large‐scale systems’ interventions.

What this study adds

Drawing upon methods in marketing research, we propose a novel application of best‐worst scaling choice experiments to identify candidate claims‐based metrics prioritized by patients and four other stakeholder groups.

This article employs a best‐worst scaling experiment to identify the claims‐based outcomes that matter most to patients and other relevant parties when evaluating pediatric antipsychotic monitoring programs in the United States.

While the metric most frequently used in evaluations of antipsychotic monitoring programs (ie, “reduction in the number of children prescribed antipsychotic treatments”) was significantly preferred over some of the candidate metrics across all five groups, only caregivers alone significantly preferred metrics of safety and quality over the most frequently used metric of antipsychotic treatment reduction.

INTRODUCTION

Over the last decade, academic research and federal funding initiatives have encouraged researchers to extend the concept of patient‐centeredness from health care delivery, itself, to research. Incorporating preferences of patients and other relevant community members into outcomes research is especially important for complex program and system‐level innovations, given the multiple program objectives and potential unintended consequences. , To demonstrate opportunities to elicit the preferences of patients and relevant parties when evaluating complex system‐level innovations, we employ a preference elicitation strategy, referred to as a best‐worst scaling (BWS) choice experiment, to identify claims‐based metrics (ie, measures derived from administrative billing databases) that are most meaningful to patients and four relevant community groups. BWS choice experiments are used as a preference elicitation strategy for a variety of purposes in health and healthcare studies, including intervention development, public opinion on health policy reform, and caregiver preferences on treatment outcomes. , , Our BWS choice experiment investigates whether evaluations of complex system‐level innovations, specifically antipsychotic monitoring programs, employ the outcomes that matter most to patients and relevant parties. , , Antipsychotic monitoring programs are typically evaluated to assess reduction in antipsychotic treatment rates, but other claims‐based metrics are available to measure whether these programs impact the safety and quality of antipsychotic prescribing and potential unintended consequences. This study specifically seeks to investigate whether the metric routinely employed by evaluations of antipsychotic monitoring programs (ie, reduction in antipsychotic treatment) aligns with the metrics prioritized by young adults who are alumni of the foster care system (ie, foster care alumni) and relevant parties including caregivers, prescribing clinicians, caseworkers, and policymakers.

Antipsychotic monitoring programs arose nationally in response to the rapid growth of antipsychotic medication use among youth in the United States at the turn of the 21st century. During this time, the Government Accountability Office issued multiple reports highlighting the disproportionately high rates of antipsychotic prescribing among Medicaid‐insured youth and those in foster care, specifically. , For example, youth in foster care are more than twice as likely to be prescribed an antipsychotic medication than other Medicaid‐insured youth, even after controlling for diagnosis and sociodemographic characteristics. Beyond elevated rates of antipsychotic prescribing, specific safety concerns in management of these medications are also well‐documented, including concerns about antipsychotic co‐pharmacy (ie, concurrent use of multiple antipsychotic medications), lack of use of psychosocial treatments as the first line of therapy, and rates of prescribing among very young children (<6 years of age). , ,

In response to concerns around elevated rates of antipsychotic treatment among youth in foster care, , rapid expansion in antipsychotic monitoring programs occurred. Antipsychotic monitoring programs are implemented by public sector agencies or insurance programs to provide oversight of prescribing antipsychotic medications either prospectively or retrospectively. Prospective antipsychotic monitoring programs require the healthcare provider to receive treatment approval from the payer agent prior to point‐of‐sale dispensing. Prospective monitoring programs include prior authorization policies with or without a mandatory peer review. Retrospective monitoring programs include drug utilization reviews that occur after the prescription is dispensed, with records examined to identify trends in patterns suggestive of potentially inappropriate prescribing. , Implementation of prospective and retrospective antipsychotic monitoring programs is widespread. For example, prior authorization (PA) programs for pediatric antipsychotic treatment were implemented in Medicaid programs across 31 states by 2013. Moreover, 45 of the 50 states and DC had implemented a program that provided routine prospective or retrospective review of psychotropic medications, including but not limited to antipsychotic medications, for youth in foster care by 2011. While these monitoring programs are the most frequently studied approaches to decrease inappropriate antipsychotic prescribing, little consensus exists on what measures should be used to assess the effectiveness of these rapidly expanding strategies.

Multiple practice parameters have emerged to align pediatric antipsychotic prescribing with safety and quality parameters. These standards reflect emerging concerns around pediatric antipsychotic use, especially around increasing evidence of cardiometabolic abnormalities (obesity, hyperglycemia, and dyslipidemia) among youth. , , , , For example, the American Academy of Child and Adolescent Psychiatrists (AACAP) endorsed practice parameters for antipsychotic medications that provide recommendations to start “low and go slow,” with routine monitoring of side effects for metabolic conditions, such as body mass index, fasting blood glucose, hemoglobin A1c [HbA1c], and fasting lipid profiles, as did the Treatment of Maladaptive Aggression in Youth (T‐MAY) guidelines published in Pediatrics in 2012. , , Recommendations also emphasize the limited safety and efficacy data available in prescribing two or more antipsychotic medication concomitantly (hereafter, “antipsychotic co‐pharmacy”), and three or more different classes of psychotropic medications concurrently (hereafter, “cross‐class polypharmacy”) and recommends avoiding such uses. Other available consensus statements also emphasize the use of psychosocial treatments as a first line of treatment, use of structured rating scales to gauge treatment response, monitoring of abnormal involuntary movements, among other clinical parameters. , , ,

Increased capacity of large administrative datasets has prompted development of claims‐based metrics aligned with these practice parameters. , For example, claims‐based metrics for antipsychotic treatment among children were developed by the National Collaborative for Innovation in Quality Measurement (NCINQ), a center of excellence funded under the Agency for Healthcare Research and Quality/Centers for Medicare & Medicaid Services Pediatric Quality Measures Program, from 2012‐2014. , The NCINQ suite of measures was developed and tested with guideline review and expert consultation, arriving at claims‐based metrics for safety concerns associated with inappropriate prescribing, including continuous antipsychotic co‐pharmacy and higher than recommended doses of antipsychotic medications prescribed. The NCINQ suite also includes quality indicators for antipsychotic management, including use of psychosocial care prior to or concomitant with initiation of antipsychotic treatment (hereafter, “provision of psychosocial therapy,”) follow‐up visits after antipsychotic initiation, and monitoring for cardiometabolic side effects. , , The investment and development of these metrics provide opportunity for adoption of new claims‐based metrics in studies evaluating antipsychotic monitoring programs. However, studies of antipsychotic monitoring programs largely rely on measurement of a reduction in antipsychotic treatment among youth rather than measurement of safety or quality of antipsychotic prescribing. Our study sought to examine whether patients and other relevant parties prioritize metrics to evaluate antipsychotic monitoring programs that extend beyond a reduction of antipsychotic utilization to indicators of safe and appropriate antipsychotic prescribing (hereafter, “safety indicators”), quality of antipsychotic prescribing and management (hereafter, “quality indicators”) and potential unanticipated consequences such as the substitution of antipsychotic medications with other medication classes (hereafter, “unintended consequences”).

However, systematic approaches to elicit the preferences of those directly affected by antipsychotic monitoring programs remain limited. Studies frequently employ Delphi method to develop consensus in outcomes measurement for policy and practice innovations. We add a new dimension to the current knowledge base with the BWS choice experiment, whose main objective is preference elicitation and differentiation, rather than consensus building. Grounded in random utility theory, BWS requires respondents to consider tradeoffs and to select one best and one worst attribute (eg, candidate claims‐based metric) among alternatives within a profile. This results in a relative ranking of candidate outcomes. In contrast, Delphi method generally quantify preferences through Likert scales that result in the potential for all values to be expressed as highly important (ie, ceiling effects) obscuring the ability to differentiate preferences across items. Second, BWS choice experiments occur as a single brief survey fielded to study participants. In contrast, the Delphi method is an iterative process in which attrition as the rounds progress can subject findings to selection biases and the iterative nature may enable investigators to mold opinions. ,

Therefore, this study employs best‐worst scaling to prioritize among the multiple candidate claims‐based metrics available to assess the impact of antipsychotic monitoring programs. Specific outcomes may hold greater value to key constituencies, justifying their use over other alternatives. The present study specifically seeks to investigate whether each of the five relevant parties prioritize evaluating antipsychotic monitoring programs with candidate claims‐based metrics for safety, quality, and potential unintended consequences over a “reduction in the number of children prescribed antipsychotic treatments,” the metric routinely employed in antipsychotic monitoring program evaluations.

METHODS

Drawing upon methods used in marketing research, we employ a BWS choice experiment to elicit preferences and prioritize candidate metrics in rank order to inform future comparative effectiveness studies of antipsychotic monitoring programs for children in foster care. BWS choice experiments are a relatively novel approach in healthcare, introduced by McIntosh and Louviere in 1992, that assesses the relative value of different attribute statements within a choice profile. Ross and colleagues demonstrate feasibility of BWS choice experiments in eliciting outcomes prioritized by caregivers who initiate medication use for a child with attention‐deficit hyperactivity disorder. We extend application of BWS choice experiments to identify candidate claims‐based metrics that matter most to relevant patients and relevant parties in assessing effectiveness of antipsychotic monitoring programs as a complex policy innovation. Participants completed the BWS choice experiment following administration of either a qualitative semi‐structured interview or focus group. The interview or focus group inquired on the experiences of respondents in engaging with system‐level interventions to address concerns regarding questionable antipsychotic prescribing among children in foster care (including delivery system enhancement, prescriber supports, and antipsychotic monitoring).

Study instrument

Study participants completed a demographic survey and the BWS choice experiment, both developed by our research team. Survey questions were fielded to assess relevant sociodemographic characteristics for each sample. While some sociodemographic characteristics (ie, sex, race/ethnicity) were consistently collected across respondent groups, other measures (eg, professional background, education) varied in response to the extant literature and differences in their relationship of respondent groups to antipsychotic prescribing. , , ,

As provided in the Appendix S1, the BWS choice experiment first introduced a written scenario detailing the well‐documented concerns regarding safe and judicious antipsychotic treatment among children in foster care, the emergence of antipsychotic monitoring programs as a response, and a description of how various outcomes might be assessed to determine program effectiveness. The BWS choice experiment was a 11‐question survey. The survey had 11 choice task profiles, each displaying four of the 11 attribute statements. For each choice task profile, respondents were instructed to select one attribute statement among the four that mattered most to them (best choice) if required to assess effectiveness of an antipsychotic monitoring program and to select one attribute statement that mattered the least to them (worst choice). The choice task profiles employed a balanced, incomplete block design in a single set form so that respondents were exposed to each attribute statement the same number of times and any two attribute statements appeared together the same number of times. Respondents therefore had equal probability of selecting an attribute statement across the 11 choice task profiles.

Candidate measures were selected from the National Committee for Quality Assurance (NCQA), the National Collaborative for Innovation in Quality Measurement (NCINQ), the Medicaid/Mental Health Network for Evidence‐Based Treatment (MEDNET), and measures used in prior studies investigating effectiveness of prior authorization programs. , The National Committee for Quality Assurance (NCQA) developed pediatric antipsychotic metrics that have been used in the Healthcare Effectiveness Data Information System (HEDIS), used by more than 90% of commercial insurers. , MEDNET was a multi‐state Medicaid quality collaborative with a public‐academic partnership housed at the Rutgers University Center for Health Services Research. Table 1 describes the respective domains, claims‐based metrics, operational definition, and the source.

TABLE 1

Claims‐based outcomes: candidate metrics

Category	Attribute statement	Claims metric	Source
Antipsychotic treatment	1. Reduced the number of children prescribed an antipsychotic medications.	The percentage of children and adolescents 0 to 21 y of age with any antipsychotic use.	NCINQ
Safety indicators	2. Reduced the number of young children aged 0‐5 prescribed antipsychotic medications.	The percentage of children and adolescents 0 to 5 y of age with any antipsychotic use.	NCINQ
	3. Reduced the number of children prescribed two or more antipsychotics.	The percentage of children and adolescents 0 to 21 y of age on any antipsychotic medication for 90 d or more during the measurement year who were on two or more concurrent antipsychotic medications for 90 d or more.	NCQA
	4. Reduced the number of children prescribed three or more medications to manage their mental health—antipsychotics and other medications.	The percentage of children and adolescents 0 to 21 y of age on three or more psychotropic medication for 90 d or more during the measurement year.	NCINQ
	5. Reduced the number of children prescribed antipsychotics with doses higher than recommended.	The percentage of children 0‐21 y on antipsychotic medication who received two or more antipsychotic medications with higher than recommended doses	NCINQ
Quality indicators	6. Increased number of children who are provided “talk” therapy, such as psychosocial services, or counseling before or just after starting an antipsychotic.	The percentage of children and adolescents 0 to 21 y of age newly prescribed antipsychotic medication who had documentation of psychosocial care 90 d prior through 30 d after the index prescription start date (IPSD)	NCQA
	7. Increased doctor's monitoring for potential side effects, like weight gain or high cholesterol.	The percentage of children and adolescents 0 to 21 y of age with an antipsychotic prescription who had metabolic screening.	NCQA
	8. Increased the number of children who continue to see their doctors after receiving an antipsychotic to assess whether the medication works and potential side effects.	The percentage of children 0 to 21 y newly prescribed antipsychotic medication who had one or more follow‐up visits with a prescriber within 30 d after the index prescription start date (IPSD)	NCINQ
Unintended consequences	9. Increased use of other medications (like mood stabilizers) to replace antipsychotics in order to avoid the tight monitoring of antipsychotics.	The percentage of children and adolescents 0 to 21 y of age with any mood stabilizer or anticonvulsant prescription fills in the calendar year.	MEDNET, Rutgers Center for Health Services
	10. Increased the number of children having overnight hospital stays for mental health.	Rate of psychiatric inpatient admissions per 1000 enrollee months among children up to age 21. This measure is calculated for three age groups: <1, 1‐9, and 10‐21	MEDNET, Rutgers Center for Health Services
	11. Increased the number of children with emergency room visits because they could not take medication they needed right away.	Rate of emergency department (ED) visits per 1000 enrollee months among children up to age 21. This measure is calculated for three age groups: > 1, 1‐9, and 10‐21.	MEDNET, Rutgers Center for Health Services

NCINQ: National Collaborative for Innovation in Quality Measurement. NCQA: National Committee for Quality Assurance. , MEDNET: A multi‐state Medicaid quality collaborative with a public‐academic partnership housed at the Rutgers University Center for Health Services Research.

Prior to pilot testing and fielding the quantitative preference survey, cognitive interviews of the drafted instruments were conducted with individuals from each of the relevant groups to increase construct validity of measures and identify strategies to reduce cognitive burden. With foundations in cognitive psychology and information processing theory, cognitive interview techniques were used to increase readability, comprehension, and relevance of information, improving the construct validity and reducing cognitive burden. Cognitive interviews provided an opportunity to verbalize thoughts, feelings, interpretations, and ideas and to suggest alternative wording. , Cognitive interviews were conducted with two members of Youth MOVE National, “a youth‐led national peer advocacy organization committed to promoting the growth and development of youth who are involved in mental health, juvenile justice, education, and child welfare systems,” a policymaker in a child welfare agency and a child and adolescent psychiatric consultant to a state child welfare agency. Following cognitive interviews, pilot testing was conducted with foster care alumni and caregivers to further assess comprehension and comfort level with the survey. Individuals who participated in the cognitive interviews or pilot testing did not participate in the study itself. Foster care alumni, in particular, indicated that selecting the most and least important measures to assess antipsychotic monitoring programs across four potential choices required considerable attention. To accommodate this challenge, the survey was administered to foster care alumni on a web‐based platform (allowing the research team to show and read the scenario and questions) so as to assist in reducing cognitive burden. For all other samples, surveys were administered electronically.

Study sample and recruitment

For this study, we purposefully sampled respondents with a diverse set of roles related to antipsychotic treatment decisions for youth in foster care and developing system‐level monitoring mechanisms to provide oversight for these treatment decisions. Accordingly, we recruited policymakers, foster care alumni, prescribing clinicians, caregivers, and child welfare caseworkers to participate in this study. The policymakers were mid‐level managers within the four states who informed the development and/or implementation of antipsychotic medication oversight mechanisms for youth in foster care. The four (4) states—Ohio, Texas, Washington, and Wisconsin—were selected based on their innovation and diversity of antipsychotic medication monitoring programs. A snowball sampling approach was used in which members of the advisory board panel from each of the respective states initially identified individuals who held greatest expertise on their respective state approaches to antipsychotic medication oversight resulting in respondent arriving from a variety of public sector agencies, as denoted in Table 2.

TABLE 2

Sample characteristics by respondent group

Variable	Respondent group
	Policymaker (n = 31)	Foster care alumni (n = 28) b	Case‐worker (n = 23)	Prescriber (n = 32)	Caregiver (n = 18)
Gender
Female, no. (%)	22 (71)	16 (57)	19 (83)	15 (47)	18 (100)
Race c
African‐American, no. (%)	0 (0)	10 (37)	6 (26)	3 (9)	5 (28)
White, no. (%)	31 (100)	10 (37)	15 (65)	25 (78)	13 (73)
Multi‐racial, no. (%)	0 (0)	4 (15)	0 (0)	0 (0)	0 (0)
Other, no. (%)	0 (0)	3 (11)	2 (9)	4 (13)	0 (0)
Ethnicity
Hispanic, no. (%)	0 (0)	9 (38)	1 (4)	0 (0)	0 (0)
Education
Masters degree [Excluding nursing]	8 (26)	N/A	10 (44)	0	5 (28)
Medical Doctor	12 (39)	N/A	0	24 (75)	0
Nursing [Masters or Bachelors]	0 (0)	N/A	0	8 (25)	0
Bachelor Degree	4 (13)	N/A	9 (39)	0	5 (28)
Other	7 (22)	N/A	4 (13)	0	8 (45)

We round to the nearest percentage value given the size of respective samples; accordingly, the percentage estimates for sociodemographic characteristics do not always total to 100% (because of rounding estimations).

Young adults who were research partners in this study requested minimal sociodemographic data be collected from alumni of the foster care system to reduce concerns around participant burden and to ensure confidentiality. Accordingly, data were not collected from foster care alumni regarding educational attainment.

Among the foster care alumni sample, race is missing for one participant. The descriptive statistics provided for race are therefore calculated from a denominator of 27 rather than 28 respondents.

The foster care alumni, prescribing clinicians, child welfare caseworkers, and caregivers were recruited into this study through different avenues. For the recruitment of foster care alumni, the team partnered with two youth advocacy organizations, Youth MOVE National and Foster Care Alumni of America. Staff at Youth MOVE National and Foster Care Alumni of America advertised the project to their national networks of members and affiliates, received inquiries from members, and subsequently assisted in screening for study criteria. In both cases, eligible youth were alumni of the foster care system, reported having had experiences with psychotropic medication treatment, English‐speaking, and 18 years of age or older. For the prescribing clinicians and child welfare caseworkers, members of the project advisory board suggested state level administrators in each state who could help advertise the project to those who met study criteria. To be eligible for this study, both prescribing clinicians and child welfare caseworkers needed to report having had experiences working with youth in foster care prescribed psychotropic medications, be English‐speaking, and 18 years of age or older. Finally, for the foster care caregivers, the advocacy group, National Foster Parents Association, assisted in the recruitment of potential participants who met study criteria; members of the respective advocacy organization assisted in the screening for the project. To be eligible for participation, caregivers needed to report having cared for youth in foster care prescribed psychotropic medications, be English‐speaking, and be 18 years of age or older. All participants except for the policymakers and child welfare caseworkers received Amazon gift cards for their participation. Due to state regulations, policymakers and caseworkers were prohibited from receiving compensation for their participation. All participants completed informed consent procedures as required by the designated Institutional Review Board at Rutgers University, the State University of New Jersey.

Administration of this quantitative survey occurred independently of data collection of the interviews and focus group studies allowing for loss‐to‐follow‐up. Attrition only occurred among caseworkers [23 of 26 caseworkers (response rate: 93%)] and caregivers [18 out of 20 caregivers (response rate: 88%)]; all recruited policymakers, foster care alumni, and prescriber participated in the BWS choice experiment. Stratified by respondent group, the sociodemographic characteristics of each group are provided in Table 2.

Analysis

Demographic characteristics are summarized using descriptive statistics. We then employ best‐worst scores to elicit preferences for each of the five respondent groups. Notably, prior research demonstrates that calculation of the best‐worst scores is strongly associated with revealed preferences and predict real behavior in a way comparable to more sophisticated regression models. , , To calculate the BWS, each choice task profile was coded into two variables, indicating the attribute statement (ie, candidate claims‐based metric) that was chosen as the best and as the worst. The statement selected as “best” and as “worst” received a score of 1 as best or worst, respectively. If the statement did not receive a score of 1 for the choice profile, the statement received a score of 0. For each respondent group, the best‐worst score was calculated as the sum of best selections subtracted by the sum of the worst selections across all respondents divided by the number of times each attribute statement was displayed (n = 4); this was then multiplied by the number of participants in the respective group. The best‐worst score when positive denoted that respondents selected the statement as “most important” more often than the null or “least important” and a negative best‐worse score indicated selection as “least important” more than the null or ”most important.” To test whether selected statements reflected stated priorities and were not chosen at random, a t test assessed whether scores differed significantly from 0. The complete set of results, including the best‐worst scores, upper and lower bound of the confidence interval (95%), sum of best and worst scores, as well as the P‐values are provided in Tables S2.1‐S2.5. To test whether candidate claims‐based were preferred significantly more than the metric for any reduction in antipsychotic treatment (as the reference group), we then estimated a conditional logit model. Results of the conditional logit models for each respondent group are available as Tables S6‐S10.

Community engagement

Throughout this study, a 19‐member advisory panel of policy leaders, professional organizations, foster caregivers, and foster care alumni was recruited and convened at least quarterly to inform the selection of the research topic, the study design and conduct, and dissemination of results. The advisory panel assisted in determination of candidate metrics, review of survey tool, face validity and interpretation of findings. The panel received a summary of the preliminary research brief used to assess face validity of findings (see Supplemental Materials‐4 for example); advisory board members’ assessment of findings and implications of the study are integrated into this article's discussion.

RESULTS

Prioritized claims‐based metrics

Table 3 presents average best‐worst scores for each candidate claims‐based metric by policymaker, foster care alumni, caseworker, prescriber, and caregiver. For policymakers, statements that ranked highest in the best‐worst scores were safety indicators, specifically reduction in the use of multiple classes of psychotropic medications concomitantly (hereafter, “cross‐class polypharmacy,” 0.33), reduction in co‐pharmacy (0.27) and reduction in use among very young children (0.23). For foster care alumni, statements ranking highest cut across the domains of utilization, safety and quality indicators, specifically prioritizing reduction of use among children (0.20), reduction in use among very young children (0.17), reduction in cross‐class pharmacy (0.11), and increase in provision of psychosocial therapies prior to or concomitant with antipsychotic treatment (hereafter, “provision of psychosocial therapies”; (0.10)). Caseworkers ranked quality and safety indicators highest, including improvement in follow‐up care (0.37) reduction in cross‐class pharmacy (0.33), and reduction in use among young children (0.27). For prescribing clinicians, attributes that ranked highest were safety indicators, including reduction in antipsychotic co‐pharmacy (0.34), reduction in cross‐class polypharmacy (0.23), and monitoring for potential cardiometabolic side effects (0.18). Finally, caregivers ranked safety and quality indicators highest, including reduction in children prescribed antipsychotics with doses higher than recommended (0.32), improvements in follow‐up care (0.31), and reduction in cross‐class polypharmacy (0.15). Across respondent groups, the number of statements that were significant (P < .05) ranged from 5 to 9.

TABLE 3

Best‐Worst Scores for 11 candidate claims‐based metrics to assess antipsychotic monitoring programs for children, by respondent group

Category and attribute statement	Policymaker (n = 31)	Foster care alumni (n = 28)	Case‐worker (n = 23)	Prescriber (n = 32)	Caregivers (n = 18)
Antipsychotic utilization
Reduced the number of children prescribed an antipsychotic medications.	0.07	0.20*	0.18*	0.06	−0.15*
Safety indicators
Reduced the number of young children aged 0‐5 prescribed antipsychotic medications.	0.23*	0.17*	0.27*	0.16*	−0.04
Reduced the number of children prescribed two or more antipsychotics	0.27*	−0.11*	0.04	0.34*	−0.14*
Reduced the number of children prescribed three or more medications to manage their mental health—antipsychotics and other medications.	0.33*	0.10	0.33*	0.23*	0.15*
Reduced the number of children prescribed antipsychotics with doses higher than recommended.	−0.01	0.06	0.15*	0.09	0.32*
Quality indicators
Increased number of children who are provided “talk” therapy, such as psychosocial services, or counseling before or just after starting an antipsychotic.	0.10	0.10	−0.24*	−0.06	0.14
Increased doctor's monitoring for potential side effects, like weight gain or high cholesterol.	−0.02	0.04	−0.04	0.18*	0.04
Increased the number of children who continue to see their doctors after receiving an antipsychotic to assess whether the medication works and potential side effects.	0.19*	0.07	0.37*	0.20*	0.31*
Unintended consequences
Increased use of other medications (like mood stabilizers) to replace antipsychotics in order to avoid the tight monitoring of antipsychotics.	−0.23*	−0.11	−0.20*	−0.34*	−0.07
Increased the number of children having overnight hospital stays for mental health.	−0.15*	−0.15*	−0.33*	−0.37*	−0.17
Increased the number of children with emergency room visits because they could not take medication they needed right away.	−0.28*	−0.38*	−0.54*	−0.50*	−0.39*

The best‐worst score was calculated as the sum of best selections subtracted by the sum of the worst selections across all respondents divided by the number of times each attribute statement was displayed (n = 4) multiplied by the number of participants in the respective group. Scores can be ranked from highest to lowest to reflect the order of importance.

Best‐worst score is statistically significantly different than zero, P < .05.

Notably, the three statements that ranked lowest for policymakers and prescribers were all in the domain of unintended consequences. For alumni, antipsychotic co‐pharmacy (−0.11) ranked among the lowest alongside the three candidate outcomes in the domain of unintended consequences. Caseworkers ranked provision of psychosocial therapies (−0.24) alongside two metrics within the domain of unintended consequences, specifically increased number of overnight hospital stays (−0.33) and increased number of emergency room visits (−0.54). Finally, caregivers ranked reduction in antipsychotic treatment (−0.15) among the lowest alongside two candidate outcomes within the domain of unintended consequences, specifically increased number of overnight hospital stays (−0.17) and increased emergency room visits (−0.39).

Findings from conditional logit models (Table 4) suggest that only caregivers significantly preferred any of the other metrics over a reduction in the number of children prescribed antipsychotic medications, specifically preferring the reduction in children prescribed antipsychotics with doses higher than recommended (3.64), improvements in follow‐up care (5.78), and provision of psychosocial therapies (4.53). However, our results suggest that multiple groups significantly prefer the metric of antipsychotic treatment reduction over others. As indicated in the column for conditional logit model for best choice (ie, CLB) in Table 4, policymakers significantly preferred a “reduction in the number of children prescribed antipsychotic medications” (ie, “antipsychotic treatment reduction”) over improvements in side effect monitoring (0.34) and the unintended consequence of an increased use of other medications as a substitution for antipsychotic treatment (0.23). Alumni of the foster care system, in contrast, significantly preferred the metric for antipsychotic treatment reduction over reduction in antipsychotic co‐pharmacy (0.21) and the unintended consequence of an increase in the number of emergency room visits (0.20). Case workers significantly preferred antipsychotic treatment reduction metric over reduction in antipsychotic co‐pharmacy (0.36), improvements in provision of talk therapy (0.33), reduction in side effect monitoring (0.23), and all three candidate outcomes in the domain of unintended consequences. Prescribers significantly preferred antipsychotic treatment reduction over all three candidate outcomes in the domain of unintended consequences.

TABLE 4

Metric preferences relative to the metric, “Reduced number of children prescribed an antipsychotic medication,” by Respondent Group—Conditional Logit Models

Category and attribute statement	Policymakers		Alumni		Case Workers		Prescriber		Caregivers
	CLB	CLW	CLB	CLW	CLB	CLW	CLB	CLW	CLB	CLW
Antipsychotic utilization
Reduced the number of children prescribed an antipsychotic medication.	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Safety indicators
Reduced the number of young children aged 0‐5 prescribed antipsychotic medications.	1.19	0.32*	1.01	1.18	1.04	0.47	0.87	0.38*	1.56	0.85
Reduced the number of children prescribed two or more antipsychotics.	1.16	0.44	0.21*	1.60	0.36*	0.78	1.25	0.22*	0.53	0.79
Reduced the number of children prescribed three or more medications to manage their mental health—antipsychotics and other medications.	1.75	0.49	0.79	1.60	1.19	0.58	1.30	0.82	2.59	0.54
Reduced the number of children prescribed antipsychotics with doses higher than recommended.	0.48	0.69	0.76	1.67	0.71	0.79	0.62	0.42*	3.64*	0.27*
Quality indicators
Increased number of children who are provided “talk” therapy, such as psychosocial services, or counseling before or just after starting an antipsychotic.	0.86	1.04	1.18	2.39*	0.33*	2.89*	0.64	1.42	4.53*	0.88
Increased doctor's monitoring for potential side effects, like weight gain or high cholesterol.	0.34*	0.83	0.66	1.76	0.23*	1.22	0.74	0.47	1.79	0.69
Increased the number of children who continue to see their doctors after receiving an antipsychotic to assess whether the medication works and potential side effects.	0.90	0.40	0.77	1.69	1.21	0.58	0.96	0.55	5.78*	0.62
Unintended consequences
Increased use of other medications (like mood stabilizers) to replace antipsychotics in order to avoid the tight monitoring of antipsychotics.	0.23*	3.25*	0.63	3.00*	0.33*	3.01*	0.19*	2.16*	1.68	0.97
Increased the number of children having overnight hospital stays for mental health.	0.51	1.71	0.46	2.92*	0.24*	3.20*	0.36*	2.36*	2.08	1.88
Increased the number of children with emergency room visits because they could not take medication they needed right away.	0.41	2.73*	0.20*	4.53*	0.24*	6.56*	0.19*	3.46*	1.07	2.48

In all models, reference is “reduction in number of children prescribed an antipsychotic treatment.”

Abbreviations: CLB, Conditional Logit Model for Best Choice; CLW, Conditional Logit Model for Worst Choice.

Odds ratio is statistically significantly different than one P < .05.

DISCUSSION

When asked to identify outcomes most meaningful in assessing the effectiveness of antipsychotic monitoring programs, respondent groups generally ranked safety indicators (eg, antipsychotic co‐pharmacy, cross‐class polypharmacy, higher than recommended doses) highly, suggesting that they are very important measures for inclusion in evaluations of antipsychotic oversight interventions to complement antipsychotic use measures. Unintended consequences were generally ranked lower than safety and use measures. The selected safety indicators were consistent with available practice parameters, , , , including the reduction of (a) antipsychotic use among very young children, (b) antipsychotic co‐pharmacy, (c) psychotropic cross‐class polypharmacy, and (d) use of antipsychotic medications at doses higher than recommended. Caseworkers, prescribers, and caregivers also prioritized the quality indicator for increases in follow‐up visits after antipsychotic initiation among the top three. Alumni of the foster care system alone prioritized a second quality indicator, the provision of psychosocial therapies, among the top three. A systematic evidence review conducted in 2018 identified only a few studies that evaluated impact of antipsychotic monitoring programs on antipsychotic safety indicators and no studies evaluating impact on follow‐up visits or provision of psychosocial therapies. In contrast, the current metric most frequently used in evaluating antipsychotic monitoring programs (ie, reductions in number of children prescribed antipsychotic treatments) was only prioritized among the top three by alumni of the foster care system. Therefore, our findings suggest that opportunities exist to diversify the outcomes used in antipsychotic monitoring program evaluations more closely with outcomes that matter most to relevant parties.

The heterogeneity of prioritized metrics across a diverse array of relevant parties holds important implications. Respondent groups, in some cases, prioritized examination of whether antipsychotic monitoring programs increased provision of psychosocial therapies and follow‐up visits. Notably, antipsychotic monitoring programs such as prior authorization programs frequently do not explicitly prioritize provision of psychosocial treatment or follow‐up visits as a primary outcome of their implementation efforts. However, prior research suggests only 65% of Medicaid‐insured youth in foster care and 29% of other Medicaid‐insured children received a psychosocial therapy within the measurement year of antipsychotic treatment. Furthermore, approximately 28% of Medicaid‐insured youth in foster care and 31% of other Medicaid‐insured children did not receive follow‐up visit within 30 days of newly starting antipsychotic medications. Accordingly, opportunities exist to target and measure improvement of these indicators in future evaluations of antipsychotic monitoring programs.

Foster care alumni prioritized the metric most commonly used, specifically “reductions in the number of children prescribed antipsychotic medications,” among the top three candidate metrics. However, only caregivers prioritized any of the other candidate metrics over antipsychotic treatment reduction, specifically preferring the reduction in children prescribed antipsychotics with doses higher than recommended, improvements in follow‐up care, and provision of psychosocial therapies. While preference for a specific safety or quality indicator over a reduction in antipsychotic utilization was not conclusive across respondent groups, our findings suggest that diversification of outcomes in antipsychotic monitoring program evaluation is necessary to be responsive to the heterogeneity of group preferences in outcome measurement. Prior studies have raised concerns that measuring only a reduction in antipsychotic utilization is not responsive to the potential underuse of antipsychotic medication prescribing when clinically optimal, a concern especially strong in rural areas with provider shortages. , The heterogeneity of safety and quality measures prioritized by respondent groups in the current study extend beyond examination of a reduction in antipsychotic utilization, alone, suggesting opportunities to diversify the outcomes used in evaluation of antipsychotic monitoring programs. , ,

At the same time, our study also suggests that efforts to measure unintended consequences associated with antipsychotic monitoring programs are a relatively low priority among our diverse set of respondent groups. Given prior literature establishing the potential for antipsychotic monitoring programs to hold unintended consequences, , our study sought to examine the relative importance of these efforts to measure these outcomes as opposed to utilization, safety, and quality indicators. We find that the unintended consequences were given lower priority across respondent groups. Our community advisory panel attributed these low scores to challenges in attributing changes in unintended consequences to the antipsychotic monitoring programs, alone. Respondents noted that multiple other factors (outside of the implementation of antipsychotic monitoring program) may influence treatment with other psychotropic medications, inpatient mental health hospital stays, or emergency room visits. Given the challenge in attribution to the program alone, our community advisory board members reported the relatively low priority placed on these outcomes to be congruent with their own perspectives. These concerns align with challenges identified by researchers in attributing changes in use of inpatient and emergency department care to concurrent policy changes, given the many other factors that shape trends in such utilization. The decreased ranking of unintended consequences may also reflect that the unintended consequences are themselves less vivid or cognitively available for participants to construe as a potential outcome of antipsychotic monitoring programs.

Our application of best‐worst scaling choice experiment presents several limitations worth noting. First, our study prioritized inclusion of diverse and relevant community groups to understand better the heterogeneity of preferences for outcomes used to assess effectiveness of antipsychotic monitoring programs. However, each of the respondent groups is highly heterogeneous within themselves, so caution is needed in generalizing to differences in the priorities held by the respective populations more generally. Second, our case scenario required that we balance the need for both accessible language and a clinically accurate description. To accommodate feedback from the cognitive interviews and instrument piloting, we provided the following description of antipsychotic medications, “Antipsychotic medications, like Abilify or Risperdal, are one type of treatment that may help manage challenging behaviors or mental illness.” Notably, the description of antipsychotic medications was broad to ensure accessible and non‐technical language; this decision may have not adequately equipped the respondents with a full understanding of the important and evidence‐based role antipsychotic medications play for specific mental health conditions, such as psychoses. The use of an alternative script that prioritized additional clinical information regarding the rationale for antipsychotic medications may have influenced prioritization of the claims‐based metrics. Finally, our selection of candidate outcomes was not comprehensive as we limited selection to claims‐based metrics; other outcomes not included in our best‐worst choice experiment may be more important to respondents. For example, alumni of the foster care system, caregivers, prescribers, and policymakers have expressed interest in whether antipsychotic monitoring programs support children in attaining functional outcomes including but not limited to behavior management, high school graduation rates, or workforce entry. However, these measures are not ascertainable in Medicaid claims‐based metrics and were therefore not included as candidate outcomes. Instead, our focus in this study was to bring patient and relevant community voices into metrics prioritization in the claims‐based datasets that are most frequently used to assess effectiveness of antipsychotic monitoring programs.

Although important to consider the preferences of patients and other relevant parties for outcome measurement, it is worthwhile to note other considerations be weighed as well, including incorporation of expert opinion and evidence‐based guidelines, as well as measurement of well‐documented inequities. For example, monitoring of cardiometabolic side effects did not rise to a priority in our study, but expert opinion, professional guidelines and claims‐based information on shortfalls in adherence to recommended practices suggest that particular attention to improving side effect monitoring may be warranted. Monitoring of glucose and lipid levels is endorsed as a quality indicator by the National Committee for Quality Assurance (NCQA) and professional guidelines, but evidence to date is that implementation of these practices has been slow. From 2004‐2006, only about one‐fifth to one‐third of antipsychotic‐treated children in Medicaid received both blood glucose and cholesterol testing, and in 2015, only 26.1% and 28% of children (aged 1‐17) prescribed antipsychotic medications received metabolic monitoring among the Medicaid‐ and commercially insured, respectively. Therefore, engagement of relevant community member's preferences to prioritize among multiple candidate outcomes is optimally informed by other considerations as well, including but not limited to the programs’ targeted outcomes and metrics where opportunities for improvement exist.

The present study suggests that BWS choice experiments can be a useful tool to align selection of outcomes to evaluate complex systems‐level innovations with preferences of patients and relevant actors. Our specific application of this methodology suggests the need for diversification of outcomes in evaluating antipsychotic monitoring programs to align fully with community preferences, as results suggest that both safety and use measures are important to some significant actors. If the conditional logit had indicated that relevant parties coalesced in prioritizing an alternative metric over the routinely used metric (ie, reduction in antipsychotic treatment), our study would have concluded that replacement of this claims‐based metric with another as the primary outcome would facilitate greater alignment. In contrast, if the conditional logit had indicated that relevant parties uniformly prioritized the metric already used routinely (ie, reduction in antipsychotic treatment), our study would have provided further support for use of this metric as a primary outcome. Accordingly, we interpret the findings as supporting the diversification of outcomes in evaluations of oversight programs. We conclude that opportunities exist for extension in the use of BWS choice experiments to identify whether candidate outcomes of other complex systems‐level interventions align with the preferences of patients and other relevant parties.

Author matrix

Click here for additional data file.

Tables S1‐S10

Click here for additional data file.