Replication of Associations With Psychotic-Like Experiences in Middle Childhood From the Adolescent Brain Cognitive Development (ABCD) Study.

The fields of psychology and psychiatry are increasingly recognizing the importance of replication efforts. The current study aimed to replicate previous findings examining the construct validity and psychometric properties of a psychotic-like experiences (PLEs) measure in middle childhood using an independent subset of the baseline Adolescent Brain Cognitive Development (ABCD) sample. Using a remainder baseline sample of 7013 nine- to eleven-year-old children with complete data, we examined measurement invariance across race/ethnicity and sex, and examined the associations between the Prodromal Questionnaire Brief-Child Version (PQ-BC) and other measures of PLEs, internalizing symptoms, neuropsychological test performance, and developmental milestones, to determine whether previously obtained results replicated in this nonoverlapping baseline sample subset. The results replicated measurement invariance across ethnicity and sex, and analyses again found higher PQ-BC scores for African American (β = .364, 95% CI = 0.292, 0.435) and Hispanic (β = .255, 95% CI = 0.185, 0.324) groups. We also replicated that higher PQ-BC scores were associated with psychosis risk measures, higher rates of child-reported internalizing symptoms (Distress: β = .378, 95% CI = 0.357,0.398), neuropsychological test performance deficits (eg, working memory; Distress: β = -.069, 95% CI = -0.096, -0.042), and motor (Distress: β = .026, 95% CI = 0.003, 0.049) and speech (Distress: β = .042, 95% CI = 0.018, 0.065) developmental milestone delays. The current results replicated many findings from the original study examining the PQ-BC. We replicated evidence for mean differences in race/ethnicity, and associations with other PLE measures, greater internalizing symptoms, cognitive impairments, and developmental milestone delays. These findings indicate robust and reliable associations between PLEs and hypothesized correlates can be found in middle childhood nonclinical samples.

Introduction

Research is increasingly highlighting the importance of reproducibility in psychology and psychiatry.[1]These fields have been hit by what is sometimes coined the “replication crisis,” with one investigation finding that only ~39% of included studies had findings that replicate.[2] Part of the replication crisis is that many researchers are not always engaging in robust research practices,[3,4] including replicating their own findings. An analysis of the publication history in the top 100 psychology journals between 1900 and 2012 found that only ~1.6% of all publications were replication attempts.[5] Thus, either attempting to replicate findings is not currently standard practice, or these replication attempts are not being published. The current study aimed to replicate and extend results from our previously published findings[6] both to assess the robustness of those findings and to support the act of replication as an important practice in the fields of psychology and psychiatry.

Large datasets, such as the Adolescent Brain Cognitive Development (ABCD) Study, can be ideal for examining the reproducibility of findings. The ABCD study is particularly suited for this task since it first released a subset of the baseline data (ie, Data Release 1.0.1, N = 4524; hereafter referred to as the original sample), enabling any findings using this subset to be reproduced on the remainder of the baseline dataset (ie, replication sample). The current study aimed to replicate recent findings from the original ABCD sample, which examined the Prodromal Questionnaire-Brief Child Version (PQ-BC),[6] one of several validated self-report measures of psychotic-like experiences (PLEs) for use in childhood and youth.[7,8] PLEs are relatively common in children, with ~17% of 9- to 12-year-olds reporting PLEs.[9] PLEs are considered a dimensional,[10] transdiagnostic marker of psychopathology[11,12] that is not necessarily specific to psychosis risk, as only a subset of these children is at risk for conversion to psychotic disorders[13] or other psychiatric disorders in adulthood.[14] Notably, research indicates that self-reported PLEs, even those not confirmed with clinical interview, are still clinically relevant and associated with higher rates of psychopathology[15-17] and reduced functioning.[17,18] Several large cross-sectional datasets over the past 2 decades have examined the correlates of PLEs, finding associations with racial/ethnic minority status,[19] internalizing symptoms,[20-22] externalizing symptoms,[20] developmental impairments,[23] and cognitive impairments,[23,24] including reading,[25] working memory,[26,27] and processing speed[28] impairments.

The aforementioned ABCD original sample study examined the psychometric properties and validity of the PQ-BC, finding that childhood PLEs were associated with family history of psychosis, internalizing symptoms, cognitive deficits (eg, working memory), and motor and speech developmental milestone delays. The current study replicated the previous analyses, using the remaining ABCD baseline sample, to further examine the psychometric properties and construct validity of the PQ-BC. First, we aimed to replicate measurement invariance across ethnicity and sex and demonstrate mean-level differences across ethnicities and sex. We also aimed to replicate associations between PQ-BC scores and associations with psychotic risk measures (eg, family history of psychotic disorder),[29] greater internalizing symptoms, cognitive impairments, and developmental milestone delays.[19,30-34] Further, we aimed to extend the original study’s findings by examining whether associations replicated using parent-reported psychotic experiences.

Methods

Participants

A sample of 11 874 individuals was obtained from the ABCD study, a large-scale study tracking 9- to 11-year-olds recruited from 21 research sites across the United States. These data were accessed from the National Institutes of Mental Health Data Archive (see Acknowledgments). The ABCD study aimed to recruit a normative sample of children, recruiting children using probability sampling from both public and private elementary schools. Study-wide exclusionary criteria were as follows: child not fluent in English, MRI contraindication (eg, irremovable ferromagnetic implants or dental appliances, claustrophobia, pregnant), major neurological disorder, gestational age less than 28 weeks or birth weight less than 1200 grams, history of traumatic brain injury, or had a current diagnosis of schizophrenia, autism spectrum disorder (moderate, severe), mental retardation/intellectual disability, or alcohol/substance use disorder.[35,36] Parents provided written informed consent and all children provided assent.

First, the previously analyzed original sample (n = 3984)[6] was removed from the overall ABCD baseline (N = 10 977 with complete data; see supplementary material for information about missing data and sample details), leaving a replication sample of n = 7013 (48.1% female; 48.0% White, 17.6% African American, 21.2% Hispanic, 13.2% Other; age: M = 9.89 [SD = 0.63]).

Measures

Prodromal Questionnaire-Brief Child Version (PQ-BC)

Participants completed the Prodromal Questionnaire-Brief Child Version (PQ-BC),[37] a 21-item self-report questionnaire, modified for use with 9- to 11-year-olds based on a series of interviews assessing children’s understanding of the items, with a visual response scale included as a distress scale.[38] Consistent with previous research,[6,37] Total and Distress scores were calculated. The Total score is the sum of endorsed questions (ie, 0 = no, 1 = yes; range: 0–21). The Distress score is the total number of endorsed questions weighted by level of distress [ie, 0 = no, 1 = yes (but no distress), 2–6 = yes (1+score on distress scale); range: 0–126].

Internalizing Symptom Measures

The validated and computerized parent-reported and child-reported Kiddie-Structured Assessment for Affective Disorders and Schizophrenia (K-SADS) for DSM-5[39-41] were used in current analyses as measures of psychopathology.[35] The computerized self-administered parent and child versions of the K-SADS show good to excellent concordance with the clinician-administered computerized K-SADS.[42] For all K-SADS modules, participants were first administered a screening interview, then the supplement was administered if the participant was positive on the screening items. For participants not positive on the screener, the supplement items were assumed to be 0. As was done in the original sample, we examined internalizing symptoms using an internalizing symptoms composite (ie, summation of depression and GAD symptoms) and bipolar symptoms. We examined the Child Behavior Checklist (CBCL) internalizing and externalizing symptom measures,[43] with similar results. We also examined parent-reported externalizing symptoms using a K-SADS composite of current attention deficit hyperactivity disorder (ADHD), oppositional defiant disorder, and conduct disorder symptom summations.[39,40]

Psychosis Risk Measures

A measure of parent-reported child PLEs was created from 4 items from the CBCL.[43-45] The questions included: “Hears sounds or voices that aren’t there,” “Sees things that aren’t there,” “Strange behavior,” and “Strange ideas.” Each question was scored from 0 = not true, 1 = somewhat or sometimes true, and 2 = very true or often true. When only examining the first 2 questions, there were still significant associations with PQ-BC scores (although of lessened magnitude, Total score: β = .047, 95% CI: 0.025, 0.069, FDR P < .001; Distress score: β = .065, 95% CI: 0.042, 0.087, FDR P < .001).

We also examined a summation of parent-reported K-SADS current psychotic symptoms (see supplementary material for agreement with PQ-BC scores; a child-reported psychosis module was not administered due to concerns about participant increasing the length of the assessment for children).[39,40]

Lastly, the history of psychotic disorder, depression, and mania in first-degree relatives was assessed using the Family History Assessment Module Screener,[46] with each scored as either present or absent (family history of psychosis was an exclusionary criteria in the scoring of family history of depression and mania).

Neuropsychological Test Battery

Participants completed all tests within the National Institutes of Health Toolbox Cognitive Battery (NIHTB-CB).[47-49] The NIHTB-CB consists of 7 tasks, grouped into 2 composite scores. The fluid composite consists of Flanker Inhibitory Control and Attention, List Sorting Working Memory, Dimensional Change Card Sort, Pattern Comparison Processing Speed, and Picture Sequence Memory. The crystallized composite consists of Picture Vocabulary and Oral Reading Recognition test (see ref.[48] for descriptions of individual NIHTB-CB tests). The current study utilized uncorrected NIHTB-CB scores, but all analyses include age and sex as covariates.

Developmental Milestones

The parent assessment battery included questions of motor and speech developmental milestone delays.[43,50-52] The motor delays composite was coded as the summation of delays in attaining motor milestones [ie, rolling over (delayed = 6 mo or later), sitting (delayed = after 9 mo), walking (delayed = after 18 mo), parent-reported concern regarding motor development (parents were asked to compare their child’s development to that of other children: 0 = earlier, 1 = average, 2 = later), and parent-reported current child clumsiness[43] and scored from 0 = not true, 1 = somewhat or sometimes true, and 2 = very true or often true]. The speech delays composite was coded as the summation of a delay in speaking first word (delayed = after 12 mo) and parent-reported concern regarding speech development (0 = earlier, 1 = average, 2 = later). All motor and speech milestones were scored from 0=achieved within a typical timeframe or 1 = delayed.

Financial adversity was measured as the summation of endorsement of 7 parent-reported questions of financial difficulties experienced during the past 12 months from a demographic questionnaire (note in contrast to the original study, which used a measure of income-to-needs as a proxy of socioeconomic status, we included financial adversity to reduce data missingness). This measure included the following questions (participants received a score of 1 for each endorsed item, and financial adversity was scored as the sum of these endorsed items): (1) Needed food but couldn’t afford to buy it or couldn’t afford to go out to get it?, (2) Were without telephone service because you could not afford it?, (3) Didn’t pay the full amount of the rent or mortgage because you could not afford it?, (4) Were evicted from your home for not paying the rent or mortgage?, (5) Had services turned off by the gas or electric company, or the oil company wouldn’t deliver oil because payments were not made?, (6) Had someone who needed to see a doctor or go to the hospital but didn’t go because you could not afford it?, and (7) Had someone who needed a dentist but couldn’t go because you could not afford it?

Statistical Analyses

Prior to examining mean-level differences in PLEs across various demographic strata, it is important to determine whether an instrument and its underlying construct behave similarly across groups, otherwise known as measurement invariance.[53] To examine measurement invariance, stepwise tests of invariance were conducted, first examining potential differences across the 4 ethnicity groups and then comparing males and females.[54] We examined stepwise changes in fit (ie, chi-square, CFI, and RMSEA) of 4 progressively restrictive nested models, to examine whether model factor loadings, thresholds, and unique/residual variance were similar across both sex and race/ethnicity (see ref.[6] and supplementary material for additional measurement invariance model details). If the stepwise tests indicate measurement invariance across groups, mean comparisons can be meaningfully conducted across groups.[55] The remainder of analyses used hierarchical linear models (HLMs), with family and the 21 ABCD research sites treated as clustered observations in order to account for nonindependence of observations, conducted using the R lme4 package[56] (multcomp package for multiple comparison analyses[57]). These analyses modeled family unit and research site as random intercepts. Every model included age, sex, financial adversity, first-degree family history of psychosis, and race/ethnicity as covariates. HLMs analyzed the relations between Total and Distress PQ-BC scores (as well as parent-reported current K-SADS psychotic experiences) and: (1) ethnicity (White, African American, Hispanic, Other [note, Asian participants were included in the Other category to be consistent with original sample analyses]; False Discovery Rate (FDR)-corrected for multiple comparisons between ethnicity groups), (2) sex, (3) history of psychosis, depression or mania in first-degree relatives (FDR-corrected for multiple comparisons between family history groups), (4) psychosis risk measures (ie, parent-reported K-SADS psychotic symptom domain and parent-reported child PLEs derived from the CBCL), (5) K-SADS internalizing, bipolar, and externalizing symptom composites, (6) neuropsychological test performance (crystallized or fluid intelligence NIHTB-CB composites, then individual NIHTB-CB tests to examine the independent contributions of each test), and (7) motor and speech developmental milestone delays composites, then individually to examine the independent contributions of each developmental milestone. Results are expressed as standardized estimates (βs) with 95% CIs and FDR multiple comparison corrected. See supplementary tables 3 and 4 for both the unadjusted and adjusted estimates for each of these models. Due to significant skew and zero inflation of PQ-BC scores, we also examined log-transformed scores and negative binomial general linear mixed models (GLMMs), with the vast majority of results generally replicated using these alternative models (see supplementary material for details). Lastly, all findings from the original sample replicated when using the entire baseline ABCD sample, including finding hypothesized associations between PQ-BC scores and family history of psychosis in models with covariates and finding similar strength of associations.

Results

Basic Properties of PQ-BC

60.60% of the sample endorsed at least one PQ-BC question (original sample: 61.8%), with 42.7% reporting distress associated with at least one PQ-BC question (original sample: 43.0%). Both the PQ-BC Total (M = 2.60, SD = 3.55; original sample: M = 2.63, SD = 3.54) and Distress (M = 6.26, SD = 10.59; original sample: M = 6.19, SD = 10.43) scores showed high internal reliability (replication sample: Total Score α = .864; Distress Score α = .865; original sample: Total Score α = .863; Distress Score α = .873), which did not increase when any item was deleted. The correlation coefficients between each item and the PQ-BC Total score were 0.334–0.521, and for the Distress Score, 0.361–0.545 (original sample: Total score = 0.369–0.516, Distress Score = 0.382–0.535). For both the Total and Distress PQ-BC scores, the results of a CFA again confirmed the appropriateness of the one-factor structure for both the Total (CFI = 0.919, RMSEA = 0.023) and Distress (CFI = 0.921, RMSEA = 0.027) scores.

Measurement Invariance and Mean Comparison Analyses

Consistent with original analyses, stepwise tests examining invariance across both sex and race/ethnicity showed good fit in terms of CFI and RMSEA scores (for sex: CFIs ≥ 0.973; RMSEAs ≤ 0.030; for race/ethnicity: CFIs ≥ 0.973; RMSEAs ≤ 0.029), and minimal decrement in fit at all steps (ie, loadings, thresholds, and unique/residual variance are similar across both sex and race/ethnicity; for sex: change in CFI ≤ |0.006|; change in RMSEA ≤ |0.004|; for race/ethnicity: change in CFI ≤ |0.009|; change in RMSEA ≤ |0.006|). These models, therefore, passed all criteria to indicate strict invariance, suggesting that both the Total and Distress scores are measuring the same construct across sex and race/ethnicity. As shown in table 1, in terms of mean comparisons, consistent with the original sample, males showed higher PQ-BC Total scores. The replication sample also showed greater Distress scores among males (see table 2 for model estimates). We replicated that African American and Hispanic participants showed higher PQ-BC scores than the White group. In the replication sample, these groups had higher scores than the Other group and the Other group had higher scores than the White group, as shown in table 1 (see table 2 for model estimates).

Table 1.

Means and Standard Errors for PQ-BC Scores by Ethnicity and Sex for Both Original Sample and Replication Sample Analyses

	Original Sample (n = 3984)				Replication Sample (n = 7013)
	Total Score		Distress Score		Total Score		Distress Score
	M	SE	M	SE	M	SE	M	SE
Ethnicity
White	2.38	0.22	5.47	0.55	2.13	0.22	4.87	0.54
African American	3.14	0.28**	7.56	0.76**	3.48	0.23**	8.72	0.60**
Hispanic	2.89	0.25*	7.18	0.67**	3.00	0.23**	7.56	0.59**
Other	2.62	0.26	6.00	0.70	2.43	0.24*	5.72	0.62*
Sex
Male	2.73	0.21	6.20	0.55	2.95	0.21	7.04	0.54
Female	2.40	0.22**	5.84	0.56	2.55	0.22**	6.37	0.54**

Note: PQ-BC, Prodromal Questionnaire-Brief Child Version; n, sample size; M, mean; SE, Standard Error.

aEvery model included age, sex, financial adversity, first-degree family history of psychosis, and race/ethnicity as covariates.

*P < .05, **P< .01 (for the post hoc mean comparisons with White as a reference group for ethnicity; male as the reference group for sex).

Table 2.

Demographic and Symptom Measure Model Estimates for Both Original Sample and Replication Sample Analysesa

	Original Sample (n = 3984)						Replication Sample (n = 7013)
	Total			Distress			Total				Distress
	β	t	FDR P	β	t	FDR P	β	95% CI	t	FDR P	β	95% CI	t	FDR P
Sex
Male	−.092	−2.976	.003	−.035	−1.121	.26	−.113	−.157, −.068	−4.922	<.001	−.063	−.108, −.018	−2.718	.007
Race/ethnicity
African American	.213	3.523	.001	.201	3.297	.002	.381	.310, .451	10.604	<.001	.364	.292, .435	10.012	<.001
Hispanic	.142	2.811	.007	.164	3.247	.002	.246	.177, .315	7.007	<.001	.255	.185, .324	7.181	<.001
Other	.067	1.302	.19	.051	0.981	.33	.084	.012, .157	2.271	.02	.080	.007, .154	2.145	.03
Family history with covariates
Psychosis	.065	4.071	<.001	.068	4.272	<.001	.015	−.008, .038	1.268	.21	.015	−.008, .038	1.243	.21
Depression	.030	1.814	.10	.021	1.294	.29	.041	.017, .065	3.396	.003	.053	.029, .077	4.325	<.001
Mania	.023	1.446	.15	.014	0.861	.39	.020	−.004, .043	1.662	.15	.020	−.004, .043	1.667	.14
Family history without covariates
Psychosis	.075	4.697	<.001	.079	4.927	<.001	.023	.00004, .046	1.966	.049	.024	.001, .048	2.033	.04
Depression	.028	1.679	.09	.020	1.217	.27	.041	.017, .065	3.346	.002	.054	.030, .078	4.363	<.001
Mania	.027	1.679	.09	.018	1.114	.27	.026	.003, .050	2.172	.045	.026	.003, .050	2.178	.04
CBCL Parent-Report Child PLEs	.083	4.245	<.001	.082	4.693	<.001	.095	.073, .118	8.30	<.001	.106	.083, .128	9.087	<.001
K-SADS Parent-Reported Current Child Psychotic Symptoms	.065	4.336	<.001	.073	4.792	<.001	.046	.024, .069	4.084	<.001	.049	.027, .072	4.279	<.001
Internalizing symptoms	.311	21.607	<.001	.353	24.785	<.001	.331	.310, .352	30.857	<.001	.378	.357, .398	35.361	<.001
Bipolar symptoms	.200	3.939	<.001	.197	13.175	<.001	.272	.250, .293	24.884	<.001	.291	.270, .313	26.476	<.001
Externalizing symptoms	.076	3.528	<.001	.062	4.496	<.001	.094	.069, .118	7.627	<.001	.100	.075, .124	8.031	<.001

Note: β = standardized regression coefficient; t = t-test test statistic; p = P-value; FDR = False Discovery Rate-corrected for multiple comparisons; CBCL = Child Behavior Checklist; K-SADS = Kiddie-Structured Assessment for Affective Disorders and Schizophrenia.

aSignificant model estimates are in bold. See ref.[6] for 95% CIs for original sample models. Every model included age, sex, financial adversity, first-degree family history of psychosis, and race/ethnicity as covariates. Please see supplementary table 3 for both the unadjusted and adjusted estimates for each of these models.

Relations With Psychosis Risk Measures

As in the original analyses, first-degree family history of psychosis showed a small magnitude association with greater PQ-BC scores in models examining family history of psychosis, depression, and mania simultaneously but with no additional covariates (Total: β = .024, 95% CI: 0.001, 0.048; Distress: β = .024, 95% CI: 0.001, 0.047). However, in contrast to the original sample, when covariates were added (specifically race/ethnicity and financial adversity), the size of the effect diminished (table 2). Also, in contrast, in the replication sample, family history of depression showed a small magnitude association with greater PQ-BC scores with or without covariates (table 2), while it was not related in the original sample. However, consistent with the original sample, higher PQ-BC scores showed hypothesized relations to both greater parent-reported K-SADS current psychotic symptoms and greater CBCL parent-reported child PLEs (table 2).

Relations With Internalizing and Externalizing Symptoms

We replicated the finding of higher PQ-BC scores being associated with greater child-reported internalizing symptoms (table 2). We replicated the hypothesized relations between higher PQ-BC scores and greater scores on each of the child-reported internalizing symptom modules (Total: βs = .199–.304; Distress: βs = .230–.345), and bipolar symptoms. As with the original sample, for the replication sample, the sizes of correlations, in general, were much smaller for parent-reported child symptoms on the K-SADS (and for parent-reported GAD symptoms with PQ-BC Total scores, FDR-corrected P = .16), though the direction was the same (ie, internalizing composite: Total: β = .037; 95% CI = 0.015,0.059; Distress: β = .054; 95% CI = 0.031, 0.077; internalizing symptom modules: Total: β = .016–.045; Distress: β = .026–.064; bipolar symptoms: Total: β = .050; 95% CI = 0.027, 0.073; Distress: β = .069; 95% CI = 0.046, 0.092). As with the original sample, parent-reported current externalizing symptoms also showed a small magnitude association with PQ-BC scores (table 2). However, in contrast with the original sample, parent-reported externalizing symptoms showed stronger associations with PQ-BC scores than parent-reported K-SADS internalizing symptoms, Zs > 3.85, Ps < .001.

Associations With Neuropsychological Test Performance

As shown in table 3, we also replicated that both lower fluid and crystallized intelligence independently showed small magnitude associations with greater PQ-BC scores, and that specifically impairments in working memory were associated with greater PQ-BC scores (table 3). Further, in the replication sample, impairments in picture vocabulary showed a small magnitude association with greater PQ-BC scores.

Table 3.

Neuropsychological Test Performance and Developmental Milestones Delay Model Estimates for Both Original Sample and Replication Sample Analysesa

	Original Sample (n = 3984)						Replication Sample (n = 7013)
	Total			Distress			Total				Distress
	β	t	FDR P	β	t	FDR P	β	95% CI	t	FDR P	β	95% CI	t	FDR P
NIHTB-CB composite scores
Fluid	−.056	−3.221	.001	−.071	−4.038	<.001	−.060	−.086, −.034	−4.499	<.001	−.065	−.092, −.038	−4.807	<.001
Crystalized	−.061	−3.307	.001	−.063	−3.379	.001	−.084	−.111, −.057	−6.061	<.001	−.093	−.120, −.065	−6.595	<.001
Individual NIHTB-CB Tests
Flanker Inhibitory Control and Attention	.024	1.398	.23	.011	.622	.53	.011	−.015, .037	.846	.47	.010	−.016, .037	.763	.45
List Sorting Working Memory	−.042	−2.419	.049	−.051	−2.907	.03	−.062	−.089, −.036	−4.588	<.001	−.069	−.096, −.042	−5.051	<.001
Dimensional Change Card Sort	−.015	−.811	.49	−.018	−1.003	.44	−.004	−.031, .023	−.289	.77	−.010	−.038, .017	−.752	.45
Pattern Comparison Processing Speed	−.041	−2.353	.049	−.037	−2.129	.12	−.025	−.050, .001	−1.887	.14	−.015	−.041, .011	−1.151	.35
Picture Sequence Memory	−.008	−.516	.61	−.011	−.684	.53	−.017	−.041, .008	−1.343	.31	−.021	−.046, .003	−1.698	.21
Picture Vocabulary	−.028	−1.414	.23	−.034	−1.726	.15	−.077	−.106, −.048	−5.194	<.001	−.077	−.107, −.048	−5.177	<.001
Oral Reading  Recognition	−.043	−2.302	.049	−.036	−1.922	.13	−.012	−.039, .016	−.831	.47	−.018	−.046, .010	−1.267	.35
Developmental Milestones Delay Composite Scores
Motor	.057	3.627	<.001	.048	3.064	.002	.025	.002, .048	2.128	.03	.026	.003, .049	2.173	.03
Speech	.042	2.625	.009	.049	3.097	.002	.041	.018, .064	3.455	.001	.042	.018, .065	3.505	<.001
Individual Milestones
Motor
Roll Over	.010	.582	.71	.020	1.196	.50	.017	−.006, .041	1.436	.32	.016	−.008, .040	1.301	.45
Sit	−.009	−.510	.71	−.014	−.797	.50	.007	−.017, .032	.601	.64	.001	−.024, .026	.084	.99
Walk	.004	.261	.79	−.003	−.202	.84	−.016	−.040, .008	−1.338	.32	−.010	−.034, .014	−.801	.59
Subjective Motor Delays	.023	1.291	.20	.017	.941	.50	.005	−.021, .030	.357	.72	.000	−.026, .025	−.015	.99
Clumsiness	.055	3.638	<.001	.055	3.582	<.001	.024	.002, .047	2.107	.12	.036	.013, .058	3.074	.007
Speech
Speak First Word	.031	1.799	.25	.037	2.130	.12	−.011	−.036, .013	−.921	.50	−.013	−.037, .012	−1.012	.55
Subjective Speech Delays	.015	.866	.68	.015	.825	.50	.055	.030, .080	4.271	<.001	.058	.033, .084	4.495	<.001

Note: β = standardized regression coefficient; t = t-test test statistic; p = P-value; FDR = False Discovery Rate-corrected for multiple comparisons; NIHTB-CB = National Institutes of Health Toolbox Cognitive Battery.

aSignificant model estimates are in bold. See ref.[6] for 95% CIs for original sample models. Every model included age, sex, financial adversity, first-degree family history of psychosis, and race/ethnicity as covariates. Please see supplementary table 4 for both the unadjusted and adjusted estimates for each of these models.

Associations With Developmental Milestones Delays

Lastly, we replicated that delays in both motor (table 3) and speech milestone delays independently showed small magnitude associations with greater PQ-BC scores, and that specifically clumsiness was associated with greater PQ-BC scores (note after FDR-correction for Total scores, FDR P = .11). In the replication sample, subjective speech delays showed a small magnitude association with greater PQ-BC scores (see supplementary material for additional analyses using log-transformed scores and negative binomial GLMMs). Family history of psychosis was associated with PQ-BC scores even with the developmental delays in the same model.

Examining Associations Using Parent-Reported Current K-SADS Psychotic Experiences

Similar to the findings using PQ-BC scores, parent-reported current psychotic experiences showed small magnitude associations with family history of psychosis (β = .043, 95% CI: 0.020, 0.067, FDR P < .001) and family history of depression (β = .050, 95% CI: 0.026, 0.074, FDR P < .001), increased CBCL parent-reported PLEs (β = .262, 95% CI: 0.239, 0.284, FDR P < .001), being male (β = −.048, 95% CI: −0.094, −0.001, FDR P = .04), both greater parent-reported internalizing (β = .139, 95% CI: 0.116, 0.162, FDR P < .001) and externalizing (β = .139, 95% CI: 0.116, 0.163, FDR P < .001) symptoms, decreased fluid intelligence NIHTB-CB scores (β = −.047, 95% CI: −0.075, −0.020, FDR P < .001), and both motor delays (β = .075, 95% CI: 0.026, 0.074, FDR P < .001) and clumsiness (β = .088, 95% CI: 0.064, 0.111, FDR P < .001).

In contrast, there were several differences compared to the findings using PQ-BC scores. First, the sizes of the effects were reduced between parent-reported current psychotic symptoms with race/ethnicity (βs < 0.046, FDR Ps > .13; for mean differences, Zs < 1.50, FDR Ps > .55), child-reported internalizing symptoms (β = .003, 95% CI: −0.020, 0.026, FDR P = .80), crystallized intelligence (β = −.007, 95% CI: −0.035, 0.021, FDR P = .61) and working memory (β = −.010, 95% CI: −0.038, 0.017, FDR P = .65) impairments, as well as with speech delays (β = .014, 95% CI: −0.010, 0.038, FDR P = .26).

Discussion

The current study is the first to examine whether associations found using a subset of the ABCD baseline sample (Data Release 1.0.1) replicated in the remainder baseline sample (Data Release 2.0.1). The current study found important evidence supporting the validity of the PQ-BC, including replicating mean differences in sex and race, associations with several psychosis risk measures, cognitive impairments, and developmental milestone delays. PLEs were associated with both internalizing and externalizing scores, consistent with previous research,[21,31,32,58] and perhaps indicating both have roles in the etiology or early clinical manifestations of PLEs. The current study found associations similar in magnitude compared to the original sample (ie, βs ≤ .38). These findings have important implications for the measurement of PLEs and additionally highlight the importance of replication as a good research practice. Furthermore, the current replication study found associations similar to other large cross-sectional studies of PLEs in childhood and adolescence.[20,22-26] The current findings underscore that robust and reliable associations between PLEs and the aforementioned factors can be found in middle childhood manifestations of non-clinical psychosis spectrum symptoms.

An important deviation in findings from the original sample is that in the replication sample we did not fully find an association between PLEs and family history of psychosis. Specifically, when we added race/ethnicity and financial adversity to the model, first-degree family history of psychosis was no longer associated with PLEs, though it did in the original sample. This is consistent with previous research finding that belonging to a minority racial or ethnic group (especially African American or Hispanic groups in the United States) is associated with increased psychosis spectrum symptoms.[59,60] Speculative hypotheses for these associations include the possibility that increased psychosis spectrum symptoms may result from the increased exposure to stress, such as discrimination, associated with membership of a racial/ethnic minority (as well as potentially increased prevalence of cultural mistrust which may lead to heightened suspiciousness).[61] Likewise, financial adversity is also associated with increased psychosis spectrum symptoms, perhaps as a result of the increased parental distress, reduced community resources, increased toxin exposure, and other related factors associated with living in impoverished conditions,[62,63] or potentially because increased genetic risk for psychosis may result in lower socioeconomic status and/or lower parental academic achievement, thereby leading to greater financial adversity.

Nonetheless, the associations between family history of depression and PLEs found in the current analyses suggest less specificity in associations with familial risk for mental health disorders.[64] Furthermore, previous research suggests that PLEs can be nonspecific markers of psychopathology, with evidence that most children reporting PLEs never later develop a psychotic disorder.[13,65] Although in this study PLEs were associated with both parent-reported psychotic symptoms and a 4-item measure of PLEs, the fact that the family history of psychosis was not robust to full replication lends some support for the notion that PLEs can be nonspecific markers of psychopathology. Further, our finding the strongest association (βs > .33) was between PLEs and child-reported internalizing symptoms provides additional evidence that PLEs, at least in middle childhood, are nonspecific markers of psychopathology, as opposed to specifically markers of psychosis risk.[11,12] Future research should examine the stability of PLEs as future waves of the ABCD study are released and examine whether elevated PLEs over time are more specifically associated with increased family history of psychosis.

The replication sample also showed several other differences with the original sample. First, in the replication sample, we found consistent effects for sex across both Total and Distress scores, whereby males showed higher PLEs overall, whereas in the original sample males only showed higher Total scores. These findings are consistent with some previous research finding males show greater incidence of PLEs,[66] although note that other research has found the opposite.[67,68] We also found that in addition to African American and Hispanic participants showing higher PLEs than White participants (which is consistent with the original sample), participants in the “Other” group, which consisted primarily of biracial and Asian participants, also showed higher PLEs in the replication sample. In sum, the replication sample is even more consistent than the original sample in finding elevated rates of PLEs in non-white participants,[66,69] which may indicate that factors such as discrimination may already be influencing PLEs in middle childhood.[59,70,71] Furthermore, the current findings of biracial and Asian participants also showing elevated rates of PLEs is consistent with previous research on prodromal symptoms.[72]

An extension of the original study included examining whether results replicated using parent-reported psychotic experiences. The associations with these scores did not entirely overlap with self-reported PLEs, whereby parent-reported psychotic experiences were not significantly associated with several risk factors, including race/ethnicity, working memory, crystalized intelligence, or speech delays. This provides important evidence for regions of convergence and divergence between self-reported PLEs and parent-reported psychotic experiences. Further, the association between child-reported PLEs and parent-reported PLEs was quite small. A number of factors may explain the low associations both between child and parent PLEs, as well as the divergence in the associations found with child versus parent PLEs, including low observability by parents of certain types of PLEs, variability in parent-child relationships, as well as parental stress and symptoms.[73] Future research should parse the extent to which these discrepancies are the result of these aforementioned factors.

This replication found many of the same relationships between PLEs and cognitive function, including associations with both fluid and crystalized IQ and working memory. However, the replication sample also showed several differences compared to the original sample. For cognitive functioning, the associations between Total scores with processing speed or reading recognition scores were even further reduced. However, these cognitive impairments were not consistently associated with PLEs in the original sample (ie, not strongly associated with Distress scores), and therefore may not show robust associations with middle childhood PLEs. Processing speed impairments may be too subtle to be reliably detected middle childhood PLEs, given the heterogeneity of this group with only a small subset who will develop a psychotic disorder.

In addition, while the original study found some evidence of an association between Total PQ-BC scores and impairments in reading recognition, the replication sample did not replicate this finding. Instead, the current study found associations between both Total and Distress scores and a measure of receptive vocabulary (ie, picture vocabulary), consistent with previous research.[23,74-76] Notably, in the original sample, there was some limited evidence that impairments in picture vocabulary were associated with increased PQ-BC scores (ie, there was an association with log-transformed PQ-BC scores). Thus, overall, the original and replication samples showed evidence of associations between PLEs and crystallized intelligence impairments, with arguably more robust evidence for associations with a task examining receptive vocabulary. Furthermore, the replication sample also found associations between PLEs and subjective speech impairments for both Total and Distress scores. The findings for receptive vocabulary impairments and increased subjective speech delays supports a wealth of previous research finding language impairments in psychosis.[77] These findings indicate that both expressive and receptive language impairments are associated with increased PLEs, perhaps due to conceptual overlap with disorganized speech, or because these impairments contribute to disorganized speech.

The current study has several limitations. First, replication is particularly strong when findings are replicated in independent samples with differing methods and sampling techniques. The current study replicates findings from a subset of the baseline sample (n = 3984) with the remainder of the baseline sample (n = 7013). While these are independent in the sense of different participants, they use the same methods and sampling techniques. Thus, while a valuable contribution, these findings need to be replicated in a sample ascertained using different methods. Furthermore, in both samples there was a relatively high rate of endorsement of PLEs (approximately ~60% of sample endorsed at least one PLE, see supplementary table 2 for rates of endorsement for each individual question), indicating the possibility that the questions are tapping into constructs other than PLEs at this developmental stage. In addition, as with the original sample, associations with other self-reports were in the small-moderate range and associations with behavioral measures tended to be small (βs ≤ .15), although findings continued to be in line with the extant literature.[23,29,66,78] However, as was discussed in regards to the original sample, these generally small magnitude associations are expected given the nature of the sample (ie, a non-help-seeking sample without schizophrenia spectrum diagnoses), although we cannot rule out that the small magnitude associations are at least in part due to the presence of some false positives (ie, participants erroneously endorsing PLEs) and the inclusion of transitory PLEs that are inherent in assessing PLEs in a middle childhood population. However, assessing distress associated with PLEs in part helps avoid these false positives. While participants may misclassify other symptoms as distressing PLEs (eg, intrusive thoughts), endorsing these experiences as distressing PLEs is regardless a clinically relevant rather than a normative experience. It should also be noted that the strength of the associations found in the current study may change over the course of childhood into adolescence.[23,79] In addition, it does perhaps indicate that future work should take steps to strengthen the PQ-BC as a PLE measure. We recently began this work by conducting item response theory analyses and initial steps to create a screening form[80] to begin the work of creating a clinically useful measure.

In addition, another limitation is that the other psychosis risk measures were parent-report, though several of the associations found for child-reported PQ-BC scores replicated when examining parent-reported current psychotic symptoms. In general, the size of associations with parent-reported symptoms (eg, the children’s internalizing symptoms) tended to be smaller in size compared to the size of associations with other child-reported symptoms. Along these lines, parent-report of developmental milestones are subject to recall bias.[81] While the current study provides important evidence for the robust nature of several associations with PLEs using the PQ-BC, future research will be required to further validate the PQ-BC, including evidence that it is associated with ‘gold-standard’ interview measures, such as the SIPS,[82] as well as evidence for test-retest reliability. Follow-up research will examine the stability of these associations over time using future waves of the ABCD sample. Overall, the current study provided important further evidence for the validity of the PQ-BC, including indicating that many of the original PLE associations were robust to replication, with several new findings emerging in the even larger replication sample.

Funding

The ABCD Study is supported by the National Institutes of Health and additional federal partners under award numbers U01DA041022, U01DA041028, U01DA041048, U01DA041089, U01DA041106, U01DA041117, U01DA041120, U01DA041134, U01DA041148, U01DA041156, U01DA041174, U24DA041123, U24DA041147, U01DA041093, and U01DA041025. A full list of supporters is available at https://abcdstudy.org/federal-partners.html. A listing of participating sites and a complete listing of the study investigators can be found at https://abcdstudy.org/Consortium_Members.pdf. ABCD consortium investigators designed and implemented the study and/or provided data but did not necessarily participate in analysis or writing of this report. This manuscript reflects the views of the authors and may not reflect the opinions or views of the NIH or ABCD consortium investigators. The ABCD data repository grows and changes over time. The ABCD data used in this report came from DOI:10.15154/1503209. This work was supported by National Institute on Drug Abuse (U01 DA041120 to D.M.B. and K.J.S.); National Institute of Mental Health (MH014677 to N.R.K.; MH018261-31 to M.S.; R01 MH107108 to T.J.S.); Eunice Kennedy Shrive National Institute of Child Health and Human Development (R01 HD042974 to T.J.S.); and National Institute on Alcohol Abuse and Alcoholism (K05-AA017242 to K.J.S.).

Conflicts of Interest

R.L.L. is a Lundbeck International Neuroscience Foundation faculty member. T.J.S. is CEO of Cognivive, Inc. a digital neurotherapeutics company (there was no overlap between the content of this manuscript and Cognivive’s business interests). No other authors reported disclosures.

Click here for additional data file.