The 2014 Ontario Child Health Study-Methodology.

OBJECTIVE: To describe the methodology of the 2014 Ontario Child Health Study (OCHS): a province-wide, cross-sectional, epidemiologic study of child health and mental disorder among 4- to 17-year-olds living in household dwellings.
METHOD: Implemented by Statistics Canada, the 2014 OCHS was led by academic researchers at the Offord Centre for Child Studies (McMaster University). Eligible households included families with children aged 4 to 17 years, who were listed on the 2014 Canadian Child Tax Benefit File. The survey design included area and household stratification by income and 3-stage cluster sampling of areas and households to yield a probability sample of families.
RESULTS: The 2014 OCHS included 6,537 responding households (50.8%) with 10,802 children aged 4 to 17 years. Lower income families living in low-income neighbourhoods were less likely to participate. In addition to measures of childhood mental disorder assessed by the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI-KID) and OCHS Emotional Behavioural Scales (OCHS-EBS), the survey contains measures of neighbourhoods, schools, families and children, and includes administrative data held by the Ministries of Education and Health and Long-Term Care.
CONCLUSIONS: The complex survey design and differential non-response of the 2014 OCHS required the use of sampling weights and adjustment for design effects. The study is available throughout Canada in the Statistics Canada Research Data Centres (RDCs). We urge external investigators to access the study through the RDCs or to contact us directly to collaborate on future secondary analysis studies based on the OCHS.

The past 30 years have witnessed changes in Canada’s demography, which may pose threats to the mental health of children and adolescents (herein child/ren). These changes include steady increases in income inequality,[1] family dissolution,[2] discrimination linked to visible minority status,[3] and neighbourhood poverty.[4] At the same time, the federal government has allocated substantial resources to child development initiatives (e.g., $3.5B between 2001 and 2007[5]) and provincial governments, such as Ontario, have developed strategies (e.g., Poverty Reduction, Comprehensive Mental Health and Addictions Strategy), created programs (e.g., Ontario Early Years Centres), and increased funding to children’s mental health and child welfare services. These government initiatives were a response in part to the concerns about the high levels of children’s mental health need identified in the 1983 Ontario Child Health Study (OCHS).[6,7] At present, nothing is known about the net impact of these demographic changes and government allocations on childhood mental disorders in Canada. The 2014 OCHS—a sequel to the original 1983 OCHS—was implemented to update our knowledge about the epidemiology of childhood mental disorders in Ontario and to inform policy decisions aimed at improving children’s mental health. The 2014 OCHS had 5 objectives, to:

Estimate the prevalence of childhood mental disorders in 2014;

Quantify changes in the prevalence of mental disorders between 1983 and 2014, and the extent to which they are associated with changes in socioeconomic disadvantage;

Evaluate the responsiveness of the healthcare system to child and youth mental health need;

Assess the burden associated with childhood mental disorders and their co-occurrence (e.g., societal costs, and loss of social and academic functioning);

Determine the potential influence of families, neighbourhoods, and schools on child and youth mental disorders and identify modifiable contextual variables to inform the development and evaluation of evidence-based prevention programs and policies.

Conceptual Framework

The conceptual framework used to select concepts for the 2014 OCHS combines Bronfenbrenner’s[8] ecological model of human development with the social determinants of health perspective.[9-11] In this framework, disorder results from adverse experiences arising from the interplay between individual characteristics and contextual-level variables found in neighbourhoods, schools, and families.[12,13]

Figure 1 depicts the relational structure of key contexts and concepts. For simplicity, neighbourhoods and schools were combined. In these contexts, we emphasized: 1) socioeconomic disadvantage, which may have an impact on children’s mental health and well-being through a lack of resources and opportunities (material pathways)[14] or negative responses to income inequality (psychosocial pathways);[15] and 2) assets (e.g., programs and services for families with children) and social processes (e.g., neighbourhood cohesion, antisocial behaviour), which may serve as positive or negative influences on health.

Figure 1.

Theoretical framework for the 2014 OCHS.

Child health and well-being are multidimensional concepts. The arrows in Figure 1 illustrate how contextual variables might influence child health. For example, policy makers are concerned about how children are affected by programs and services operating in neighbourhoods and schools. Do these services or “assets” show evidence of muting the adverse influences of social and economic disadvantage on child health? Although alternative strategies are available, these contextual questions are investigated optimally by selecting individuals nested in groups (cluster sampling) and analyzing responses using multilevel models (MLMs).

Methods

Concepts and Measures

In the 2014 OCHS, key concepts were measured at the neighbourhood, school, family, and individual levels. In addition to evidence of reliability and validity, 3 priorities guided our selection of measures: 1) maintaining comparability with the 1983 OCHS to assess changes in the epidemiology of childhood mental disorders; 2) drawing on multiple respondents and methods, including data linkage, to improve measurement scope and quality; and 3) including a structured interview to classify mental disorder based on the Diagnostic and Statistical Manual (DSM).

Neighbourhood/school level

We measured neighbourhood-level variables, such as the socioeconomic status (SES) and demography of residents, using aggregate information contained in dissemination areas (DAs) and census tracts (CTs) from the 2011 Canada Census (e.g., % of households led by a single parent). To quantify area-level resource allocations to children’s mental health, we obtained aggregate administrative data from the Ontario Ministry of Children and Youth Services (MCYS). Area-level social processes, such as neighbourliness, collective efficacy, and antisocial behaviour, were measured by standard questions and scales completed by parents participating in the 2014 OCHS and by interviewer ratings of the local environment, aggregated to the area level.

A sub-study, called the School Mental Health Surveys (SMHS),[16] was implemented to obtain information on school SES, demography, and school climate. Based on the location of households participating in the OCHS, we identified 359 schools likely to have 10 or more OCHS 4- to 17-year-olds in attendance and asked them to participate. We enlisted 248 (69%) schools into the SMHS with 2,266 OCHS children/youth in attendance. All students in grades 6 through 12 in these schools reported anonymously on 5 aspects of school climate; principals, teachers, and support staff assessed the school capacity to address student mental health needs; and the province’s Education Quality and Accountability Office (EQAO) provided administrative record data on school-level characteristics and student achievement test results. All student (n = 31,124), teacher (n = 3,373), and principal (n = 206) assessments collected within schools were aggregated to the school level, combined with administrative record data, and linked to the survey responses of individual 2014 OCHS participants in those particular schools.

Family level

Standard questions taken from the 2011 Canada Census were used to measure the SES and demographic characteristics of the family, (e.g., parental education, household income, family structure, race/ethnicity, language spoken in the home). We also used standard questions and scales to measure characteristics indicative of parental capacity and family processes.

Individual child level

The 2014 OCHS collected assessment data on childhood mental disorder, physical health, social competence, and academic functioning for all children, and self-reported adolescent experiences and behaviour for those 12 years and older.

Mental disorder

To classify the most common mental disorders occurring in the past 6 months based on DSM-IV-TR (attention-deficit hyperactivity disorder, oppositional defiant disorder, conduct disorder, major depressive episode, separation anxiety disorder, generalized anxiety disorder, social phobia and specific phobia), we used a modified version of the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI-KID).[17] To complement the classifications of disorder measured by the MINI-KID, we included the OCHS Emotional Behavioural Scales (OCHS-EBS) developed to measure these same disorders.[18,19] To enable secular comparisons, identical measures of 3 disorders (conduct disorder, hyperactivity and emotional disorder) included in the 1983 OCHS[6,7] were embedded in the OCHS-EBS.

Physical health

To classify child functioning on 8 health attributes (vision, hearing, speech, mobility, dexterity, feelings, cognition and pain) and provide an overall numerical estimate of health on a 0 to 1 scale, we used the Health Utilities Index Mark III.[20] In addition to collecting survey responses on chronic medical conditions or illnesses lasting more than 6 months, the 2014 OCHS used linkage with the Ontario Health Insurance Plan records to corroborate disease identification.

Social competence and academic functioning

The 2014 OCHS collected information on child friendships, interpersonal functioning, bullying, and maltreatment. Administrative record data held by the Ministry of Education in the Ontario Student Information System (OnSIS) was used to capture assessments of individual students over the previous 3 years: 1) student achievement (grades in English and Math and EQAO scores); 2) status (identification as a student with exceptionalities and use of special programs at any time; e.g., compensatory programs, English as a Second Language [ESL]); and 3) behaviour (attendance, expulsion, suspensions).

Service utilization, barriers to services, and satisfaction with service

Extensive information was collected from 2014 OCHS participants on the use of children’s health services by provider type (children’s mental health,[21] child welfare, juvenile justice, family physicians) and location where services were accessed (e.g., emergency rooms, urgent care, etc.). Information on services received by individual children was also abstracted from administrative record data held by the Ministry of Health and Long-Term Care (MOHLTC) from 1998 to 2017. These included the use of physician services (Ontario Health Insurance Plan), outpatient services (National Ambulatory Care Reporting System), and inpatient care (Discharge Abstract Database).

Survey Design

The target population included all children aged 4 to 17 years whose usual place of residence was a private household in Ontario. The sampling unit consisted of all households occupied by families with 4- to 17-year-olds listed in the 2014 Canada Child Tax Benefit (CCTB) file. The sampling frame was the 2014 CCTB file. The sample selection was done by stratified, clustered, and random sampling of households from the CCTB file. In the 2011 Census, there were about 2 million children and adolescents in this age range.[22] According to the 2011 Census, about 63% of Aboriginal children live in households off reserve[23] and were eligible for inclusion, while those living on reserves were not. The CCTB file was used because of evidence that it provides a more reliable and efficient frame for sampling 0- to 17-year-olds than other options, such as the Census and birth registries.[24]

The survey design (Figure 2) included stratification and cluster sampling of residential areas and siblings within families. Residential areas were defined by census tracts (CTs) in urban areas and dissemination areas (DAs) in “other” urban and rural areas. CTs are relatively stable geographic areas, akin to neighbourhoods of 2,500 to 8,000 individuals; they are located in census metropolitan areas and in census agglomerations that have a core population of 50,000 or more. DAs are small, relatively stable geographic units of 400 to 700 individuals drawn from one or more adjacent dissemination blocks; they are the smallest standard geographic area for which all census data are disseminated.

Figure 2.

Basic survey design for the 2014 OCHS. Areas (stage 2 rows) and households (stage 3 columns) are cross-classified by income. The bolded numbers in the grid are participating households (percent response), and the italicized numbers are participating children (percent response).[24] CT, Census Tract; DA, Dissemination Area; P80, above the 80th percentile of income.

Basic survey design for the 2014 OCHS. Areas (stage 2 rows) and households (stage 3 columns) are cross-classified by income. The bolded numbers in the grid are participating households (percent response), and the italicized numbers are participating children (percent response).[24] CT, Census Tract; DA, Dissemination Area; <P20, below the 20th percentile of income; P20-80, between the 20th and 80th percentiles of income; >P80, above the 80th percentile of income.

Based on the CCTB file, sampling was done in 3 stages. In stage 1, contiguous CTs and DAs throughout Ontario were combined to identify 1,102 primary sampling units (PSUs) having on average 1,066 eligible families (700 to 2,700). PSUs were classified as urban or rural and sub-classified according to family income based on the CCTB: below the 20th percentile of income, between the 20th and 80th percentiles of income or above the 80th percentile of income (<P20, P20 to P80; >P80). PSUs were selected using probability proportional to size (PPS); within a particular stratum, the chance of selecting a PSU was based on the number of eligible families in a particular PSU divided by the total number of families within the stratum. A total of 180 PSUs was selected: 153 from urban areas and 27 from rural areas. These PSUs were equally allocated (EA) across the strata, with 60 chosen from each income strata.

In stage 2, the contiguous CTs and DAs comprising each of the 180 PSUs were grouped separately into 2 sub-strata: those that were consistent v. those inconsistent with the income designation of the PSU. The 2 sub-strata contained a total of 939 super elements. Up to 4 super elements were selected from each PSU: 2 consistent with and 2 inconsistent with the income designation of the PSU. This resulted in the selection of 484 super elements for inclusion. The sub-strata were created to ensure that the geographical boundaries selected for study inclusion encompassed homogeneous economic areas. In PSUs with inconsistent CTs and DAs, both were sampled to enable adequate variance estimation at area levels.

In stage 3, within each selected super element, eligible households were stratified by family income in the CCTB file (<P20, P20 to P80; >P80). Within each of these income strata, equal numbers of households were selected using simple random sampling (SRS). About 75% v. 25% of households from any particular area were selected from the sub-stratum that was consistent v. inconsistent with the income designation of the PSU.

Sample and Response

Among the 15,796 households selected from the CCTB, 12,871 were eligible and 6,537 participated (50.8%). Cross-classified by area and family income, Figure 2 shows the number of households and children participating as a percent of those eligible. The numbers in the figure show a gradient of positive response from lower to higher income among households and areas.

To obtain unbiased estimates, Statistics Canada created survey weights based on the probability of selection (dwelling design weight) with adjustments for survey non-response and post-stratification. This ensured that the final survey weights sum to known counts of dwellings with children in Ontario.[25] Table 1 compares selected socio-demographic characteristics (weighted) of participants in the 2014 OCHS with population estimates derived from the 2011 National Household Survey. The most notable difference is for family income: while the mean level is lower in the OCHS (100.5 v. 106.3), the standard deviation is higher (162.6 v. 128.5).

Table 1.

Sample Characteristics.

Characteristics	Weighted Sample[95% CI]		Population Estimatesa
Super Elements n = 484
% families below the poverty   line	21.9	[19.9 – 23.9]	21.0
% single-parent families	22.5	[20.5 – 24.5]	24.4
% families living in a rented  dwelling	21.9	[19.5 – 24.4]	24.3
Families n = 6,537
Family income in $1,000 s (M)	100.5	[95.5 – 105.5]	106.3
% one or both parents born  outside Canada	44.6	[42.4 – 46.8]	43.2
% one or both parents visible  minority	35.4	[34.2 – 38.4]	35.7
% families living in a rented  dwelling	18.1	[16.6 – 19.6]	21.8
% rural	12.4	[11.1 – 14.0]	12.8
% small and medium urban	16.3	[14.5 – 18.1]	16.5
Children n = 10,802
% male	51.3	[49.6 – 53.0]	51.6
Age in years (M)	10.6	[10.5 – 10.8]	10.8

aNational Household Survey 2011.

CI, Confidence Interval.

Data Collection and Processing

The fieldwork for the 2014 OCHS was conducted by Statistics Canada, the federal statistical agency responsible for collecting and analyzing data at both the national and provincial levels, including the Canada Census and Labour Force surveys. Data collection took place from Oct 2014 to Sept 2015. Interviewers were assigned selected households listed on the CCTB file with one or more children aged 4 to 17 years in those residential areas sampled for the study. Interviewers telephoned or visited the household in person, asked to speak with the person most knowledgeable (PMK) about the household, presented the study, screened for eligibility, and, through the PMK, invited eligible families within these households to participate.

After collecting basic information on all household members and identifying the PMK (mothers in 88.3% of families), interviewers scheduled home interviews at times convenient to families. A common set of measures were used for up to 4 children aged 4 to 17 years (selected randomly in families with more than 4). In addition to these common measures, one of these children was identified randomly as the “selected child” who had enriched assessments that included the parent (of 4- to 17-year-olds) and youth (12- to 17-year-olds) versions of the MINI-KID.

Figure 3 shows informants linked with selected concepts (see supplemental Appendix for more detailed information). A computer-assisted personal interview (CAPI) with the PMK was used to obtain information about all participating children aged 4 to 17 years (i.e., birth history, physical health, service use, activities, and school); and the family (i.e., housing, immigrant/refugee status, and socio-demography). To facilitate disclosure, the PMK answered personal questions on a laptop about their substance use, personal mental health, and perceptions of neighbourhood characteristics. Parent assessments of childhood mental disorder were obtained by: 1) an interviewer-administered paper version of the MINI-KID about the selected child; and 2) a paper and pencil self-report checklist of emotional and behavioural problems applicable to all participating children (OCHS-EBS and items measuring the disorders in the 1983 OCHS). A paper and pencil questionnaire was used to keep the mode of data collection (structure, ordering and content) as similar as possible to the 1983 study. Finally, a paper and pencil questionnaire was left for the PMK’s spouse/partner to complete and return by mail (3,133 [62.1%] response among 2-parent households). This questionnaire included checklist assessments of the selected child’s emotional-behavioural problems, their impact on the family, and the physical and mental health of the spouse/partner, their parenting behaviour and childhood exposure to violence.

Figure 3.

Data sources and concepts for the 2014 OCHS. †link with Administrative data ‡report from spouse/partner.

All adolescents aged 12 to 17 years willing to participate in the study completed a laptop questionnaire in private. Youth were asked questions on different aspects of their health, school, social relationships and other activities, such as work and civic engagement. Modules on sensitive topics, such as anti-social behaviour, self-harm, suicidal behaviour, and exposure to maltreatment, were administered only to youth aged 14 to 17 years. Finally, if the 12- to 17-year-old was also the selected child in the family, s/he was administered the youth version of the MINI-KID.

Before leaving the household, interviewers asked for signed parental consent to request teacher assessments for children attending elementary school. Based on a mailed survey, we obtained teacher assessments of child emotional and behavioural problems, social relationships, and academic achievements on 3,072 children (38.9% of 4- to 13-year-olds). Interviewers also asked parents for their consent to share their identifying information with the MOHLTC (6,173 [94.4%] agreement) to facilitate linkage with administrative records.

The 2014 OCHS was a voluntary survey conducted under the Statistics Act, which provides respondents guarantees of their privacy and confidentially. Parents and children were asked without coercion for their consent to participate. The study procedures were approved by the Hamilton Integrated Research Ethics Board at McMaster University and Research Ethics Committees at participating School Boards. Interviews were conducted in either English or French, depending on respondent preference. All assessment data underwent qualitative interview testing in a pilot phase, and interviewer training, data collection, and information processing were performed according to standardized procedures developed by Statistics Canada. A sub-sample of 180 households with 280 children participated in a test-retest reliability study of all the 2014 OCHS measures.

Statistical Analyses

The questions posed in the 2014 OCHS can be addressed by simple analyses to estimate prevalence, or more complex analyses using MLM to test hypotheses about associations between childhood mental disorders and problem behaviours, as functions of the independent variables measured at different levels—children, families, neighbourhoods, and schools.

The complex design of the 2014 OCHS (stratification, clustering leading to data dependencies and different household selection probabilities) reduces the precision of estimates, and this needs to be considered by data analysts. This loss of precision is called the survey design effect (ratio of the sampling variance of an estimator under a complex design to the sampling variance of an estimator under simple random sampling).[26] Statistics Canada has developed bootstrap weights to generate proper variance estimates (standard errors) for coefficients obtained in analyses conducted at the individual level, which do not account for data dependencies. Although MLM do account for data dependencies rendering bootstrap weights inapplicable, sampling weights are still needed to produce unbiased population estimates. Analysts are urged to check the software documentation about specifying the use of sampling weights to generate proper variance estimates.

In our experience with the 2014 OCHS, the Statistics Canada bootstrap weights are associated with substantial losses in precision. There are also model-based approaches that can account for complex sampling in survey estimation.[27] Although these approaches offer the possibility of generating unbiased estimates with greater precision, they have yet to be investigated for the 2014 OCHS and are beyond the scope of this report.

Sample Size and Question Non-response

Non-response associated with self-completed modules of the study will affect the sample sizes for secondary analyses. In particular, partial response (80% or more of item non-response) was high for the partner questionnaire (23%) and for certain PMK and youth questionnaire components: 10% of computerized questionnaires completed by youth and 6% of parent, family, and neighbourhood assessments based on computerized questionnaires completed by PMKs. We estimate that 75% to 90% of respondents will have complete data depending on the variables under consideration. Researchers using the 2014 OCHS will need to evaluate the extent of missed responses, assess their collective impact on findings, and choose an appropriate analysis strategy. In the OCHS reports appearing in this journal issue, we examined complete case analysis (listwise deletion) and 2 options for addressing question non-response: multiple imputation (MI) and full information maximum likelihood (FIML).[28] MI imputes values into newly created data sets and is a useful approach for addressing missed responses in descriptive analyses that estimate prevalence. FIML estimates parameters on the basis of the available complete data as well as the implied values of the missing data given the observed data, and is a useful approach for addressing missed responses in MLM used to test specific hypotheses. A companion threat in descriptive papers is multiple testing, which increases the risk of rejecting a true null hypothesis (Type I error). To ensure that nominal P values (levels of significance) remain constant for all tests, researchers are advised to use appropriate methods such as the Benjamini-Hochberg[29] procedure. Finally, data users are urged to read the Microdata User Guide prepared by Statistics Canada[25] for the 2014 OCHS.

Discussion

In the past 30 years, there have been many cross-sectional surveys of childhood mental disorder in the general population. These studies have drawn attention to the mental health needs of children, to variables that increase or decrease the risk for mental disorder, and to the limited ability of the healthcare system to respond to these needs. This information has proven to be effective for advocacy purposes, raising public concern about the mental health needs of children, and prompting policy and program responses among governments.

Cross-sectional studies in the general population, such as the 2014 OCHS, also have limitations: they contribute little to our knowledge about developmental processes that could help tailor prevention and early intervention efforts and are unable to represent youth who may have special needs, such as Aboriginal children on reserves, street youth, and children touched by the child welfare and youth justice systems. Finally, as evidenced in our study, non-response has become a serious concern for general population surveys—the past 25 years have seen a precipitous decline in participation, particularly among those experiencing socioeconomic disadvantage.

Acknowledging the inherent limitations associated with cross-sectional surveys, a number of design elements and unique features were built into the 2014 OCHS to strengthen its usefulness and impact. One, cluster sampling was used to enlist all 4- to 17-year-olds in families and to over-sample families in the same residential areas to assess contextual influences. This enables us to estimate the potential population health impact of attending to these contextual influences when developing new children’s mental health policies and programs. Two, stratification by income was used to select relatively more neighbourhoods and families cross-classified at the lower and higher ends of the continuum. This design element provides a more reliable basis to better understand the adverse effects of socio-economic disadvantage and the potential for other contextual variables (e.g., neighbourhood safety) to mute these effects. Three, some of the measurement and data collection strategies used in the original 1983 OCHS were replicated to facilitate an examination of differences between 1983 and 2014 in prevalence and socio-economic gradients for childhood mental disorder. Four, record linkage to administrative files was used to strengthen measurements (e.g., diagnosis of chronic diseases) and capture variables not studied previously (e.g., service use and physician billings). Five, survey assessments provided by 2014 OCHS respondents were included to represent important process-related contextual variables in neighbourhoods, such as collective efficacy, which are unavailable through census statistics. Six, a separate study of schools (School Mental Health Surveys) was done to create new variables for study (e.g., quantity of school mental health services) and an opportunity to disaggregate neighbourhood from school influences. Seven, the study was led by a large, diverse group of academic researchers in close collaboration with Statistics Canada and policy partners—the Ontario Ministries of Children and Youth Services (MCYS), Health and Long-Term Care (MOHLTC), Education (EDU)—to increase the policy relevance and impact of the study. Finally, we are counting on the use of statistical methods (i.e., use of sampling weights and control variables) and over-sampling of households with low income to compensate and adjust for selective sample losses associated with income. These methods work well as long as participants and non-participants defined by the characteristics linked to nonresponse (e.g., income) are similar to one another on other variables (e.g., health).

Conclusion

The 2014 OCHS represents a large public research investment in children’s mental health. The anticipated dividend of this investment will be the knowledge gained in future secondary analysis studies that capitalize on the data opportunities and enhancements built into the 2014 OCHS. The study is accessible in Canada through the Statistics Canada Research Data Centres (RDC) program to all investigators vetted by Statistics Canada. We urge interested researchers to help maximize the usefulness of the 2014 OCHS by conducting secondary analyses in the years ahead.

Click here for additional data file.

Supplemental Material, 833675_supp_appendix for The 2014 Ontario Child Health Study—Methodology by Michael H. Boyle, Katholiki Georgiades, Laura Duncan, Jinette Comeau, Li Wang and 2014 Ontario Child Health Study Team in The Canadian Journal of Psychiatry