Assessing the validity of administrative health data for the identification of children and youth with autism spectrum disorder in Ontario.

Population-level identification of children and youth with ASD is essential for surveillance and planning for required services. The objective of this study was to develop and validate an algorithm for the identification of children and youth with ASD using administrative health data. In this retrospective validation study, we linked an electronic medical record (EMR)-based reference standard, consisting 10,000 individuals aged 1-24 years, including 112 confirmed ASD cases to Ontario administrative health data, for the testing of multiple case-finding algorithms. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and corresponding 95% confidence intervals (CI) were calculated for each algorithm. The optimal algorithm was validated in three external cohorts representing family practice, education, and specialized clinical settings. The optimal algorithm included an ASD diagnostic code for a single hospital discharge or emergency department visit or outpatient surgery, or three ASD physician billing codes in 3 years. This algorithm's sensitivity was 50.0% (95%CI 40.7-88.7%), specificity 99.6% (99.4-99.7), PPV 56.6% (46.8-66.3), and NPV 99.4% (99.3-99.6). The results of this study illustrate limitations and need for cautious interpretation when using administrative health data alone for the identification of children and youth with ASD. LAY
SUMMARY: We tested algorithms (set of rules) to identify young people with ASD using routinely collected administrative health data. Even the best algorithm misses more than half of those in Ontario with ASD. To understand this better, we tested how well the algorithm worked in different settings (family practice, education, and specialized clinics). The identification of individuals with ASD at a population level is essential for planning for support services and the allocation of resources. Autism Res 2021, 14: 1037-1045.

INTRODUCTION

Autism spectrum disorders (ASD) are a group of lifelong neurodevelopmental disorders characterized by impaired communication and social interaction, repetitive behaviors, and restricted interests. These core features of ASD are further complicated by the cooccurrence of other neurodevelopmental disorders (e.g., attention deficit hyperactivity disorder [ADHD]; Abdallah et al., 2011), many of which have overlapping traits, and can also include intellectual disability (Taurines et al., 2012). Autistic individuals are also more likely to be diagnosed with a host of other medical conditions including: epilepsy, schizophrenia, gastrointestinal disturbances, sleep disorders, anxiety, and asthma, among others (Nazeen, Palmer, Berger, & Kohane, 2016). Together this means that individuals with ASD and their families, frequently require sustained support and increased resources, from both the public and private sectors, across the life‐course (Liptak, Stuart, & Auinger, 2006; Weiss et al., 2018). These services are associated with a significant economic burden on publicly funded health, education, and social services, but are also associated with significant out‐of‐pocket expenses by the families of individuals with ASD (Buescher, Cidav, Knapp, & Mandell, 2014; Tsiplova et al., 2019).

In 2015, it was estimated that approximately one in 59 eight‐year‐olds in the United States had ASD (Baio et al., 2018). Recently, the Public Health Agency of Canada (PHAC) reported that one in 66 Canadian children between the ages of 5 and 17 years has the condition (Ofner et al., 2018). The diagnosis of autism is three‐ to four‐times more common in males versus females, varies based on geographical region, and is increasing over time; factors that are reflected in both the American and Canadian estimates (Baio et al., 2018; Loomes, Hull, & Mandy, 2017; Ofner et al., 2018).

The identification of children and youth with ASD at a population level is essential for surveillance and planning for support services and resource allocation. Further, there is a need to provide evidence‐based recommendations for programs and services for individuals with ASD and their families. To address this need, researchers are increasingly developing different approaches for the identification of individuals with ASD, both within electronic medical records (EMR; Brooks et al., 2021; Bush, Connelly, Perez, Barlow, & Chiang, 2017; Coleman et al., 2015; Lingren et al., 2016), and using administrative health data (Coo, Ouellette‐Kuntz, Brownell, Shooshtari, & Hanlon‐Dearman, 2018; Dodds et al., 2009).

Administrative data have the benefit of being population‐wide and longitudinal in nature, allowing for the tracking of population trends over time. However, these data are not collected for research purposes, relying on routine practices which may not be consistent across health care providers or settings (Jutte, Roos, & Brownell, 2011). Further, the data used to generate population‐based estimates of ASD vary across regions and can include information from social services, health services and educational data, alone or in combination. Researchers in some Canadian provinces have developed and validated algorithms for the identification of individuals with ASD using available administrative data. An algorithm developed in Nova Scotia using multiple administrative health databases achieved a sensitivity of 69.3% (Dodds et al., 2009). Researchers in Manitoba were able to do better by using a combination of administrative health and education data to generate a case‐finding algorithm with 88% sensitivity (Coo et al., 2018). In a recent study by Bickford et al. (2020), the investigators concluded that administrative data was insufficient for the identification of children with autism, in particular an inability to discriminate between children with ASD and children with other neurodevelopmental outcomes.

Given the importance of identifying individuals with ASD to support health system planning, resource allocation, and the monitoring of long‐term outcomes in autistic individuals, the objective of this study was to develop and validate an algorithm for the identification of children and youth with ASD using administrative health data in Ontario Canada, where population‐based estimates of ASD prevalence are not currently available.

METHODS

We conducted a retrospective validation study of multiple algorithms to identify ASD in administrative health data, using a reference standard of individuals with known ASD status from family practice EMR (Figure 1). All research was carried out at ICES, an independent, nonprofit research institute whose legal status under Ontario's health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement. ICES was formerly known as the Institute for Clinical Evaluative Sciences, and formally adopted the initialism ICES as its official name in 2018. This project was approved by the Research Ethics Board at the University of Toronto and Sunnybrook Health Sciences Center, Toronto, Canada.

FIGURE 1

Flow diagram for the linkage of various data sources for the validation and development of an administrative data‐based algorithm including application to the province of Ontario, and three external validation cohorts

Reference standard

The Electronic Medical Record Primary Care (EMRPC; formerly known as EMRALD) data housed at ICES consist of the EMRs for over 400,000 patients, from over 350 family physicians using PS Suite EMR (formerly Practice Solutions). This includes data on a patient demographics, vital status, prescription medications, current diagnoses, progress notes, and billing information, as well as a free‐text field for physician comments. Participation in EMRPC on the part of the physician is voluntary and only requires them to have used the EMR for at least 2 years (Tu et al., 2015). A reference standard for ASD was created using a population‐based cohort of 10,000 children and youth aged 1 to 24 years within the EMRPC, details have been published elsewhere (Hauck, Lau, Wing, Kurdyak, & Tu, 2017). Briefly, trained nurse chart abstractors manually reviewed the EMR histories of 10,000 children and youth for diagnoses of ASD (including Asperger's syndrome and pervasive developmental delay [PDD] per DSM‐V criteria), and other neurodevelopmental (e.g., ADHD) and mental health (e.g., depression) conditions. Abstractor‐identified cases were confirmed by a family physician (KT), and if confirmed, classified as “definitely” having ASD. In total, 112 individuals were identified from the cohort of 10,000 as having a confirmed diagnosis of ASD (mean age 12.7 years, 1:3 ratio females to males) and 9888 were determined to not have an autism diagnosis. These 112 individuals with ASD and 9888 without, were linked with Ontario administrative health data and used as the reference standard for the development and testing of algorithms for the identification of individuals with ASD in the Ontario population.

Administrative health data

People living in Ontario are insured under a single‐payer system that covers physician and hospital services and procedures. Administrative health data are generated through patient contact with the health care system and maintained in multiple databases that can be linked using a unique encoded identifier (Table S1). These include: the Registered Persons Database (RPDB) which provides information on demographic variables (e.g., age, sex, neighborhood‐level income quintile, and geographic location [the latter two are derived from Statistics Canada Census data]), the Canadian Institute for Health Information Discharge Abstract Database (DAD) and National Ambulatory Care Reporting System (NACRS) which provide information on diagnoses, procedures and other characteristics for hospital visits, community‐based ambulatory care, outpatient clinics, outpatient surgery, and emergency department visits. The Ontario Health Insurance Plan (OHIP) physician billing database provides information on diagnoses and medical service billings by Ontario physicians (both general practitioners and specialists). These datasets were linked using unique encoded identifiers and analyzed at ICES (Figure 1).

Algorithm development, testing, and selection

The EMR reference standard was first linked to administrative health data sources (RPDB, DAD, NACRS, and OHIP). Established methods for algorithm development, testing and selection, routinely employed at ICES (Tu et al., 2014), were used to generate and evaluate case‐finding algorithms for ASD. Tested algorithms (N = 153) included data from multiple sources: physician billing, hospital visits, outpatient surgeries, and emergency department visits. The OHIP billing code 299 “childhood psychoses e.g., autism” was used. While this billing code is not specific to autism, it is the code most frequently used by physicians to indicate the use of health services related to autism. Codes from all other sources included the ASD ICD‐9 diagnostic codes (299.x) and ICD‐10 codes (F84.x). OHIP billing codes from specific specialists (pediatricians, neurologists, and psychiatrists) were also considered. The tested algorithms varied the type of data source(s) included (physician billing codes, hospital discharge, emergency department visits, and outpatient surgery), and the number and frequency of codes required (e.g., 1–4 codes in 1–4 years). A final, optimal algorithm was selected to maximize both the positive predictive value (PPV) and sensitivity of the algorithm (Figure 1).

External validation

The optimal algorithm was validated using three external cohorts. These cohorts included individuals diagnosed or identified as having ASD in a range of contexts including primary care, education, and specialized clinical settings. The first validation was carried out in the primary care setting, among children and youth aged 1–24 years old in the full EMRPC cohort (as of March 1, 2016, N = 80,237). Individuals were classified as having ASD or not using a previously developed EMR‐based algorithm, which identified 1062 individuals with, and 79,175 without (sensitivity 82.1%, PPV 98.9%; Brooks et al., 2021). Notably this algorithm had good predictive ability and consisted of a key word search of the Cumulative Patient Profile, a free text field within the EMR that summarizes key patient information.

The second validation was conducted using a cohort of 5‐ to 6‐year‐olds in Ontario who completed the Early Development Instrument (EDI) between January and June 2015, at the end of their senior kindergarten year (N = 103,948). The Early Development Instrument (EDI) is a questionnaire developed by Dr. Dan Offord and Dr. Magdalena Janus at the Offord Center for Child Studies at McMaster University, Hamilton, Ontario, Canada. The EDI is a teacher‐completed checklist that measures children's developmental health in five domains: physical health and well‐being, social competence, emotional maturity, language and cognitive development, and communication skills and general knowledge (Janus & Offord, 2007). Using this tool, 1503 individuals with, and 102,445 without ASD were identified.

Finally, the third validation was carried out using the Province of Ontario Neurodevelopmental Disorder (POND) Network. POND is an ongoing cross‐sectional study of children and youth with neurodevelopmental disorders (e.g., ASD, ADHD, obsessive compulsive disorder [OCD]) for which recruitment began in 2012. Participants are identified through four recruitment centers in Ontario: Holland Bloorview Kids Rehabilitation Hospital and The Hospital for Sick Children in Toronto, McMaster University in Hamilton, and the Lawson Health Research Institute‐Western University in London Ontario. This study population is referred to as the “clinical cohort” for the purpose of algorithm validation as these individuals were diagnosed within these specialized clinical settings. For the purpose of this validation, we used 661 POND participants, 415 of whom have a diagnosis of ASD.

Each external validation cohort was linked to administrative health data at ICES using unique encoded identifiers to allow for testing of algorithm discrimination (Figure 1).

Statistical analyses

The 153 tested algorithms developed using the reference standard (N = 10,000 including N = 112 with ASD) were applied to the population of Ontario residents aged 1–24 years in 2016 (N = 3,960,763). Sensitivity, specificity, PPV, negative predictive value (NPV), and corresponding 95% confidence intervals (CI) were calculated for each algorithm. For further examination, the optimal algorithm was applied to the Ontario population, using the three cohorts: the broader EMRPC cohort using the EMR algorithm (N = 80,237, N = 1062 with ASD), those identified using EDI data (N = 103,948, N = 1503 with ASD) and those with a primary diagnosis of ASD included in the POND Study (N = 661, N = 415 with ASD), as the reference standard.

RESULTS

A selection of the tested algorithms is presented in Table 1 (all algorithms are presented in Table S2). Overall, the algorithms tended to be highly specific (i.e., low false negative rate) while the sensitivity varied substantially. The most sensitive algorithm was one physician billing claim or a single hospital discharge or emergency department visit or outpatient surgery with a diagnosis of ASD (75.9%), however, this algorithm had a very low PPV (probability that individuals identified by the algorithm as having ASD actually had ASD) of only 42.9%. The optimal algorithm was selected to maximize both sensitivity and PPV. This algorithm included (three ASD physician billing codes in 3 years or a single hospital discharge or emergency department visit or outpatient surgery with a diagnosis of ASD). This algorithm had a sensitivity of 50.0% (95% CI 40.7–88.7%), specificity of 99.6% (95% CI 99.4–99.7), PPV of 56.6% (95% CI 46.8–66.3), and NPV of 99.4% (95% CI 99.3–99.6), maximizing both sensitivity and PPV.

TABLE 1

Selected algorithms tested for the identification of children and youth (ages 1–24 years) with ASD in Ontario

Description a	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
1 physician billing claim ever	74.1 (66.0–82.2)	98.9 (98.7–99.1)	42.6 (35.6–49.5)	99.7 (99.6–99.8)
1 physician billing claim code by any specialist	66.1 (57.3–74.8)	99.1 (98.9–99.2)	44.3 (36.8–51.8)	99.6 (99.5–99.7)
Hospital discharge or emergency department visit or outpatient surgery or 1 physician billing claim	75.9 (68.0–83.8)	98.9 (98.6–99.1)	42.9 (36.0–49.8)	99.7 (99.6–99.8)
Hospital discharge or emergency department visit or outpatient surgery or 1 physician billing claim by any specialist	67.9 (59.2–76.5)	99.0 (98.9–99.2)	44.7 (37.2–52.2)	99.6 (99.5–99.8)
Hospital discharge or emergency department visit or outpatient surgery or (2 physician billing claims in 2 years)	57.1 (48.0–66.3)	99.3 (99.1–99.5)	48.5 (40.0–57.0)	99.5 (99.4–99.7)
Hospital discharge or emergency department visit or outpatient surgery or (2 physician billing claims in 2 years at least 1 physician billing claim by any specialist)	52.7 (43.4–61.9)	99.4 (99.2–99.5)	48.4 (39.5–57.2)	99.5 (99.3–99.6)
Hospital discharge or emergency department visit or outpatient surgery or (2 physician billing claims in 3 years)	59.8 (50.7–68.9)	99.3 (99.1–99.5)	49.3 (40.9–57.7)	99.5 (99.4–99.7)
Hospital discharge or emergency department visit or outpatient surgery or (2 physician billing claims in 3 years at least 1 physician billing claim by any specialist)	53.6 (44.3–62.8)	99.4 (99.2–99.5)	48.4 (39.6–57.2)	99.5 (99.3–99.6)
Hospital discharge or emergency department visit or outpatient surgery or (3 physician billing claims in 2 years)	45.5 (36.3–54.8)	99.6 (99.4–99.7)	54.3 (44.2–64.3)	99.4 (99.2–99.5)
Hospital discharge or emergency department visit or outpatient surgery or (3 physician billing claims in 2 years at least 1 physician billing claims by any specialist)	45.5 (36.3–54.8)	99.6 (99.5–99.7)	56.0 (45.8–66.2)	99.4 (99.2–99.5)
Hospital discharge or emergency department visit or outpatient surgery or (3 physician billing claims in 3 years)	50.0 (40.7–59.3)	99.6 (99.4–99.7)	56.6 (46.8–66.3)	99.4 (99.3–99.6)
Hospital discharge or emergency department visit or outpatient surgery or (3 physician billing claims in 3 years at least 1 physician billing claim by any specialist)	49.1 (39.8–58.4)	99.6 (99.5–99.7)	57.9 (48.0–67.8)	99.4 (99.3–99.6)

Abbreviations: ASD, autism spectrum disorder; NPV, negative predictive value; PPV, positive predictive value.

Physician billing claims included 299 “childhood psychoses e.g., autism.” Codes from all other sources (e.g., hospital discharge, emergency department visit) included ICD‐9 codes (299.x) and ICD‐10 codes (F84.x). Specialists include pediatricians, psychiatrists, and neurologists.

Secondary review of the EMR data for all misclassified individuals in the reference standard revealed complex medical histories. False‐positives (i.e., individuals in the reference standard classified as having ASD by the algorithm, but not by chart review) tended to have a high burden of other neurodevelopmental or mental health conditions including diagnoses or features of ADHD, anxiety and depression, or other possible NDDs. Conversely, many of the false‐negatives (about half) were identified in the medical record review as having Asperger's Syndrome. Asperger's Syndrome used to be considered a clinically distinct higher functioning form of ASD, but since 2013 with the introduction of DSM‐V is captured under the diagnosis of ASD.

When applied to the 2016 Ontario population (ages 1–24 years), the optimal algorithm resulted in 36,713 out of 3,960,763 children and youth being identified as having ASD. This corresponds to a prevalence of 0.93% or one in 108. The characteristics of individuals in Ontario identified with and without ASD are shown in Table 2. Individuals identified as having ASD were slightly older with a mean age of 12.7 years, as compared to those without ASD (mean age 12.5 years). The sex distribution also differed between the two groups with approximately 80% of those identified as having ASD being male.

TABLE 2

Characteristics of children and youth (ages 1–24 years) with and without ASD in Ontario (2016)

	No ASD N = 3,924,032	ASD N = 36,731
Mean age (±SD) a	12.5 ± 7.3	12.7 ± 5.8
Age group (years), N(%)
1–4	726,767 (18.5)	2,631 (7.2)
5–9	761,472 (19.4)	9,847 (26.8)
10–14	751,718 (19.2)	10,124 (27.6)
15–19	793,060 (20.2)	8,369 (22.8)
≥20	891,015 (22.7)	5,760 (15.7)
Sex
Female	1,924,366 (49.0)	7,201 (19.6)
Male	1,999,666 (51.0)	29,530 (80.4)
Geographic Location
Rural	412,632 (10.5)	3,304 (9.0)
Suburban	282,152 (7.2)	2,812 (7.7)
Urban	3,229,248 (82.3)	30,615 (83.3)
Neighborhood Income Quintile
1 (Lowest)	769,673 (19.6)	7,815 (21.3)
2	728,467 (18.6)	7,110 (19.4)
3	779,880 (19.9)	7,205 (19.6)
4	808,081 (20.6)	7,258 (19.8)
5 (Highest)	829,172 (21.1)	7,216 (19.6)

Abbreviation: ASD, Autism spectrum disorder.

Age as of March 1, 2016.

Results from the external validation are presented in Table 3. Application of the algorithm to the EMRPC population (family practice), found discrimination of the algorithm was similar to that seen for the reference standard used in algorithm development (sensitivity = 51.8%, specificity = 99.7%, PPV = 62.5%, and NPV = 99.4%). In the EDI cohort (education), while the algorithm had a similar sensitivity to that seen using the reference standard or full EMRPC cohort (55.9%), the PPV was higher at 70.1%. Finally, the algorithm performed well in the clinical cohort from the POND Study, with 72.8% sensitivity, 94.3% specificity, 95.6% PPV, and 67.2% NPV.

TABLE 3

Algorithm validation in external cohorts

Cohort details	EMRPC a	EDI b	POND c
Total N	80,237	103,948	661
Number with ASD	1062	1503	415
Estimated ASD Prevalence (%)	1.3	1.4	N/A d
Age range	1–24 years	5–6 years	1–21 years
Performance of selected algorithm
Sensitivity (%)	51.8	55.9	72.8
Specificity (%)	99.7	99.6	94.3
PPV (%)	62.5	70.1	95.6
NPV (%)	99.4	99.4	67.2

Abbreviations: ASD, autism spectrum disorder; EDI: early development instrument; EMRPC: electronic medical record primary care; NPV, negative predictive value; POND: province of Ontario neurodevelopmental disorders; PPV, positive predictive value.

EMRPC data, housed at ICES. Individuals with ASD were identified using an EMR‐based case‐finding algorithm.

EDI is a teacher‐completed measure of early development outcomes of children in kindergarten. Data is from a cohort of 5–6‐year‐old children in Ontario in 2015.

POND Study is an ongoing cross‐sectional study of children and youth with neurodevelopmental disorders.

Prevalence of ASD is not estimated in POND as it is a cohort of children and youth with neurodevelopmental disorders.

DISCUSSION

We tested and validated over 150 algorithms for the identification of children and youth with ASD using population‐based administrative health data in Ontario. The optimal algorithm used information from multiple sources and included: three ASD physician billing codes in 3 years or a single hospital discharge or emergency department visit or outpatient surgery, with an associated ASD diagnostic code. Notably, while this algorithm was considered “optimal” among all those that were tested, it is far from ideal with a sensitivity of only 50% and a PPV of 57%. To better understand who this algorithm is identifying as having autism, and conversely who it is not capturing, we applied this algorithm to the Ontario population and conducted several external validations.

When applied to the province of Ontario, this algorithm identified over 36,000 individuals between the ages 1–24 years as having ASD, corresponding to a prevalence of 0.93%. This estimate is lower than the estimated prevalence of ASD in Canada and the United States of approximately 1.5% (Baio et al., 2018; Ofner et al., 2018). Sex differences in the prevalence of ASD are well documented (Baio et al., 2018; Ofner et al., 2018). Here, we found that of the individuals determined by the algorithm to have ASD, about 80% were male. This is slightly higher than the expected ratio of about 1:3 or 1:4 (Baio et al., 2018; Loomes et al., 2017; Ofner et al., 2018), suggesting that the algorithm preferentially identifies male over female individuals as having ASD.

Autism can be reliably diagnosed as early as 24 months of age (Corsello, Akshoomoff, & Stahmer, 2013; Steiner, Goldsmith, Snow, & Chawarska, 2012). In Ontario, and other Canadian provinces, the earliest opportunity to assess developmental concerns across multiple domains (e.g., language, social) is typically with a child's physician, as routine visits are frequent in the early years of life. Still, most children are not clinically diagnosed until after the age of four (Baio et al., 2018; Brett, Warnell, McConachie, & Parr, 2016; Coo et al., 2012; Ofner et al., 2018).

When examining the characteristics of the individuals identified by the algorithm as having ASD versus not, there is a slight difference in the age distribution. A lower proportion of children under 5 years of age are classified as having ASD (7.2% with ASD and 18.5% without ASD, Table 2). This is likely to be partly due to the timing of clinical diagnosis, but may also be related to a bias in the algorithm towards the identification of older children, with one of the elements of the algorithm being the occurrence of three physician billing codes in 3 years. This requires that the individual be alive, and in Ontario for at least 3 years to be classified as having ASD based on this criterion. However, algorithms that used shorter time periods (e.g., 1 or 2 years) did not perform as well (Table 1).

Discrimination of the optimal algorithm was tested against multiple cohorts including children with ASD diagnosed/identified across a variety of settings including primary care, education, and specialized clinics. When we linked these cohorts to the administrative data and observed who was captured by the algorithm, we saw differences in the algorithm's ability to discriminate between individuals with and without ASD across the different settings. The algorithm performed somewhat better in the EDI data, where the sensitivity was still low (55.9%) but the PPV was higher at 70.1%. This could be because the EDI data include individuals that were 5‐ to 6‐years old in 2015 and so are able to be captured by the three OHIP codes in 3 years criterion. Not surprisingly, the algorithm performed best in the specialized clinical cohort from the POND study with a sensitivity of 72.8% and a PPV of 95.6%, this is likely to be partly due to the high prevalence of ASD in this study cohort. However, it is also likely because these patients are being seen within a highly specialized clinical setting, suggesting that the algorithm is likely missing individuals who are not as intensely engaged in clinical care.

A particular challenge to the use of these data for the identification of individuals with ASD is that ASD is one of a series of neurodevelopmental disorders (e.g., ADHD and OCD) that share common traits (Krakowski et al., 2020; Kushki et al., 2019; Taurines et al., 2012), and etiology (Cross‐Disorder Group of the Psychiatric Genomics Consortium, 2013; Faraone & Larsson, 2019; Gonzalez‐Mantilla, Moreno‐De‐Luca, Ledbetter, & Martin, 2016; Lowther et al., 2017; Zarrei et al., 2019), frequently co‐occurring within individuals (Lai et al., 2019). A recent meta‐analysis found that 28% of individuals with ASD also have ADHD and 9% have OCD (Lai et al., 2019). The combination of within disorder heterogeneity, co‐occurrence, and cross‐disorder homogeneity, makes the use of administrative data for case‐identification particularly challenging.

Indeed, when we reviewed the EMRs of individuals classified as false positives, they were found to have complex medical histories with multiple on‐going neurodevelopmental diagnoses. Conversely, approximately half of the individuals with ASD not captured by the algorithm were found to have Asperger's Syndrome, a higher functioning autism. Together this suggests that the algorithm, favors the identification of individuals with more complex needs, requiring multiple interactions with the health care system.

Prior efforts have examined the potential of using provincial administrative health data to identify individuals with ASD with variable degrees of success (Bickford et al., 2020; Coo et al., 2018; Dodds et al., 2009), and those able to incorporate data from multiple settings, outside of administrative health data having the highest sensitivity (Coo et al., 2018). Our results, like those of Bickford et al. (2020), suggest that the use of administrative health data alone is not sufficient for the identification of individuals with ASD. This is likely because autistic individuals receive services from other health professionals (e.g., psychologists, social workers, occupational, and physical therapists), and educational support staff, all of whom are known to play an important role in the diagnosis and care of individuals with ASD (Lai, Anagnostou, Wiznitzer, Allison, & Baron‐Cohen, 2020), but whose activity is not captured through physician billing.

Numerous studies have used administrative health data to identify individuals with autism (Alexeeff et al., 2017; Angell et al., 2021; Burke et al., 2014; Coleman et al., 2015; Jain et al., 2015; Zerbo et al., 2018). While this approach has several obvious potential benefits including the ability to capture population‐level estimates of ASD prevalence, prevalence of comorbidities, and health system utilization, both retrospectively and prospectively, the results of the current study illustrate some of the limitations of identifying children and youth with ASD in this manner.

Overall, we found that our optimal algorithm for the identification of children and youth with ASD within administrative health data favors the identification of individuals with ASD who are older (>age 5 years), male, seen within a specialized clinical setting and/or with more complex presentation. This reinforces the need for validation studies across multiple jurisdictions (Nicholls, Langan, Sørensen, Petersen, & Benchimol, 2016) and the incorporation of data outside of administrative health data alone, to better capture the complex and varied pathways to an ASD diagnosis.

The current study has numerous strengths. In particular, the use of a reference standard developed through chart abstraction and confirmation by a family physician. This is further strengthened by our use of multiple cohorts for additional external validation, to better understand who is and is not being captured by the algorithm. There are of course several limitations that must also be considered. First, is our current inability to utilize education data or services administered by other health professionals into the case‐finding algorithms. We were also not able to distinguish between individuals with autism with and without intellectual disability or language impairment (as described in DSM‐V). It is possible that these individuals may have different patterns of health services use and may be differentially captured by the algorithm. Further, while the use of multiple validation cohorts is a strength of the study, each of these cohorts has their own limitations. First, autistic individuals in the EMPRC cohort were algorithm identified (Brooks et al., 2021). While having good accuracy, we cannot be 100% certain patients have been correctly classified. The same is true for the EDI instrument which is not a screening instrument but rather a teacher‐recorded confirmation of a predetermined/preexisting diagnosis. Finally, this study did not address the issue of identifying adults with autism in the population, but rather focused on children and youth. Autistic individuals require sustained but changing supports across the life course (Lai et al., 2020) and an understanding of these needs is essential. Ultimately, it is hoped that individuals with autism can be identified in childhood and followed‐up longitudinally to understand long‐term outcomes.

The identification of children and youth with ASD is essential for ASD surveillance, identification of trends in incidence and prevalence, and planning for services, supports and resource allocation. The challenge of accurately identifying individuals with autism using administrative health data limits our ability to conduct population‐level research on ASD and associated outcomes. This has widespread implications for the study of autism using administrative health data, requiring cautious interpretation of studies using administrative health data alone to identify individuals with autism, supporting the need for validation studies across multiple jurisdictions, as well as the integration of data across sectors (e.g., health, education, and social services).

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

TABLE S1: Description of ICES data holding used to characterize burden of comorbid conditions and health system utilization of children and youth with and without ASD

TABLE S2: Tested algorithms (N = 151) for the identification of children and youth (ages 1–24 years) with ASD in the Ontario population using administrative health data

Click here for additional data file.