Our Health Counts Toronto: using respondent-driven sampling to unmask census undercounts of an urban indigenous population in Toronto, Canada.

OBJECTIVES: To provide evidence of the magnitude of census undercounts of 'hard-to-reach' subpopulations and to improve estimation of the size of the urban indigenous population in Toronto, Canada, using respondent-driven sampling (RDS).
DESIGN: Respondent-driven sampling.
SETTING: The study took place in the urban indigenous community in Toronto, Canada. Three locations within the city were used to recruit study participants. PARTICIPANTS: 908 adult participants (15+) who self-identified as indigenous (First Nation, Inuit or Métis) and lived in the city of Toronto. Study participants were generally young with over 60% of indigenous adults under the age of 45 years. Household income was low with approximately two-thirds of the sample living in households which earned less than $C20 000 last year. PRIMARY AND SECONDARY OUTCOME MEASURES: We collected baseline data on demographic characteristics, including indigenous identity, age, gender, income, household type and household size. Our primary outcome asked: 'Did you complete the 2011 Census Canada questionnaire?'
RESULTS: Using RDS and our large-scale survey of the urban indigenous population in Toronto, Canada, we have shown that the most recent Canadian census underestimated the size of the indigenous population in Toronto by a factor of 2 to 4. Specifically, under conservative assumptions, there are approximately 55 000 (95% CI 45 000 to 73 000) indigenous people living in Toronto, at least double the current estimate of 19 270.
CONCLUSIONS: Our indigenous enumeration methods, including RDS and census completion information will have broad impacts across governmental and health policy, potentially improving healthcare access for this community. These novel applications of RDS may be relevant for the enumeration of other 'hard-to-reach' populations, such as illegal immigrants or homeless individuals in Canada and beyond. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

The study included a large sample of a hard-to-reach population recruited using a probability-based method.

The study used novel applications of respondent-driven sampling (RDS) and census completion information to enumerate the size of a hard-to-reach population.

Census completion information was collected by retrospective self-report, thus participant responses may be subject to recall bias.

Baseline data relating to the census were obtained in 2011 and may no longer be valid.

We are confident that our study is robust; nonetheless, care must be taken in the interpretation of respondent-driven sampling studies and generalisation to the target population.

Introduction

National censuses are considered the gold standard for population-level data and commonly provide the sampling frame for national surveys.1–3 Ensuring full census participation of socially marginalised groups and adjusting for undercounts is a common problem throughout the world.4 5 This is particularly challenging in homeless6 or indigenous subpopulations,7 who are at risk of being undercounted and thereby lacking access to appropriate healthcare services.

In what is now known as Canada, there are three distinct groups of indigenous peoples (Indian, Inuit and Métis) that are recognised in the Canadian constitution as having inherent rights that predate Canadian confederation.8 The group referred to in the constitution as ‘Indian' represents a highly diverse set of cultural linguistic population groups, who commonly self-identify by their specific cultural linguistic name and/or the collective term First Nations, which came into use in the 1970s as the term ‘Indian’ was considered by some to be offensive. The Inuit traditionally lived above the tree line and are part of a larger circumpolar Inuit population that includes Greenland, Alaska and Russia. The origins of the Métis Nations are found in the historic intermarriage of European men with First Nations/Indian women in the 1700s. There has always been intermarriage across First Nations groups.9 Métis have their own unique identities which has been shaped by over 250 years of history that intertwines with but is distinct from the history of First Nations.

The majority of First Nations and Métis peoples in Canada now live in cities, and the urban Inuit population is rapidly increasing.10 Urbanisation increases opportunities for intermixing between and across First Nations, Métis and Inuit population groups. Importantly, and in keeping with their distinct cultural–linguistic identities, histories and differences in externally imposed colonial policies, each group also has unique urban experiences and needs.11

Urban indigenous peoples are at risk of non-participation in the census due to factors such as poverty and its associated lack of living at a fixed address,12 13 historical distrust of government due to past and present colonial policies14 and migration between geographical locations.12 In 2011, despite concerns of the reduced validity of using a voluntary questionnaire, the Canadian government made a significant change in the collection of Canadian population data and eliminated the mandatory long-form census, replacing it with the voluntary National Household Survey (NHS).15 16 This further complicated the accurate enumeration of indigenous peoples in Canada because questions regarding indigenous identity were part of the content that was transferred from the long-form census to the NHS, which had a much lower global response rate (68.6%) than the previous long-form census (93.8%).13 Additional NHS data reliability issues have been identified for estimates of the prevalence of low-income households, estimates involving smaller geographical areas and for specific indigenous subpopulations.14

Within this context, we investigated methods to more accurately enumerate the indigenous population of Toronto, Ontario, Canada’s largest urban centre. The purpose of our study was twofold: (1) To provide evidence of the magnitude of undercount of this ‘hard-to-reach’ indigenous subpopulation and (2) To improve estimation of the size of the urban indigenous population in Toronto, Canada, using respondent-driven sampling (RDS) for healthcare access and planning purposes.

Methods

From March 2015 to March 2016, the Seventh Generation Midwives Toronto (SGMT), in partnership with the Centre for Urban Health Solutions at St. Michael’s Hospital recruited indigenous adults who lived, worked or received health services in the city of Toronto, Canada, using a community-based RDS17 18 design.

Community partnerships

This study was co-led by the SGMT and the Well Living House Action Research Centre for Indigenous Infant, Child and Family Health and Wellbeing (Well Living House) at St. Michael’s Hospital. SGMT is a midwifery clinic in Toronto that specialises in providing care for indigenous mothers and their babies. Given the traditional role of midwives as knowledge keepers of birth stories and family information, SGMT was identified as an appropriate organisational custodian for indigenous community health data in Toronto.19

Study design

In contrast to indigenous populations living in First Nations on-reserve communities or Inuit land claim territories, there is no comprehensive registry of indigenous people living in cities. In the face of these challenges and the socially networked community of urban indigenous people, we chose RDS17 18 as the sampling method. Based on the design effects that were observed in our previous urban indigenous RDS studies,20 a target sample size of approximately 1000 urban indigenous adults was chosen to provide appropriate power for descriptive and comparative measures in this study.

Recruitment methods

Participant recruitment took place at three health and social services locations spanned across the city of Toronto. Specifically, interviews took place 2 days per week at the Queen West Central Toronto Community Health Centre, SGMT and the Native Canadian Centre of Toronto, respectively, and also included an option for a mobile survey service which could come to a participant’s location of choice. All interviewers had strong pre-existing indigenous community engagement skills and were further trained to ensure that all interviews were conducted in a respectful and culturally sensitive manner. As an incentive, participants were provided with $C20.00 for completing the survey as well as $C10.00 for each individual that they recruited into the study. In addition, adults who had children under the age of 14 residing with them were asked to complete a child-specific survey and provided with an additional honorarium of $C10.00 per completed child survey. All financial incentives were in Canadian dollars.

Eligible participants had to self-identify as indigenous (First Nation, Inuit or Métis) and live, work or receive healthcare services in Toronto. However, for the purposes of this study, all analyses are restricted to only those who reside in the city of Toronto. In collaboration with our community partners, 10 seeds were initially selected in consultation with partner indigenous health and social service organisations to ensure that they captured the diversity of the indigenous community and had sufficiently large social networks to ensure the study would continue past its initial waves. Later in the study, 10 seeds were added to increase recruitment speed. Initially, three coupons were issued per person; however, this was later augmented to five to further increase the recruitment rate. On increasing the number of coupons available, previously recruited individuals were contacted and asked if they would like two additional coupons. A total of 77 people were eligible for an additional two coupons; 65 (84%) of them were contacted by phone, email or in person and of these, 41 individuals did receive an additional two coupons.

There was no explicit time limit for recruitment; however, community posters and information boards at recruitment locations included the study completion date of 31 March 2016 and encouraged all participants to complete the study prior to that date. Participants who were recruited in 2016 were also reminded of the study’s end date in their interviews. Efforts to reduce duplicates included using a preliminary screener to assess eligibility, matching on voluntarily provided provincial health insurance numbers and examination for duplicated survey questionnaires. Note that 97% of study participants voluntarily provided their unique provincial health insurance number, thus we are confident that the risk of duplicate surveys is minimal. A flow chart of the sample acquisition and data cleaning process is presented in figure 1. The personal network size question was phrased as ‘Approximately how many Aboriginal people do you know (ie, by name and that know you by name) who currently live, work or use health and social services in Toronto?’. The recruiter–recruit relationship was tracked using the unique numbers on each coupon in our database. Participants gave informed consent before taking part in the study. Study results are reported in accordance with the STROBE-RDS (Strengthening the Reporting of Observational Studies in Epidemiology-RDS) guidelines.21

Figure 1

Our Health Counts Toronto sample acquisition and data cleaning, Toronto, Canada, March 2015 to March 2016, indigenous adults.

Patient involvement

The SGMT was supported by a broader reference group of indigenous and allied health and social service organisations. Our project’s community-based research, publication and data governance protocols assure indigenous community leadership in all aspects of study design, data collection/recruitment, analyses and interpretation of data. Results will be disseminated to study participants by a community event and report. Specifically, SGMT retains full ownership and control over all data in accordance with the project’s academic–community partnership agreement. This community-determined research approach ensured study success and is the most important factor in the successful participation of Toronto’s indigenous community in this RDS study.

Primary outcome

Obtaining accurate estimates of the size of the indigenous population residing in metropolitan Toronto was of significant research interest, thus, a question was included in the Our Health Counts Toronto questionnaire asking whether or not a participant had completed the census, ‘Did you complete the 2011 Census Canada questionnaire?’. The inclusion of this question enabled us to use actual census counts in order to re-estimate the size of Toronto’s indigenous population, which was currently residing in Toronto.

Statistical analysis

All RDS analyses, including demographic and primary outcome variables were performed in the RDS package (V.0.7.7) in R22 using RDS-II weights.23 Bottleneck plots and appropriate statistical tests were used to examine convergence and test assumptions of RDS such as recruitment and non-response biases.24 Seeds were excluded for analysis purposes. Age was our only continuous measure and was classified into five categories of 10-year intervals and a category of older adults (65+). Household type (private dwelling, institutional or homeless) was also included to estimate the proportion of individuals who should have completed the census. Note that household size was only asked of participants who lived in private dwellings. For demographic purposes, household income is presented in this manuscript without adjustment for household size. There were no participants with missing data on the primary census survey question or on gender, age or indigenous identity. However, a small amount of missing data were present for the household types and income variables (2.2% and 1.5% missing, respectively) while household size had one missing case. Given the small amount of missing data, missing values were simply treated as a category for analytical purposes.

Estimating the size of Toronto’s indigenous population

Estimating the size of Toronto’s indigenous population is challenging and requires a number of assumptions and extrapolation of available information. The general analysis strategy uses the preliminary estimates of Toronto’s indigenous population size, household composition and number of households that are available from the 2011 NHS. These estimates correspond to the metropolitan Toronto area only and are derived from both the census estimates of the population and distribution of indigenous identity from the voluntary NHS.

Using this information, in conjunction with assumptions regarding the proportion of indigenous people who should have completed the census, and our study results obtained through RDS, we calculate revised estimates of the size of Toronto’s indigenous population. These estimates account for the large number of individuals who did not complete the census and were undercounted.

As a preliminary estimate, the NHS estimates the size of Toronto’s urban indigenous population as 19 270, including 15 650 adults (15+) distributed across 12 270 private dwellings.25 From this information, we estimate that each indigenous household contains approximately 15 650/12 270=1.3 indigenous adults (15+) and 19 270/12 270=1.6 indigenous people (including children). Note that these are simplified average estimates and do not require any assumptions on household characteristics or homogeneity of household composition. To ensure a conservative estimate of the population size, we recognise that not all adults are expected to complete the survey as only one responsible adult per household typically completes the questionnaire and individuals in institutional settings (eg, homeless shelters, seniors home, student housing, etc) are counted using institutional records and do not complete the census directly.

Results

A total of 917 indigenous adults who lived, worked or received health services in the city of Toronto, Canada, were recruited and deemed eligible for the study. Of these, 908 were currently living in the city of Toronto and were included in this study. A flow chart of the sample acquisition is included in figure 1. Over the course of the study period, a total of 3505 coupons were issued and 918 were returned. The RDS recruitment network is presented in figure 2, and table 1 reports the number of recruits by seed and number of RDS recruitment waves for each seed.

Figure 2

Our Health Counts Toronto respondent-driven sampling recruitment diagram. Seeds are represented by squares and circles denote recruits. Toronto, Canada, March 2015 to March 2016, indigenous adults.

Table 1

Number of respondent-driven sampling recruits and waves per seed in the Our Health Counts Toronto study

Seed ID	No of recruits (includes the seed in count)	No of waves
00-1-001	18	5
00-1-002	34	9
00-1-004	1	0
00-1-013	15	4
00-1-100	1	0
00-1-156	1	0
00-2-001	196	16
00-2-002	509	19
00-2-003	2	1
00-2-022	1	0
00-2-080	2	1
00-2-121	1	0
00-2-167	24	4
00-2-177	3	1
00-2-181	1	0
00-2-366	1	0
00-3-001	102	10
00-3-002	1	0
00-3-003	1	0
00-3-010	4	3

Baseline demographic characteristics, including indigenous identity, age, gender, income, household type and household size are presented in table 2. RDS design effects ranged from 1.4 to 5.5. From this information, we note that Toronto’s indigenous adults are relatively young with a very low proportion of older adults, specifically, only 3.3% (95% CI 1.5% to 5.2%) of Toronto’s adult indigenous community is 65 years of age or older. Moreover, Toronto’s urban indigenous community suffers from a disproportionate level of poverty as 66.5% (95% CI 65.6% to 67.5%) of individuals live in households which earn less than $C20 000 per year, while nearly 90% of individuals live in households which earn less than $C30 000 per year.

Table 2

Demographic characteristics of Toronto’s urban indigenous population, March 2015 to March 2016

	Unweighted counts	RDS adjusted point estimate, %	RDS adjusted 95% CI
Indigenous identity
First Nations	799	85.7	82.2 to 89.2
First Nations and Metis	8	0.5	0.0 to 2.5
Inuit	11	0.4	0.0 to 3.3
Metis	87	13.2	12.0 to 13.5
Other	3	0.2	0.0 to 0.5
Age, years
15–24	105	21.3	15.8 to 26.8
25–34	201	19.3	12.6 to 26.1
35–44	184	21.5	16.7 to 26.3
45–54	233	24.0	15.0 to 32.9
55–64	138	10.5	7.8 to 13.7
65 and over	47	3.4	1.5 to 5.2
Gender
Female	468	47.8	39.6 to 56.0
Male	423	50.7	42.5 to 58.9
Other	5	0.5	0.0 to 1.3
Transgendered (eg, transgender, transsexual, genderqueer)	12	1.0	0.4 to 1.6
Household income
< $C20 000	564	66.5	65.2 to 67.7
$C20 000 to <$C30 000	135	18.5	17.0 to 20.0
$C30 000 to <$C40 000	52	3.3	0.0 to 8.2
$C40 000 to <$C50 000	37	3.1	0.0 to 10.0
$C50 000 to <$C60 000	28	1.5	0.0 to 3.3
$C60 000 to <$C70 000	17	1.0	0.0 to 9.5
$C70 000 to <$C80 000	5	0.2	0.0 to 1.3
$C80 000 to <$C90 000	6	0.3	0.0 to 5.3
$C90 000 to <$C100 000	9	0.7	0.0 to 1.7
> $C100 000	41	1.3	0.0 to 2.7
Missing	14	3.7	2.7 to 4.8
Household type
Private dwelling	707	75.1	68.9 to 81.3
Institutional*	133	16.7	10.8 to 22.5
Homeless	42	4.6	2.7 to 9.3
Missing	26	2.2	0.9 to 3.6
Household size†
1 person	250	29.6	18.9 to 40.2
2 people	183	30.5	25.0 to 36.0
3 people	124	19.7	16.6 to 28.8
4 people	77	12.5	10.8 to 14.3
5 people	35	3.1	2.0 to 4.2
6 people	16	2.2	0.0 to 4.8
7 people	4	0.5	0.4 to 0.6
8 people	6	0.9	0.5 to 1.3
9 people	1	0.1	0.0 to 0.8
10 or more people	10	0.9	0.0 to 11.4
Missing	1	0.1	0.0 to 0.8

*Institutional includes households that do not complete the census questionnaire, such as homeless shelters, rooming houses, nursing homes and so on, where residences are counted using administrative data.

†Household size was only asked to 707 adults who lived in non-institutional settings.

RDS, respondent-driven sampling.

Continuing to our estimation of Toronto’s indigenous population size, based on our analysis in table 2, a minimum of approximately 68.9% of participants lived in private households that should have completed the census. However, not all adults complete the census, as it’s only completed by one responsible household member per household. Combining this information with the number of adults living per household from the NHS, we can infer that approximately 70%×1/1.3=53.0% of this adult population should have completed the census. After adjusting responses for recruitment using RDS, we have strong evidence that the large majority of urban indigenous people in Toronto did not complete the 2011 census (table 3).

Table 3

Adjusted proportions of Toronto’s urban indigenous population who report completing the 2011 Canadian census under two respondent-driven sampling (RDS) conditions

	Unweighted counts	RDS adjusted point estimate % (95% CI)	Conservative RDS estimates* % (95% CI)
Report completing	221	14.1 (10.1 to 18.2)	18.9 (14.5 to 23.3)
Report not completing	630	81.1 (76.6 to 85.5)	81.1 (76.6 to 85.5)
Don’t know/ don’t remember/unreliable	57	4.8 (2.7 to 6.9)

*Don’t know/remember/unreliable (4.8%) are added to the RDS adjusted point estimate to obtain the conservative estimate of the population size.

To continue with a conservative approach to our estimations, we will assume that those who stated they did not remember if they had completed the census did in fact complete it (4.8% from table 3). Under this set of assumptions, we can now determine that we are conservatively underestimating the number of households in the indigenous community by a factor of 2.8 (53.0%/18.9%) with 95% CI 2.3 to 3.7.

According to the NHS, there are 1.3 indigenous adults (15+) living per census household, suggesting that there are approximately 2.8×1.3×12 270=45 000 (95% CI 37 000 to 59 000) indigenous adults living in Toronto (table 4). From NHS data, we can also infer that each household has 1.6 indigenous people (including adults and children), suggesting that there are approximately 55 000 (95% CI 45 000 to 73 000) indigenous people living in the city of Toronto, Canada (table 4). Moreover, analyses from Our Health Counts Toronto study data show that 40.9% of indigenous adults in Toronto live in households of at least three people and the average indigenous household size is 2.5 people. While the household size counts for Our Health Counts Toronto include all household members (both indigenous and non-indigenous), the assumption of 1.6 indigenous people per household from the NHS is nevertheless likely conservative, given the much larger Our Health Toronto household size calculations.

Table 4

Toronto’s indigenous population based on data from the Canadian census and two estimates from respondent-driven sampling (RDS)

	Census and National Household Survey (2011)	Population size estimates from RDS (95% CI)	Conservative population size estimates* from RDS (95% CI)
Adults	15 650	60 000 (47 000 to 84 000)	45 000 (37 000 to 59 000)
Children	3620	14 000 (10 000 to 19 000)	10 000 (8000 to 14 000)
Total	19 270	74 000 (57 000 to 103 000)	55 000 (45 000 to 73 000)

*Don’t know/remember/unreliable assumed to have completed the census.

Finally, note that we assumed that those who did not remember had actually completed the census. In discussion with our community partners and a small subsample of participants, we suspect this is unlikely as individuals may have been more likely to say they do not remember (instead of ‘no’) due to fear of reprisal as completion of the census was legally required. If we assume that those who did not remember completing the census in fact did not complete it, our estimates of the size of Toronto’s indigenous population increases to 74 000 (95% CI 57 000 to 103 000), suggesting an underestimation of the size of Toronto’s indigenous community by the 2011 Canadian census by a factor of approximately 2.9–5.2 (table 4).

Discussion

In a conservative model of the indigenous population in Toronto, where individuals who did not remember completing the census were assumed to have completed the census, the Canadian census underestimated the size of Toronto’s urban indigenous population by a factor of approximately 2 to 4. This estimate may in fact be too conservative as we incorporated conservative assumptions at each step of the calculation and given completing the census is required by law, participants may have been reluctant to acknowledge not completing the questionnaire.

It has previously been suggested that migration to urban centres and population increases may contribute to the consistent underestimation of this population size,26 but an additional analysis of this data has concluded that among individuals who have lived in Toronto for 5 or more years, only 16.8% (95% CI 12.0% to 21.7%) had completed the census—further supporting the hypothesis of systematic biases rather than mobility or other factors.

Nonetheless, there are several limitations to consider. First, as the data are self-reported and pertain to completion of the previous census, participant responses may be subject to recall bias. A second key limitation is that the baseline data from the NHS were obtained in 2011 and may no longer be valid, nonetheless, this is the most recent data available until the results of the 2016 Canadian census are fully released late 2017. Third, we recognise that there is some heterogeneity in our population with respect to First Nations, Métis and Inuit identities and their social networks. Examination of our recruitment patterns for indigenous identity reveal the predominance of First Nations participants, which is not unexpected, given the known majority of indigenous people who identify as First Nations. We did recruit a substantial number of Métis participants, who were distributed across First Nations recruitment chains, reflecting historic and current social links between Métis and First Nations. It is possible, however, that this represents only a subgroup of Métis and there are other Métis in Toronto who are not socially networked with First Nations. The small number of Inuit participants in our study most likely reflects a strong homogeneity of Inuit-specific social networks, which we were not able to sample successfully. Finally, the limitations of RDS studies continue to be elucidated. As Gile et al24 state, RDS is “designed to enact a near statistical miracle: beginning with a convenience sample … then treating the final sample as a probability sample,” and this does not come without a suitable cost in untestable assumptions. Although we are confident that our study is robust, care must still be taken in the interpretation of RDS studies and generalisation to the target population.

This study provides a cautionary note on the shortfall of national census data on ‘hard-to-reach’ populations in Canada and beyond. In particular, although the short-form Canadian Census had an overall 97.1% response rate27 when it is combined with data from the NHS to obtain estimates by ethnicity, the result is a severe undercount of the indigenous population in Toronto. The results of this study may have potential implications for the enumeration of other impoverished or marginalised groups, including homeless people or illegal immigrants who may not have fixed addresses or who may be reluctant to identify themselves. Accurate population counts are particularly important for socially marginalised groups as specialised health and community services are commonly required to bridge and address their social exclusions. Alternative sampling strategies such as RDS in conjunction with community-based partnerships may be added to the other tools used to ensure that individuals are accurately enumerated across all subgroups and jurisdictions, thereby improving healthcare access and equity for these marginalised communities.