A selective follow-up study on a public health survey.

BACKGROUND: The non-response rates in surveys are increasing which is problematic as it means that a progressively smaller proportion of the population represents the majority, and it is uncertain how health survey results are affected. This follow-up was performed on the non-responders to the postal questionnaire in the public health survey Life and Health, conducted in Örebro County Council, Sweden, where large differences in response rates had been found between different socio-demographic groups and geographical areas. The main objective was to analyse non-response bias regarding self-rated health.
METHODS: This follow-up study was conducted as a census to all non-responders in the area that had the lowest response rate and, in one other geographical area used as a control. It was carried out by telephone interviews, 49.3% (580 individuals) answered the follow-up. The outcome variable was self-rated health, a main variable in public health surveys. Differences in response patterns between responders and initial non-responders were approximated by prevalences with confidence intervals and adjusted odds ratios.
RESULTS: Poor health was more common in the initial non-response group than among the responders, even with consideration given to sex, age, country of birth and education. However, good health was equally common among responders and initial non-responders.
CONCLUSIONS: Public health surveys can be biased due to certain groups being under-represented or not represented at all. For this reason, in repeated public health surveys, we recommend selective follow-ups of such groups at regular intervals.

Introduction

In order to be able to say anything about public health and its determinants, national and regional surveys are carried out and these are then used as the basis for planning and research. For a number of years, however, the number of people who participate in these surveys has diminished which is both a national and international trend in developed countries. This is problematic as it means that a progressively smaller proportion of the population represents the majority, and it is uncertain how survey results are affected. In the Swedish national public health survey ‘Health on Equal Terms’, which is a postal questionnaire, the non-response level increased from 39% in 2004 to 48% in 2009. Another postal public health survey ‘Life and Health’ carried out since 2000 in five Swedish counties had a non-response level of 35% in that year. In 2004, it had increased to 36% and to 41% in 2008. In Sweden, the non-response in The Labour Force Surveys has increased from <2% in the year 1970 to >18% in 2008, which means a non-response increase of ∼0.44 percentage points annually. Internationally, Morton et al. have calculated that in cross-sectional surveys, there has been an average reduction of 0.67 percentage points per year in response rate during the period 1970–2003.

The non-responders which are the focus here consist of people who for various reasons do not participate in the survey. The problem is that one does not know if those who do respond are representative of the groups being investigated as the non-responders can be deviating in one or several respects. The main reasons for non-response are that the selected people refuse to participate or that they are not reached. These people represent two different groups whose non-participation can have different consequences for the survey results.,

We know from the Life and Health surveys that women respond to a higher degree than men, the middle-aged more than the young, individuals with a higher level of education more than those with a lower level, Swedish-born more than foreign-born and those living in residential areas more than those in blocks of flats. This therefore entails a bias in the responses regarding demographic and socio-economic conditions. This has been confirmed in a number of other studies. There are also studies which show that non-responders have higher levels of morbidity and poor self-rated health, higher levels of mortality and higher proportions of daily smokers.

Several studies show that one can not assume that higher levels of response provide different and ‘safer’ results, and concentrating on the response rate is therefore insufficient to determine the quality of survey data. In a meta-analysis of 59 surveys, no clear relation was found between the level of response rate and non-response bias., Certain studies with a response rate ∼20% showed a similar bias to studies with >70% response rate. Deviations were more likely related to the variables being investigated. In line with this, Stang and Jöckel confirm that studies with a low response rate can have less systematic bias than those with high response rate, especially in situations in which the prevalence of a phenomenon increases after various collection waves in a study. Therefore, there exists no minimum level of response rate at which the non-response bias is expected to be too large to make inferences about a population from a survey sample. And correspondingly, it is not possible to say that there is an upper response rate at which the results are not unevenly distributed due to non-response.

What is known from a number of studies is that prevalence estimates can be influenced by non-response but associations are not necessarily affected. For example, it is confirmed in the Oslo Health Study that unhealthy people did not participate to the same degree as others but that the association between social inequalities in health in relation to various socio-demographic conditions remained. A Finnish study shows that survey non-response did not seriously bias analysis of social class inequalities in sickness absence and possibly health inequalities more generally. In de Winters’ study, response rate increased from 66% to 76% which changed the prevalence of mental health but this had no significance in the association found between individual characteristics and mental health.

Health surveys are usually carried out using probability sampling in which there is a known degree of probability of individuals being included. The statistical methods widely used for population estimates presume that the entire sample will participate in the survey. As we know, this is not the case and instead, we must deal with a large degree of non-response that may generate bias. Potential non-response bias can, to some extent, be estimated and/or corrected for in various ways. An approach relies on standard statistical methods for treatment of non-response in surveys: imputation or calibration weighting. Other approaches are either to estimate the non-response bias by comparing early and late responders or to conduct a non-responders follow-up. When follow-ups are carried out, they usually take the form of a new probability sample. An alternative method which we have used in this study is to carry out a selective follow-up which is performed as a census of the non-responders in the area with the lowest response rate.

A key objective of this study is to investigate whether there is any non-response bias regarding self-rated health. Further objectives are to study the generalizability of the results for areas with a different socio-demographic structure, to identify effects of calibration weightings, and to study the change of response rates for different socio-demographic groups by using a follow-up.

Methods

The survey Life and Health is a public health survey which has been carried out by five county councils in Sweden in the years 2000, 2004 and 2008. It addresses persons in the age group of 18–84 years with questions on health, lifestyle and living conditions. The sampling procedure used was a random sample stratified by sex, age group and geographical area. The data were collected by a postal questionnaire. In 2004, Örebro County Council decided to carry out a non-response follow-up as large response rate variations were evident between geographical areas and different socio-demographic groups.

In this study, it was decided to make a selective sample without the use of probability sampling in order to study the area characterized by especially low response frequencies as a whole, but also within various socio-demographic groups. The area concerned consists of blocks of flats, has many people on low incomes, many with a low level of education, foreign-born and single-parent households and in this respect it is an unfavourable area in Örebro municipality. There is evidence that there is an association between low response rate and deprived areas., The ‘Study area’ consisted of non-responders who lived in blocks of rental flats in the municipality of Örebro, Sweden. This grouping of neighbourhoods has been used in previous studies,, and these areas consist of small administrative geographical areas, key-codes, with similar characteristics but not necessarily adjacent. Another of the county's medium-sized municipalities was selected in order to function as a ‘Control area’. The sample size in these two areas was 3120 persons (population size 28 484). After identifying over coverage in the initial sample of 104 persons, non-responders in these areas amounted to 1188 and all were included in the follow-up. The study population is described in more detail in table 1.

Table 1

Population and sample size, response rate, number of initial responders and initial non-responders in different areas

	Population	Survey sample	Initial responders	Initial non-responders
	N	n (response rate, %)	mr	mn−r
Region	1 043 350	68 460 (64)	43 589
County	208 091	17 160 (63)	10 781
Study area	20 777	2080 (58)	1207	359
Control area	7707	1040 (60)	621	221

Population (N), survey sample size (n), initial response rate (after consideration of over-coverage), initial responders (postal survey) (mr) and initial non-responders (interview) (mn−r) for study and control area.

The telephone interview

During November–December 2004, Statistics Sweden carried out a follow-up among everyone in the selected areas who had not responded to the postal questionnaire. Those contacted were requested in a telephone interview (after up to five contact attempts by telephone), to answer a limited number of questions from the questionnaire.

The individuals in the sample were informed that the information regarding sex, age, geographical area, education level, occupation and native country obtained from the Swedish official registries would be linked to their answers if they responded to the interview. The responders thus accepted the linking of official registry data to the questionnaire data by informed consent. The personal identification numbers were deleted directly after the record linkage with the national registers and the interview data are thus anonymous.

After identifying an additional over-coverage of 12 persons the non-responders in the follow-up were 596 persons. There were 326 persons who for various reasons could not be reached, 96 persons who were unable to participate due to disability or difficulties with the language and 172 persons who refused to participate in the survey, 2 persons claimed that they had participated in the main study. A total of 580 of the initial non-responders answered to the interview, response rate 49.3%.

Measures

In this study, self-rated health is used as the outcome variable. This was measured, both in the postal questionnaire and the interview, with the question ‘How do you rate your general health?’ with the options ‘Very good’, ‘Good’, ‘Neither good nor poor’, ‘Poor’ and ‘Very poor’. The two last alternatives were used to estimate poor self-rated health and the first two to estimate good self-rated health. Other data used were registry data from Statistics Sweden regarding the responders’ sex, age, native country, education and employment status. In order to avoid too small study groups, country of birth is divided into three groups: people born in Sweden, people born outside Sweden but in the Nordic countries (Denmark, Finland, Iceland, Norway) and people born outside the Nordic countries. Education is categorized into low level of education (≤9 years), medium level (comprehensive school up to 12 years), high level (>12 years) and unknown. The last category applies mainly to some 75- to 84-year-olds and people born outside the Nordic area. Employment status is divided into four categories: gainfully employed, student, unemployed and other. The last group includes disability pensioners, retired people, those on parental leave and domestic workers.

Statistical procedures

Population weightings have been created in order to estimate prevalences at the population level. For questionnaire data, calibration weightings have been calculated., The weightings were produced by Statistics Sweden with the help of information from registers of the total population. Apart from adjustment for various selection sizes in the different strata, the register data are used for calibration of non-response bias for various groups of individuals. The calibration weightings are constructed by use of population level information on a set of auxiliary variables to adjust the design weightings. The calibrated weightings are based on auxiliary population totals and the individual value for the auxiliary variables for the survey responders. The register data used to produce the calibration weightings are sex, age, education level, employment status, country of birth and year of immigration.

The differences in the response patterns for self-rated health between the responders vis-à-vis the initial non-responders and the influence of calibration weightings are approximated by studying the overlap of the point estimates [95% confidence intervals (95% CI), SAS 9.2, Proc survey means]. Logistic regression (SPSS 17.0, Enter method) is used to study the effect of the initial non-responders compared with the responders adjusted for sex, age, country of birth and education with the dependent variable self-rated health. By using these methods in Study and Control area, the differences in response patterns are used to analyse the generalizability of the results for areas with different socio-demographic structure. The response rates for different socio-demographic groups before and after the follow-up are presented by percentage points.

Results

The results indicate that poor health is significantly more prevalent in the initial non-response group compared with responders in the total group for both the Study and Control areas (table 2). This is also the case among women in the Study area. When the responses from the initial non-responders are added to the responders, there is no longer any indication that the proportions of poor health differ between the two groups. The results also show that there are certain differences between the areas studied regarding level of education and employment status (not in table). Among the initial non-responders in the Study area, the proportion with low educational level is significantly higher (95% CI for initial non-responders: 29.2–40.4 and responders: 21.7–28.6) and the proportion of gainfully employed significantly lower (95% CI for initial non-responders: 36.6–47.0 and responders: 47.8–55.1). This is not the case in the Control area. The calibration weighting of the responders tend to increase the proportion of poor health and decrease the proportion of good health.

Table 2

Percentage proportion (P) and 95% CI for the proportion to have poor/very poor self-rated health or good/very good self-rated health by sex, response group and area

		Study area		Control area
		P (%)	95% CI	P (%)	95% CI
Poor/very poor self-rated health
Women	Responders	9.5	7.05–11.87	7.7	4.73–10.72
	Initial non-responders	22.4	15.87–28.91	17.8	10.28–25.24
	Responders+initial non-responders	12.0	9.61–14.34	9.7	6.80–12.64
	Responders calibrated	10.3	7.66–12.99	8.8	5.38–12.19
Men	Responders	12.0	8.65–15.31	4.9	2.43–7.34
	Initial non-responders	17.0	11.62–22.41	12.7	5.92–19.52
	Responders+initial non-responders	13.4	10.63–16.26	7.0	4.44–9.50
	Responders calibrated	12.9	9.18–16.58	5.4	2.64–8.08
Total	Responders	10.7	8.63–12.70	6.3	4.34–8.19
	Initial non-responders	19.8	15.54–24.07	15.1	10.09–20.16
	Responders+initial non-responders	12.7	10.86–14.51	8.3	6.37–10.22
	Responders calibrated	11.6	9.30–13.80	7.0	4.85–9.18
Good/very good self-rated health
Women	Responders	65.7	61.66–69.75	65.7	60.46–70.92
	Initial non-responders	55.8	47.84–63.74	62.4	53.08–71.79
	Responders+initial non-responders	63.6	59.98–67.17	65.1	60.57–69.68
	Responders calibrated	64.7	60.57–68.83	63.7	58.31–69.12
Men	Responders	70.3	65.63–75.06	72.0	66.78–77.16
	Initial non-responders	67.1	60.80–73.48	71.7	63.08–80.26
	Responders+initial non-responders	69.2	65.50–72.83	72.4	68.16–76.70
	Responders calibrated	69.4	64.48–74.30	71.1	65.75–76.42
Total	Responders	67.9	64.84–71.01	68.9	65.24–72.61
	Initial non-responders	61.3	56.12–66.38	67.3	60.93–73.60
	Responders+initial non-responders	66.3	63.70–68.84	68.9	65.79–72.03
	Responders calibrated	67.0	63.76–70.14	67.5	63.72–71.33

The difference which exist between the ‘Responders+initial non-responders’ and the ‘Responders’ remains when compared with ‘Responders calibrated’ but the differences tend, however, to be somewhat smaller.

Also in the logistic regression, adjusted for the background variables sex, age, country of birth and education level, it was clear that for those in the initial non-response group there was a higher probability of poor health. This was evident in both Study and Control area. When it comes to good health, there are no statistically significant variations between the response group and the initial non-response group (table 3).

Table 3

OR and 95% CI for for having poor/very poor or good/very good self-rated health adjusted for sex, age, country of birth, educational level and employment status by area

Poor/very poor self-rated health	Study area	Control area
	OR (95% CI)*	OR (95% CI)**
Responders	Ref.	Ref.
Initial non-responders	2.01 (1.43–2.82)	2.33 (1.35–4.03)
Women	Ref	Ref
Men	0.86 (0.63–1.16)	0.70 (0.41–1.18)
Age (years)
18–34	Ref.	Ref.
35–49	3.40 (1.96–5.89)	1.36 (0.50–3.70)
50–64	3.77 (2.16–6.59)	4.01 (1.70–9.47)
65–84	3.12 (1.72–5.69)	1.16 (0.42–3.22)
Sweden	Ref	Ref
Nordic countries	1.80 (0.89–3.65)	2.05 (0.81–5.17)
Outside Nordic countries	2.30 (1.62–3.28)	2.21 (0.78–6.26)
Higher educational level	Ref.	Ref.
Medium educational level	1.33 (0.80–2.20)	1.10 (0.42–2.88)
Low educational level	2.28 (1.36–3.83)	2.08 (0.78–5.53)
Educational level unknown	2.49 (1.32–4.71)	6.73 (2.00–22.64)

*Multiple Logistic regression, n = 1524, Nagelkerke R2 0.114; **Multiple Logistic regression, n = 820 Nagelkerke R2 0.125; ***Multiple Logistic regression, n = 1524, Nagelkerke R2 0.113; ****Multiple Logistic regression, n = 820, Nagelkerke R2 0.129

The Odds ratios that are significantly above or under one are all marked in bold.

When consideration is given to the initial non-response group, the proportion of responders born in Sweden, as well as those born outside the Nordic countries increased by more than 20 percentage points. The highly educated had the highest response rate in the postal questionnaire and this also is the case when complemented with the responses from the non-response interviews. For those with a low level of education, however, the response rate increased by a higher degree. Men increased their response rate more than women and the younger more than the older (table 4). Especially the group younger men who were the most under-represented in the questionnaire survey achieved a response rate of 70%, which represents an increase of 33 percentage points (not in table).

Table 4

Response rates (percent) by sex, age, education level, employment status and country of birth for Responders and Total response (Responders + Initial non-responders responders) in Study and Control area

	Responders (%)	Total response (%)
Women	62	80
Men	49	74
Age (years)
18–34	48	76
35–49	53	74
50–64	65	80
65–79	65	79
80–84	59	72
Low educational level	48	72
Medium educational level	58	80
Higher educational level	67	86
Educational level unknown	48	60
Gainfully employed	61	82
Student	45	76
Unemployed	59	82
Other employment status	51	68
Born in Sweden	59	80
Born outside Sweden but in the Nordic countries	57	73
Born outside the Nordic countries	44	68

Discussion

The main reason this follow-up was performed on parts of the non-response group in the ‘Life and Health’ survey was the large differences in response rates found between different response-groups and areas, and also the general decline in response rates in public health surveys experienced both nationally and internationally in developed countries.,, This situation is problematic as it means that a progressively smaller proportion of the population represents the majority, and it is uncertain how this affects survey results. It is questionable if it is possible to rely on the results of public health surveys that have a non-response rate as large as ≥40%, which is the case in the ‘Life and Health’ survey. We decided to carry out this follow-up study as a census in the area that had the lowest total response rate as a whole and also in relation to socio-demographic groups where we expected to find the greatest variations in the response pattern between responders and non-responders.

In the study that was carried out by telephone interviews, 49.3% of the follow-up sample answered. In comparison with several other non-response follow-ups, this is a high proportion. In various Swedish non-response surveys carried out during the 21st century concerning public health questionnaires, the proportion has varied between 7% and 27%. The comparatively high number of contact attempts (five) and the small number of questions asked were certainly factors that contributed to increasing the response rate. When the initial non-responders were added to the responders, the total number of the response group was substantially increased. In spite of this there are still half of the non-responders we know nothing about which is a weakness. There is a risk that these people deviate both in regard to the socio-demographic data and to the response pattern from those who responded. The most common reason for the selected persons not being included in the follow-up was that they were not reached (54.7%), which to a large extent was due to the fact that there was a lack of information regarding telephone numbers. This is a factor that can be expected to be of increasing importance as the number of people with a fixed telephone is diminishing in the population.

Self-rated health is an important variable in public health surveys and poor health is often used as the outcome measure in analyses of associations with different determinants of health. The results show an underestimation of poor health in the initial non-response group which is in line with other surveys., This is the case even with consideration given to sex, age, country of birth and level of education. It does not, however, apply to good health where there was no significant difference between questionnaire responders and the initial non-response group either for the proportion with good health or for the adjusted odds ratio (OR). Depending on which part of the variable that is studied, the result is therefore different. This is interesting considering that the salutogenic perspective is the main focus in public health promotion. Here, the starting point is the determinant for good health and not poor health.

In the comparison of the two areas, significant differences are seen between those who responded to the main survey postal questionnaire and those who responded in the interviews of initial non-responders concerning level of education and employment status. The initial non-responders in the Study area had lower level of education, lower proportion of gainfully employed and a higher proportion of unemployed. This can but does not necessarily lead to systematic errors in the results., The differences found in the Study area could not be seen in the Control area. A conclusion that can be reached from this is that one should be careful concerning generalizations regarding other areas with different population structures.

In the ‘Life and Health’ survey, calibration weighting was used to adjust for the non-response., The statistically significant variations which exist between the initial non-responders and the questionnaire responders remained even when compared with the calibration-weighted questionnaire responses but the differences tend, however, to be somewhat smaller. If calibration weightings are provided using good auxiliary variables, this is a more cost-effective method than adjusting for these differences by using follow-ups. It is a question of resource allocation in large-scale surveys. It is also plausible that the two different methods, postal questionnaire and telephone interview, themselves generate different responses when it comes to questions about health. From the results, it is clear that those in the initial non-response group had a higher probability of poor health. This is in line with other studies., The use of calibration weighting instead of complementing the questionnaire data with interview data from the non-response survey means that one avoids the uncertainty that results from the combination of the two methods. Due to the reasons mentioned, we therefore recommend calibration weighting for the adjustment of variations in response frequencies in large-scale surveys. The use of non-response follow-ups in interview form should mainly be used to estimate non-response bias, to increase understanding and to facilitate the interpretation of the results obtained.

The follow-up contributed to an increased response rate in all socio-demographic groups. It also decreased the large differences that were seen from the postal questionnaire between men and women, younger and older, lower and higher levels of education, Swedish-born and foreign-born and between various employment groups. Variations in the likelihood of response between different groups can accordingly be levelled out with follow-ups in which a small number of questions are put in telephone interviews.

Can we use the results from public health surveys that have progressively lower response frequencies? Yes, we believe we can. We must be aware, however, that there can be bias in the results due to certain groups being under-represented or perhaps not represented at all. This is especially important when studying certain characteristics for which there can be assumed to exist bias, such as poor health. For this reason in public health surveys, we recommend selective follow-ups of under-represented groups at regular intervals.

Funding

The County Council of Örebro, Sweden funded the population survey and financed this study.

Conflicts of interest: None declared.

The non-response group has a significantly higher proportion of poor self-rated health even with consideration given to sex, age, country of birth and level of education. There is, however, no significant variation in the case of good self-rated health.

Results from a selective non-response follow-up in an under-represented area cannot be automatically transferred to another area.

The calibration weighting tended to adjust the proportions of self-rated health towards the proportions of self-rated health among the initial non-responders.

The follow-up contributed to an increased response rate in all socio-demographic groups. It also decreased the large differences that were seen from the postal questionnaire between men and women, younger and older, lower and higher levels of education, Swedish-born and foreign-born and between various employment groups.

In health surveys, selective non-response follow-ups carried out at regular intervals are valuable in order to estimate non-response bias.