Reliability and consistency of a validated sun exposure questionnaire in a population-based Danish sample.

An important feature of questionnaire validation is reliability. To be able to measure a given concept by questionnaire validly, the reliability needs to be high. The objectives of this study were to examine reliability of attitude and knowledge and behavioral consistency of sunburn in a developed questionnaire for monitoring and evaluating population sun-related behavior. Sun related behavior, attitude and knowledge was measured weekly by a questionnaire in the summer of 2013 among 664 Danes. Reliability was tested in a test-retest design. Consistency of behavioral information was tested similarly in a questionnaire adapted to measure behavior throughout the summer. The response rates for questionnaire 1, 2 and 3 were high and the drop out was not dependent on demographic characteristic. There was at least 73% agreement between sunburns in the measurement week and the entire summer, and a possible sunburn underestimation in questionnaires summarizing the entire summer. The participants underestimated their outdoor exposure in the evaluation covering the entire summer as compared to the measurement week. The reliability of scales measuring attitude and knowledge was high for majority of scales, while consistency in protection behavior was low. To our knowledge, this is the first study to report reliability for a completely validated questionnaire on sun-related behavior in a national random population based sample. Further, we show that attitude and knowledge questions confirmed their validity with good reliability, while consistency of protection behavior in general and in a week's measurement was low.

Introduction

Intermittent and chronic sun exposure from the natural sun and artificial tanning is level I carcinogenic according to the WHO (IARC, 2011). Campaigns aimed at changing UV behavior, by reducing exposure at solar noon and using sun protection like clothe, shade hat and sunscreen, in the general population have been launched in several countries (Koster et al., 2009; Koster et al., 2010; Diffey and Norridge, 2009; Forsea and del Marmol, 2013; Dobbinson SJW et al., 2008; Garvin and Eyles, 2001; Stanton et al., 2004). The effects of these initiatives are generally evaluated by distribution of questionnaires (Saraiya et al., 2004), which are suitable for monitoring representative population-based samples and thus suited, when information regarding knowledge, attitude and behavior is desired. Few studies also used other methods than questionnaire (Andersen et al., 2016; O'Riordan et al., 2008). However, bias (recall, selection, social desirable answers) can potentially limit the reliability of conclusions drawn based on questionnaire data and it is thus essential that questionnaires are evaluated for validity and reliability (Edwards et al., 2009).

Reliability is an important aspect of questionnaire validation. Questionnaires should be able to reproduce results to be valid. Knowledge and attitude are concepts people normally do not change over short periods of time, however behavior questions relate on a specific period and therefore items addressing behavior loses reproducibility by time (Koster et al., 2016a). When evaluating sun related behavior on a weekly basis it is however relevant to know if that week reflects peoples' behavior in general. It has been shown that people might behave differently when being monitored by e.g. a personal dosimeter (Koster et al., 2016a; McCarney et al., 2007; Best and Neuhauser, 2006; Haggerty and Ericson, 2000; Strub, 1989).

Recently the first studies describing criteria validity from objective measurements showed that questionnaires could be applied for evaluation of sun-related behavior. All of these studies used measurement periods in the range of 7–10 days and with short recall period i.e. the respondents answered the questionnaire shortly after participating in the study (Cargill et al., 2012; Sun et al., 2014; Koster et al., 2016b).

Some questionnaires used for evaluations of health interventions aimed at reducing skin cancer have been tested for validity and reliability (Branstrom et al., 2002) and a few studies attempted to validate self-reported measures of UV-exposure by testing behavioral questions against objective measurements or against other self-reported data sources. However, these studies showed that it is possible to measure various aspects of people's behavior in the sun validly (Thieden, 2008; Thieden et al., 2006; van der Mei et al., 2006; English et al., 1998; Dwyer et al., 1996; Lower and Sanson-Fisher, 1998). These studies used diaries to assess sun related behavior, but diaries are not feasible for campaign evaluation, as they are an intervention per se. Recently, we published comprehensive studies of sun exposure criteria validity and of sun related knowledge, attitude and behavior conceptualized scales, which demonstrated high validity and described relevant scales to use as milestones in campaign evaluation (Koster et al., 2016a; Koster et al., 2016b; Koster et al., 2017; Koster et al., 2015a). Important aspects of the validated sun exposure questionnaire were not described, however.

The aims of this study were to examine reliability of attitude and knowledge items and behavioral consistency of a developed questionnaire for monitoring and evaluating population sun-related behavior. To our knowledge, this is the first study to report reliability and behavioral consistency for a completely objectively validated questionnaire on sun-related behavior in a national random population based sample.

Methods

Study design and population

In March 2013, a random sample of Danes in the age 15–65 years was drawn from the Danish civil registration system. An invitation to participate in the study was sent by mail in the end of April. To be eligible to the study potential participants should be able to wear a personal dosimeter wristband for one week of their summer vacation in Denmark in the weeks 19–35 (May–August) and complete an electronic questionnaire afterwards. The invitees signed up on the project page www.mituv.dk and indicated available weeks. Potential participants where then allocated to a participation week and contacted by phone at least one week in advance to receive instructions. Potential participants with more than one summer vacation week were allocated to a low season week, if available, to increase sample utilization. Participants who confirmed their participation by phone were sent a dosimeter including instructions and a prepaid envelope by ordinary mail. After participation they returned the dosimeter for data retrieval and were sent a questionnaire (Q1) the following week to assess their sun-related behavior in the measurement week as well as attitudinal and knowledge deficit scales. After additional four weeks the participants were sent a reliability questionnaire (Q2) including only attitudinal and knowledge deficit scales. Finally, from September 2013 or at least 2 weeks after Q2 participants were sent a questionnaire (Q3) to assess their sun-related behavior during the entire summer. Fig. 1 shows the flow of the project.

Fig. 1

Flow diagram.

The study population was aimed to be representative of the Danish population within gender, age groups (15–24, 25–34, 35–44, 45–54, 55–65) and region. The recruitment of the 15–17-year-olds required parental consent in which case the invitation letter was initially directed to one of the parents. Persons who have inquired not to be drawn for research projects were excluded from the sample.

Ultraviolet dosimeter

The items described in this paper are relatable to objective measurements of UV-exposure. The use of personal dosimetry methods was previously described (Koster et al., 2016b; Koster et al., 2017) as well as their association to items.

Sample size, bias and confounding

The initial sample size was given by the restricted availability of qualified dosimeters in combination with the summer study period as well as a measurement period of 1 week and was previously described (Koster et al., 2016a; Koster et al., 2015a). Potential confounding was accounted for by including personal factors (gender, age region, skintype, education, family history, sun protection behavior) and external factors (Ambient UV, week of participation) in the analysis of questionnaire reported and registered dosimeter data.

Questionnaire and scales

In addition to the questionnaire, Q1 previously described (Koster et al., 2016b), a questionnaire Q2 which was identical, but included only questions on knowledge and attitude was distributed. This questionnaire was used to examine the reliability in a classical test-retest setup (Branstrom et al., 2002; McMullen et al., 2007; Westerdahl et al., 1996). The third questionnaire Q3 included questions on the behavior only. However, the Q3 questionnaire was similar in design to the original evaluations of the Danish SunSmart campaign, addressing the behavior of the past summer (Behrens, 2014). We examined reliability by testing Q2 vs Q1, which compared identical questions and we examined consistency by testing Q3 vs Q1, however Q3 and Q1 compared similar questions only with different time ranges e.g. ‘In the past week did you experience sunburn?’ vs ‘In the past summer did you experience sunburn?’ and ‘In the week of your vacation how much of the time did you use the following protection?’ vs ‘In the past summer when you were of work and the sun was shining – how much of the time did you use the following protection?’ etc. Skin type was assigned according to Fitzpatrick (Fitzpatrick, 1988) by self-evaluated skin tan/burn reaction upon season's first exposure to the sun. The questionnaire was applied in Danish. English translated version of the questionnaires is available as supplement as well (Koster et al., 2016b).

Statistics

For all tests, p values <0.05 were considered statistically significant. We used SAS version 9.3 (SAS Institute, Cary, North Carolina, USA) for the analyses. We compared on item level and on scale level whenever relevant. We calculated Kappa for items and intra class correlation for scales.

The project was sent to The National Committee on Health Research Ethics who decided that their approval was not necessary. Danish Data Protection Agency gave approval number 2012-41-0100.

Results

Participants

In Fig. 1, we show the flow of the study. Six thousand persons were invited and of those 25% signed up for participation. We collected data from 749 successful dosimeter measurements and we received 736 completed questionnaires and for 664 persons we have complete data for both dosimetry and questionnaire Q1 with a response rate of 89%. For these 664 participants 89% and 82% respectively completed Q2 and Q3.

Descriptive data

In Table 1, we show the distribution of demographic characteristics among participants who completed Q1, Q1 and Q2, and Q1 and Q3. The loss of participants was evenly distributed for all characteristics. More women than men were enrolled in the final sample compared to the Danish population. Regarding the other demographic characteristics only minor deviations to representativity occurred.

Table 1

Distribution of demographic characteristics, sunburn, measures of time and UV-exposure in a cross-sectional sample of 664 Danes in Denmark in 2013.

	Respondents Q1	Respondents Q1 and Q2	Respondents Q1 and Q3
	N (%)	N (%)	N (%)
Total	664 (100%)	591 (100%)	546 (100%)
Gender
Male	251 (38%)	221 (37%)	200 (37%)
Female	413 (62%)	370 (63%)	346 (63%)
Age
15–24 years	100 (15%)	80 (14%)	72 (13%)
25–34 years	104 (16%)	94 (16%)	83 (15%)
35–44 years	118 (18%)	105 (18%)	94 (17%)
45–54 years	132 (20%)	120 (20%)	113 (21%)
55–65 years	210 (31%)	192 (32%)	184 (34%)
Skintype
I	54 (8%)	45 (8%)	44 (9%)
II	383 (58%)	342 (59%)	305 (58%)
III/IV	227 (35%)	191 (33%)	163 (34%)
Region
Capital	187 (28%)	170 (29%)	158 (29%)
Zealand	103 (16%)	87 (15%)	87 (16%)
Northern Jutland	68 (10%)	62 (11%)	57 (10%)
Central Jutland	167 (25%)	149 (25%)	131 (24%)
Southern Denmark	139 (21%)	123 (21%)	113 (21%)
Education
Primary school	117 (18%)	100 (17%)	93 (17%)
Secondary school	90 (14%)	78 (13%)	76 (14%)
Vocational	91 (14%)	78 (13%)	70 (13%)
Higher education (<2y)	67 (10%)	55 (9%)	51 (9%)
Higher education (2–4½y)	213 (32%)	197 (33%)	183 (34%)
Higher education (>4½y)	80 (12%)	77 (13%)	68 (12%)
Own or family related melanoma
Yes	147 (22%)	132 (22%)	119 (22%)
No	517 (78%)	459 (78%)	427 (78%)
Weather
1 (Most of the time sunny)	191 (29%)	164 (28%)	150 (27%)
2	176 (27%)	164 (28%)	149 (27%)
3	152 (23%)	133 (23%)	125 (23%)
4	83 (12%)	77 (13%)	70 (13%)
5 (Most of the time clouded)	61 (9%)	52 (9%)	51 (9%)

Sunburn consistency

In Table 2a, we show the distribution of persons sunburned in the measurement week by sunburn for the entire summer. Of the 391 participants not sunburned in the measurement week, 52 (13%) reported to be burnt by the end of the summer. Of the 150 participants burned in the measurement week 96 (64%) did either not recall being burnt by the end of the summer or were not detected as burned by the dichotomized sunburn question. There was 73% partial agreement between burns in the measurement week and the entire summer. Perhaps as much as 82% as it is possible to get sunburnt after the measurement week. The kappa value for burns in the measurement week and the entire summer is 0.25.

Table 2a

Agreement between general summer and specific week measures of sunburn in Denmark in 2013.

n = 541	Sunburn summer Q3
	Yes	No
Sunburn week Q1		p < 0.001
Yes	54 (36%)a	96 (64%)b
No	52 (13%)c	339 (87%)a

Agreement.

Disagreement.

Possible agreement.

Table 2b shows the difference in outdoor time estimated for the week of measurement and estimated as general outdoor exposure time. There is only 27% complete agreement, while a larger part 50% where outdoor longer in the measurement week as compared to their self-estimation of their average outdoor time. A minor fraction (23%) where outdoor shorter time in the measurement week.

Table 2b

Agreement between general summer and specific week measures of outdoor exposure time in Denmark in 2013.

n = 524	General outdoor time summer Q3
	General exposure time = week exposure time - >2 h	General exposure time = week exposure time - 1 h	General exposure time = week exposure time	General exposure time = week exposure time + 1 h	General exposure time = week exposure time + >2 h
Week outdoor exposure time Q1
N (%)	129 (25%)	126 (24%)	143 (27%)	91 (17%)	35 (6%)

Reliability of scales

Table 3 shows means, intraclass correlation coefficient for scales and weighted kappa values for protection behavior as well as phenotypic traits. The scale properties have previously been described (Koster et al., 2017). We showed that most of the attitudinal and knowledge deficit scales showed substantial reliability with values between 0.6 and 0.8, while a few scales only showed moderate reliability 0.4–0.6. We also showed that protection behavior in general is a less reliable measure for protection behavior in a specific week. This is seen for both the protection scale and for the individual protection methods, except for use of sunscreen, which is vaguely moderate with a kappa value of 0.44. Hair and eye color both had kappa values of about 0.9, while number of naevi and skin type where about 0.7.

Table 3

Scale and item scores and reliability of (Top): Q1–Q2 (attitude and knowledge deficit) and consistency of (Bottom): Q1–Q3 (Protection behavior), in Denmark in 2013.

Scale/item (range) n = 581	Mean		Measure	Value
	Q1	Q2
Knowledge UV penetration (0–15)	5.9 (5.7–6.2)	6.2 (5.9–6.4)	ICC	0.58
Knowledge UV types (0–9)	4.5 (4.3–4.7)	4.5 (4.3–4.7)	ICC	0.64
Knowledge UV and vitamin D (0–18)	8.1 (7.8–8.3)	7.8 (7.6–8.1)	ICC	0.62
Knowledge UV risk of melanoma (0–18)	4.2 (3.9–4.5)	5.0 (4.7–5.3)	ICC	0.61
Attitude toward tanned look (0–28)	16.9 (16.6–17.3)	16.8 (16.4–17.2)	ICC	0.75
Perceived barrier toward avoiding sun 12–15 (0–16)	8.6 (8.3–8.8)	8.3 (8.0–8.6)	ICC	0.69
Routine (0–16)	9.5 (9.3–9.8)	9.4 (9.1–9.6)	ICC	0.62
Perceived barrier not tanning (0–16)	6.3 (6.0–6.5)	6.4 (6.1–6.7)	ICC	0.71
Skin examination self efficacy (0–6)	4.3 (4.2–4.4)	4.3 (4.2–4.4)	ICC	0.71
Perceived importance of protection behavior (0−12)	6.1 (5.9–6.3)	6.4 (6.2–6.6)	ICC	0.57
Perceived benefits of protection behavior (0−32)	23.2 (22.8–23.6)	22.9 (22.5–23.3)	ICC	0.52
Perceived barriers clothing (0−20)	8.8 (8.5–9.2)	9.4 (9.0–9.7)	ICC	0.53
Perceived barriers spf (0–12)	3.6 (3.5–3.8)	4.0 (3.8–4.2)	ICC	0.68
Peceived barriers hat (0–12)	5.1 (4.8–5.3)	5.3 (5.1–5.6)	ICC	0.61
Severity of Melanoma (0–12)	2.8 (2.6–2.9)	2.9 (2.8–3.1)	ICC	0.49
Worry about Melanoma (0–12)	3.5 (3.3–3.7)	3.8 (3.6–3.9)	ICC	0.67
Perceived efficiency of skin examination (0–20)	12.7 (12.4–12.9)	12.6 (12.4–12.9)	ICC	0.51
Perceived barriers for skin examination (0–20)	8.1 (7.8–8.3)	8.4 (8.2–8.7)	ICC	0.60

The original values of q3 protection scale was 4.9 (4.7–5.2). We included hat and clothe twice as Q1 protection had two measures of each: cap & wide brimmed hat and long trousers & long sleeves. Statistically significant differences (p < 0.05) are in bold.

Discussion

We have shown important properties of reliability of attitude and knowledge deficit scales and of consistency of protection behaviour and sunburn, for evaluation of exposure to ultraviolet radiation in a population-based sample. First, consistency of questions of sunburn may be underestimated in questionnaires summarizing the behaviour over too long periods or using only dichotomized sunburn questions. Second, people's perceived opinion of their own general protection behaviour is not a reliable measure of protection behaviour in a specific week. Third, reliability of the attitudinal scales were substantially acceptable. Reliability is an important part of the questionnaire validation and emphasizes applicability of the questionnaire.

Strengths and limitations

The strengths of this study are a sample based on the Danish civil registration system, with very high participation and response rates and objective personal dosimetry measurements. Contrary to traditional studies of exposure to ultraviolet radiation based on questionnaires this study reduced recall bias maximally by short measurement periods and short response periods. Persons wearing a dosimeter could be more aware of their behavior and this could change their behavior, however we tested that in a smaller intervention study and we did not find an effect on the behavior from wearing a dosimeter (Koster et al., 2015b). In addition, the number of repeated questionnaire could induce an attitude change or inspire to gain knowledge, however most scale scores were stable. Another limitation of the study is a majority of persons with pale skin types in the study population in northern Europe, which may need to be addressed if applied in other settings e.g. tropic climate, darker skin types. Behavior during sunny vacations comprises a special risk (Agredano et al., 2006; Dos Santos et al., 2009; Koster et al., 2011; Buller et al., 2016). We did however conduct another part of the study in sunny vacations with similar results (reported elsewhere). Skin type was self-assessed and not objectively validated. Consistency of Skin-type between questionnaires was good though. Selection bias is possible if e.g. persons with certain skin type or social status would be more or less likely to participate, however we did not observe that.

Interpretation

To our knowledge, this is the first paper on the reliability and consistency, which included a complete UV-exposure questionnaire validation. We are first to compare agreement between behavior in a specific validated measurement week and perceived general protection behavior. We show reliability of new scales developed (Koster et al., 2016b; Koster et al., 2017) and of previously developed modified, scales (Bränström et al., 2004; Branstrom et al., 2001a; Branstrom et al., 2010). The reliability of the modified scales was similar to the original scales. The lack of agreement of reported sunburn between the measurement week and in general indicates that sunburn should be assessed by thorough questions about sunburn and that the period estimated is of relevance when the questions are posed. To achieve better agreement between actual behaviour and reported behaviour questions beneficially could be phrased ‘on your vacation’ or ‘in the past week’ instead of ‘the entire summer’ or ‘in the past year’. We previously showed that protection behaviour is correlated to the weather and that agreement of the general protection behaviour and the week protection measure was low. This could indicate that the protection behaviour also could achieve higher precision from a week measure rather than a general measure as there is large variation in the weather. More participants reporting longer outdoor time in the entire summer compared to the measurement week could also be a psychological phenomenon biased by past achievements or experiences like persons reporting higher physical activity levels compared to actual measurements (Golubic et al., 2014). The scales reliability confirms the validity as a tool to operate in the process of reducing the skin cancer incidence. The scales were concept validated (Koster et al., 2017; Branstrom et al., 2010; Branstrom et al., 2001b) and their relation to criteria validated questions of the sun-related behaviour was established.

One of the reasons why the week based measure is more valid is that people have perhaps 2–3 weeks of intense outdoor behaviour of relevance to measure, while their general summer behaviour perhaps summarizes 10 weeks of working indoor and the 2–3 weeks of vacation. Danish studies showed that indoor workers received only a small fraction of the UV radiation received by persons on vacation (Thieden et al., 2001), and importantly, especially if going on vacation to southern destination 100% of participants were sunburned (Petersen et al., 2013a; Petersen et al., 2013b).

The questionnaire developed and validated can be found at www.mituv.dk and can be applied by short or long term studies that need to assess the ultraviolet radiation exposure in a study group or population e.g. studies of skin cancer (Birch-Johansen et al., 2010; Birch-Johansen, 2011; Fuglede et al., 2011), sun protection, vitamin D (Kimlin et al., 2014) or even outdoor behavior.

Conclusion

Reliability is an important property of a questionnaire and an important part of the questionnaire validation process. The reliability of the validated questionnaire has provided evidence of the attitude and knowledge constructs that the questionnaire aimed to measure. Thus, the questionnaire presented is a tool to measure predictors of the UV exposure in a population in addition to the actual UV exposure. This demonstrates that it is an important tool to measure relevant predictors for UV exposure. The week-based design minimizes bias from recalling past sun-related behavior. For future studies and evaluations, the week-based design will be expected to give estimates that are more accurate and eventually better knowledge for future preventive interventions or research programs.

Funding

TrygFonden and The Danish Board of Research and Innovation supported this study.

Conflicts of interest

None declared.