An international measure of awareness and beliefs about cancer: development and testing of the ABC.

OBJECTIVES: To develop an internationally validated measure of cancer awareness and beliefs; the awareness and beliefs about cancer (ABC) measure. DESIGN AND
SETTING: Items modified from existing measures were assessed by a working group in six countries (Australia, Canada, Denmark, Norway, Sweden and the UK). Validation studies were completed in the UK, and cross-sectional surveys of the general population were carried out in the six participating countries. PARTICIPANTS: Testing in UK English included cognitive interviewing for face validity (N=10), calculation of content validity indexes (six assessors), and assessment of test-retest reliability (N=97). Conceptual and cultural equivalence of modified (Canadian and Australian) and translated (Danish, Norwegian, Swedish and Canadian French) ABC versions were tested quantitatively for equivalence of meaning (≥4 assessors per country) and in bilingual cognitive interviews (three interviews per translation). Response patterns were assessed in surveys of adults aged 50+ years (N≥2000) in each country. MAIN OUTCOMES: Psychometric properties were evaluated through tests of validity and reliability, conceptual and cultural equivalence and systematic item analysis. Test-retest reliability used weighted-κ and intraclass correlations. Construction and validation of aggregate scores was by factor analysis for (1) beliefs about cancer outcomes, (2) beliefs about barriers to symptomatic presentation, and item summation for (3) awareness of cancer symptoms and (4) awareness of cancer risk factors.
RESULTS: The English ABC had acceptable test-retest reliability and content validity. International assessments of equivalence identified a small number of items where wording needed adjustment. Survey response patterns showed that items performed well in terms of difficulty and discrimination across countries except for awareness of cancer outcomes in Australia. Aggregate scores had consistent factor structures across countries.
CONCLUSIONS: The ABC is a reliable and valid international measure of cancer awareness and beliefs. The methods used to validate and harmonise the ABC may serve as a methodological guide in international survey research.

Knowledge and beliefs about cancer have been implicated in delay in symptomatic presentation which can reduce the proportion of cancers diagnosed at an early stage and lead to lower survival rates.

Existing studies of knowledge and beliefs have focused on individual countries, but international differences may also contribute to international differences in cancer survival.

This paper describes the development of an internationally validated measure of cancer knowledge and beliefs; the awareness and beliefs about cancer (ABC) measure.

The ABC is a reliable, validated instrument for measuring knowledge and beliefs about cancer.

Validated versions have been developed for six countries (the UK, Australia, Canada, Sweden, Denmark and Norway) and in five languages (English, Canadian French, Swedish, Danish and Norwegian).

The development of the ABC provides a blueprint for other countries that wish to examine their citizens’ ABC.

The approach used to validate this international measure, which has evolved from current guidelines, should be useful to other researchers carrying out international surveys.

Improvements could still be made to the measures as a small number of items did not perform well in one or the other of the reliability or validity tests.

Initial item selection for the ABC measure was hampered by a lack of pre-existing validated measures of cancer beliefs.

Introduction

Internationally, there are variations in cultural attitudes to cancer, provision of public education about cancer and delivery of healthcare. These may shape awareness or beliefs about cancer, and ultimately lead to differences in early detection behaviours that could partly explain international differences in the proportion of cancers diagnosed at an early stage.1 2

An internationally valid measure of awareness and beliefs about cancer (ABC) is essential to take forward this research. There are challenges associated with using measures developed in one language and culture in international surveys,3 4 but guidelines are available to support the development of internationally reliable and valid measures.3 5 One key issue is that direct, word-for-word translations are not always appropriate where the aim is to be culturally sensitive and yet assess the same constructs. Psychometric properties also need to be checked in each country where it is used to ensure that any observed international differences are not due to measurement error.6

In the cancer field, a variety of human and system factors have been identified as potential determinants of patient delay in the presentation of cancer symptoms. Lack of awareness of possible early cancer symptoms can influence attributions and interpretation,7 and reduce the chance of rapid help-seeking.8 Research in the UK has shown that lower symptom awareness is associated with a lower likelihood of attributing symptoms to cancer, and longer intended delay in help-seeking.7 9 10 Fatalistic attitudes about survival have also been linked with lower screening uptake and greater delay in symptomatic presentation.11 12 Perceived or actual barriers to accessing medical care may also play a deterrent role.7 9 We have only found one validated measure of cancer awareness,13 and it did not include items on beliefs or attitudes, and has not been assessed internationally.

This paper describes the development and validation of an international measure: ABC measure. The ABC was developed systematically to maximise equivalence across the countries of the International Cancer Benchmarking Partnership (ICBP; Australia, Canada, Denmark, Norway, Sweden and the UK) to identify differences in awareness and beliefs that could explain established differences in cancer survival.

Methods

Item selection

Item selection for the ABC was informed by theoretical frameworks outlining processes of patient delay,14 15 the English Department of Health's National Awareness and Early Diagnosis pathway16 and existing surveys.13 17–19 An initial item pool was assessed by the ICBP Working Group with representatives from all participating countries. This resulted in selection of 32 ‘core’ items, plus modules on cancer screening (8 items) and risk factors (13 items). The ABC was designed to be administered by telephone interview in order to be practical for data collection across diverse geographic areas.

The core measure included (1) awareness of cancer symptoms (1 open ‘symptom recall’ item, 11 closed ‘symptom recognition’ items), (2) awareness of cancer outcomes (4 items), (3) help-seeking intentions (4 items), (4) beliefs about cancer; including beliefs about outcomes and the value of early presentation (6 items), (5) beliefs about barriers to symptomatic presentation (5 items) and (6) estimated age at which people are most likely to develop cancer (1 item). The optional modules were awareness of risk factors for cancer (13 ‘risk factor recognition’ items) and beliefs and behaviour in relation to breast and colorectal screening (8 items). One module assessing ovarian cancer awareness was administered only in Wales (Brain KE, personal communication, 2012). The full and final version (postvalidation) of the UK ABC measure, including response categories, can be viewed in the online supplementary file.

Four aggregate scores were developed during testing. Symptom awareness and risk factor awareness aggregate scores were calculated by adding the total number of items endorsed from the recognition lists. ‘Barriers to symptomatic presentation’ and ‘beliefs about cancer outcomes’ aggregate scores were created using weights derived from factor analyses (see section on International validity for a full description of the above).

Pilot testing

The draft ABC was piloted in UK English before translation and testing in Australian English, Canadian English, Canadian French, Danish, Swedish and Norwegian. Data collection for the validated studies took place in January–May 2011 and was carried out by a market research company (Ipsos MORI, London, UK) and members of the ICBP Working Group. Statistical analyses used the IBM SPSS V.19 and Stata V.12 computer software packages.

UK testing: reliability and validity

Cognitive interviews were carried out by telephone with 10 people aged ≥50 years; this age group was selected because it was the primary target group for the international survey. A trained interviewer asked participants to read each item and answer it while ‘thinking aloud’.20 This process identified items where the participant's understanding differed from the interviewer's, or where the items caused confusion or distress.

Test–retest reliability was assessed in 97 adults aged ≥50 years (55% women, 45% ≥65 years), who completed the ABC on the telephone twice, 2 weeks apart. Agreement over time was assessed with a combination of linear weighted-κ for individual (ordinal) items, and intraclass correlations (ICC consistency) for aggregate (continuous) scores (total number of symptoms/risk factors recognised, barriers to symptomatic presentation and beliefs about cancer outcomes).21 The magnitude of the associations was judged according to classifications proposed by Landis and Koch22 and Cicchetti.23 Internal consistency of aggregate scores was assessed with Cronbach's α.

A content validity assessment was carried out by six independent academic researchers with expertise in psychometrics or cancer awareness, to assess both the clarity and the relevance of the items against the predefined constructs.24 Each item was scored (1=poor to 4=excellent) on clarity and on relevance to the construct. Raters were asked for comments on any items scoring less than 3. For each item, and each set of items measuring any one construct, we calculated a Content Validity Index (CVI) (number of raters giving rating of 3–4/total number of raters). There is no universal agreement on the definition of an adequate CVI score; we used the criterion of >78%25 because with six raters, this allows only one rater to score below ‘3’ on any item, following the recommendation by Lynn.26

International testing: quality and validity of translated measures

The ABC was then translated into Danish, Swedish, Norwegian and Canadian French. Minor amendments to UK English version were also made for the Australian and Canadian English versions. Translations were done by native speakers who also spoke good English and were familiar with medical terminology, and were checked by bilingual members of the ICBP Working Group. Items were rephrased as needed to reflect differences in healthcare systems or other cultural contexts.

We aimed for conceptual and cultural equivalence rather than a verbatim translation. The process used to achieve this was informed by the WHO guidelines,4 the International Workshop on Comparative Survey Design and Implementation (http://www.csdiworkshop.org/), as well as other guidelines for cross-cultural adaptation.5 We also drew on processes followed in other international studies27 28 to develop a method appropriate to the current project.

Up to 10 (range=8–10) cognitive interviews using the final versions were carried out with people aged 50+ in each country except for Canada, which carried out 10 each in French and English. Interviews took place over the telephone except in Denmark, where they were face-to-face. The interviews led to minor changes which were discussed collaboratively with members of the ICBP Working Group in each country to reach consensus.

Quantitative assessment of equivalence of meaning

Quantitative assessment of equivalence and relevance was carried out for the French, Swedish, Danish and Norwegian versions of the ABC.29 The process was analogous to the UK content validity testing, but focused on conceptual and cultural equivalence. Each country assembled 4–8 experts in the field who were fluent in both the relevant target language and English and who were not members of the ICBP Working Group.25 30 They scored the items (and introductory sentences and response options) from 1 (poor) to 4 (excellent) on equivalence of meaning comparing the local translation to the UK ABC, and relevance to the constructs as understood within each cultural setting. A CVI score was calculated in the same way as for the UK (see above) with a cut-off of 78% representing adequate agreement.

Bilingual cognitive interviewing

Three further telephone cognitive interviews were carried out with non-expert bilingual translators (the target language plus English) for each version of the ABC. This checked that changes made throughout the adaptation process had not led to divergence of meaning. At this stage, members of the ICBP Working Group from the three Scandinavian countries also carried out a check to ensure equivalence in Danish, Norwegian and Swedish because of their close linguistic links.

International validity of items and constructs

As part of the validation process, we explored the equivalence of the data collected across participating countries. After completing a tendering process, ABC surveys were carried out by a market research company (Ipsos MORI) between May and September 2011 using computer-assisted telephone interviews. The aim was to draw population-representative samples with a minimum of 2000 people aged 50+ completing the core ABC measure in each country. The method is described briefly here and further details can be found elsewhere.31 In the UK, Canada and Australia, random samples of telephone numbers were drawn from commercial listings. The final two digits of each selected telephone number were replaced with two random numbers, which brought unlisted numbers into the sampling frame. Where there was more than one eligible person living in the household, one person was selected using the ‘Rizzo’ method to ensure an equal chance of selection for interview.32 In the Scandinavian countries random samples of eligible people were drawn from complete population registers and linked to commercial telephone listings. Additional age ranges and extra modules were completed in some countries (see table 1). Ethical approval for the survey was secured via local ethics boards in each participating jurisdiction, with the exception of Norway where ethical approval was not required. Interviewers took part in survey-specific training to ensure consistency and sensitivity. In total, 19 079 people completed the interview. The response rate (American Association for Public Opinion Research response rate formula3; http://www.aapor.org) ranged from 23% in Norway to 47% in Australia. Samples were roughly population-representative for age, but women were over-represented (59% vs 53% in the population aged 50+). Weights were calculated and applied, where appropriate, to adjust for recruitment method, population size and non-response bias.

Table 1

Participation in the international survey of cancer awareness and beliefs

	England	Northern Ireland	Wales	Australia NSW	Australia Victoria	Canada	Denmark	Sweden	Norway
Core ABC (2000 aged 50+)	✓	✓	✓	✓	✓	✓	✓	✓	✓
Module 1—screening (2000 aged 50+)	✓	–	–	✓	–	–	✓	✓	✓
Module 2—risk awareness (2000 aged 50+)	–	✓	–	–	–	–	✓	✓	✓
Module 3—ovarian cancer awareness (women aged 50+)	–	–	✓	–	–	–	–	–	–
ABC—younger sample (1000 aged 30–49)	–	–	–	–	–	–	✓	✓	✓

ABC, awareness and beliefs about cancer; NSW, New South Wales.

We assessed how ‘well’ the items performed (discrimination and difficulty) by examining the consistency of response patterns across countries using the full international dataset. For each country, we identified items with very low variation (≥95% of participants answering in the same category). In a second analysis we examined the proportions of ‘invalid’ responses. In the main, we considered ‘don't know’, ‘refused’ or a blank (no answer recorded) as ‘invalid’.[i] We judged that where >10% of responses were ‘invalid’, the item was too difficult. Large variations between countries in this analysis were taken as an indication that items might not be equivalent, and therefore that good functional translation might not have been achieved.

We explored the performance of the two variables derived as aggregated combinations of items weighted to align with latent constructs. Items selected for inclusion in the aggregate scores had adequate mutual intercorrelations. ‘Barriers to symptomatic presentation’ combined four items (too embarrassed, too busy, worried about wasting the doctor's time and worried about what the doctor might find). ‘Beliefs about cancer outcomes’ combined three items (‘cancer can(not) often be cured’, ‘a diagnosis of cancer is a death sentence’ and ‘people with cancer can(not) expect to continue with normal activities and responsibilities’). Having checked that the Cronbach's α for the items in each score were above 0.45, factor analysis was applied to the mutual item intercorrelations to identify the latent factor that maximally accounted for common variation.33 Means of the six country-specific factor-based weighting coefficients were used as the coefficients for the overall aggregate scores to be applied across all countries. Validity was defined in two ways: (1) a unipolar, general factor should account for a proportion of the communality exceeding the baseline level by 40% in each country and (2) the resultant weighting coefficients should be consistent across countries. This was checked by examining the correlations of the overall aggregate score with the local equivalents derived in each country.

Results

UK testing

As a result of the UK cognitive interviews, minor adjustments were made to the wording of 12 items and five response options and some of the introductory text was modified to improve clarity and accessibility.

The internal consistency for the aggregate symptom and risk factor scores was good (Cronbach's α >0.70), but internal consistency was lower for the aggregate scores for barriers to symptomatic presentation (0.52) and beliefs about cancer outcomes (0.49). The relatively low α values follow from the relatively low correlations among the constituent questions (range 0.2–0.4). While the communality variance of the constituent questions is lower than ideal for definition of a latent factor, a balance is evident in that each constituent question contains substantial unique information.

In test–retest reliability analyses, the aggregate scores all reached ‘substantial’ (≥60%) agreement between administrations. ICC were also good (total symptoms r=0.70, total risk factors r=0.67, barriers to symptomatic presentation r=0.62, beliefs about cancer outcomes r=0.61).

The majority of individual items had a good agreement between the two administrations, but three fell below 20%. These were one item in the ‘screening’ module (‘breast cancer screening could reduce my chances of dying from breast cancer’, k=0.11), and two items in the ‘risk factor module’ (smoking, k=−0.02; sun bed use, k=−0.04). The breast screening item was tested only in women (N=53), 10 changed their response between test and retest, of whom 7 changed more than two points on the response scale. Clearly this belief is not temporally stable, possibly reflecting the public's confusion about the primary purpose of screening and this is likely to vary between countries. For the risk factor items, the κ-coefficient was affected by the very small numbers who ‘disagreed’ with these items at the first administration (two and seven, respectively) who all ‘agreed’ at the second administration. In fact the overwhelming majority of participants ‘agreed’ at both time points and as these items are well-established risk factors for cancer they were retained.

Content validity assessment showed good (>78%) agreement for all individual items for clarity and relevance. Raters also made a content validity assessment of groups of items representing the different areas (constructs described in ‘item selection’ section). The overall clarity rating for the ‘barriers’ construct was 67%, which the raters’ noted was because of lack of clarity in the introductory sentences.

International testing: conceptual and cultural equivalence

A number of translation issues were highlighted by the cognitive interviews. The verbal interviewer script was shortened and changes made to item ordering to enhance the ‘flow’ of the interview. Additional explanations for some terms (eg, ‘persistent’, ‘HPV’ and ‘processed meat’) were either added or made available to interviewers for respondents who needed further explanation.

In general, the items achieved high ratings of equivalence and relevance in the quantitative assessment of the quality of translation. Table 2 shows the number of items that performed less well. Overall, more problems in the translations were identified by the ‘equivalence’ than the ‘relevance’ test. The common issues included: the translation and coding of the ‘help-seeking intentions’ items, the wording of the item about ‘difficulty getting an appointment with a doctor’, the translation of ‘embarrassment’ as a barrier to seeking medical help and the wording of the screening module items because of the local characteristics of screening programmes. Discussion of the intended meaning of the items took place between the ICBP Working Group and the research company team to enable translators to reword these items to ensure equivalence of meaning. These adjustments were checked during the bilingual cognitive interviews, where a few further minor adjustments were made.

Table 2

Results of the quantitative assessment of equivalence of meaning—number of items with agreement index <78%

Module	Canadian French (N=4 raters)	Danish (N=6 raters)	Swedish (N=8 raters)	Norwegian (N=6 raters)
Core module (32 items)
Equivalence	6	4	6	5
Relevance	5	2	0	3
Risk factors module (13 items)
Equivalence	–	0	0	0
Relevance	–	0	0	0
Screening module (8 items)
Equivalence	–	0	4	2
Relevance	–	2	0	2

Table 3 shows the items that performed less well in terms of either difficulty or discrimination in the international surveys. Very few items were non-discriminatory; the exceptions were awareness of ‘change in the appearance of a mole’ and ‘unexplained lump or swelling’, which were recognised as symptoms of cancer by over 95% of participants in most countries. This indicates that they are well-known symptoms and analyses using them as single-item variables would have limited usefulness; however, they were included on the grounds of face validity. The items with the highest number of ‘invalid’ responses were those asking: “Out of 10 people diagnosed with X cancer, how many do you think would be alive 5 years later.” In particular, the ovarian cancer outcomes item performed poorly in both women and men. Participants in Australia tended to find all of the items about cancer outcomes difficult to answer.

Table 3

Items that performed less well in terms of either difficulty or discrimination

Item (category)*	The UK	Australia	Canada	Denmark	Sweden	Norway
Non-discrimination (>95% in any single category) (%)
Knowledge of unexplained lump or swelling as a symptom of cancer (Yes)	96.6	95.7	–	–	–	–
Q15—change in the appearance of a mole (Yes)	96.7	98.2	95.4	97.0	96.5	98.3
High % ‘invalid’ (>10%)
Q29—cancer treatment is worse than cancer itself (tend to agree)	15.9	–	–	11.1	10.6	–
Q34—how many out of 10 survive bowel cancer (5 people)	–	12.1	–	–	–	–
Q35—how many out of 10 survive breast cancer (8 people)	–	10.3	–	–	–	–
Q36—how many out of 10 survive ovarian cancer (5 people)	14.9	21.1	10.1	–	–	–
Q37—how many out of 10 survive lung cancer (5 people)	–	12.9	–	–	–	–

*The full range of response options for each item can be seen in the online supplementary file containing the UK ABC measure.

ABC, awareness and beliefs about cancer.

Results of analyses exploring the comparability of the factor structure of aggregate scores across countries are shown in tables 4 and 5 which show the percentage of communality accounted for by the first factor and the factor score coefficients for each country. Also shown are the correlations of the overall aggregate score with each country's factor score variable, and the means that were used to generate the overall aggregate scores. The percentage of communality explained by the first factor was consistently at least 40% above the baseline level for the number of factors extracted, which is considered satisfactory. Correlations between the overall measure and the factor scales in each country were >0.99 except for one instance of 0.98. This was considered satisfactory.

Table 4

Development and validation of the aggregate score: barriers to symptomatic presentation

Country	Factor score coefficients				Communality explained (%)*	Correlation with aggregate score
	Q24 ‘too embarrassed’	Q25 ‘time wasting’	Q26 ‘what the dr might find’	Q27 ‘too busy’
Canada	1.1330	0.3925	0.4860	0.3626	41.4	0.989
The UK	0.9129	0.4206	0.3709	0.3561	42.4	0.996
Australia	1.2711	0.4797	0.4356	0.3695	39.8	0.997
Denmark	1.5537	0.7108	0.2863	0.3098	39.7	0.992
Norway	1.3142	0.6716	0.3616	0.3147	39.6	0.999
Sweden	1.0438	0.8786	0.3046	0.3002	34.7	0.984
†Mean	1.2048	0.5935	0.3742	0.3355

*The target is ≥35% (a 40% excess above the baseline of 25%).

†The mean factor score coefficients become the weights for the aggregate score.

Table 5

Development and validation of the aggregate score: negative beliefs about cancer outcomes

Country	Factor score coefficients			Communality explained (%)*	Correlation with aggregate score
	Q28 ‘normal activities’	Q31 ‘cure’	Q33 ‘death sentence’
Canada	0.4319	0.5327	0.2724	50.0	0.991
The UK	0.3776	0.6330	0.2895	50.0	0.998
Australia	0.4185	0.6664	0.2385	50.2	0.999
Denmark	0.3242	0.7554	0.2367	50.2	0.998
Norway	0.3462	0.8330	0.2182	46.1	0.996
Sweden	0.3549	0.7936	0.2453	47.0	0.998
†Mean	0.3756	0.7024	0.2501

*The target is ≥47% (a 40% excess above the baseline of 33%).

†The mean factor score coefficients become the weights for the aggregate score.

Discussion

This paper describes the methods used to develop an international measure of cancer awareness and beliefs administered by telephone interview. Good test–retest reliability and internal and content validity were established in a UK sample prior to translation into other languages. The process of adaptation targeted conceptual and cultural equivalence rather than verbatim translation. To determine where changes to a direct translation were needed to reflect this goal, we used a process comparable to a quantitative content validity exercise with expert raters assessing items in terms of equivalence and relevance. This highlighted items needing revision in order to achieve equivalence of meaning across cultures.

Overall, the ABC measure showed good international validity as assessed by consistency in item performance and the factor structure of aggregate scores. Although the Cronbach's α for the aggregate scores were somewhat low, factor analysis results and stability across countries demonstrated that the latent variable was well defined. A small number of items did not perform well in the reliability or validity tests. Some of these were retained for face validity; in other cases they were retained because of the disproportionate time investment needed to repeat the validation process in all the languages for a small number of items. Improvements could still therefore be made to the measure in the future.

Another limitation was that the selection of the initial items to include in the ABC measure was hampered by lack of preexisting validated measures of cancer beliefs. Although there are validated measures for some areas, for example for breast cancer fear34 and cancer fatalism,35 these did not encompass the variety of beliefs potentially associated with help-seeking behaviour. The field would be enhanced by research into the full range of beliefs about cancer. The two belief constructs validated in the ABC (barriers to symptomatic presentation and beliefs about cancer outcomes) are a start on this process.

The ABC is a reliable and valid measure of cancer awareness and beliefs available in five languages (English, Danish, Swedish, Norwegian and Canadian French), and with demonstrable international validity. The development provides a blueprint for other countries who wish to compare their citizens’ ABC with those of the ICBP partners. This approach to validating international measures, which has evolved from the guidelines on the topic, should also be useful for other international survey research.