A brief measure of Smokers' knowledge of lung cancer screening with low-dose computed tomography.

We describe the development and psychometric properties of a new, brief measure of smokers' knowledge of lung cancer screening with low-dose computed tomography (LDCT). Content experts identified key facts smokers should know in making an informed decision about lung cancer screening. Sample questions were drafted and iteratively refined based on feedback from content experts and cognitive testing with ten smokers. The resulting 16-item knowledge measure was completed by 108 heavy smokers in Houston, Texas, recruited from 12/2014 to 09/2015. Item difficulty, item discrimination, internal consistency and test-retest reliability were assessed. Group differences based upon education levels and smoking history were explored. Several items were dropped due to ceiling effects or overlapping constructs, resulting in a 12-item knowledge measure. Additional items with high item uncertainty were retained because of their importance in informed decision making about lung cancer screening. Internal consistency reliability of the final scale was acceptable (KR-20 = 0.66) and test-retest reliability of the overall scale was 0.84 (intraclass correlation). Knowledge scores differed across education levels (F = 3.36, p = 0.04), while no differences were observed between current and former smokers (F = 1.43, p = 0.24) or among participants who met or did not meet the 30-pack-year screening eligibility criterion (F = 0.57, p = 0.45). The new measure provides a brief, valid and reliable indicator of smokers' knowledge of key concepts central to making an informed decision about lung cancer screening with LDCT, and can be part of a broader assessment of the quality of smokers' decision making about lung cancer screening.

Introduction

Lung cancer is the leading cause of cancer deaths among men and women in the United States (American Cancer Society, 2012). In 2011, the findings from the National Lung Screening Trial showed a 20% reduction in lung cancer deaths among high-risk smokers randomized to the low-dose computed tomography (LDCT) arm compared to those in the chest X-ray arm (Aberle et al., 2011). Shortly thereafter, the United States Preventive Services Task Force (USPSTF) released an updated statement endorsing lung cancer screening with LDCT, and the Centers for Medicare & Medicaid Services (CMS) released the national coverage determination for Medicare coverage of lung cancer screening with LDCT (Moyer, 2014, Centers for Medicare and Medicaid Services, 2015). Thus, there is growing interest in implementing lung cancer screening.

In its landmark beneficiary eligibility determination, CMS requires that a patient counseling and shared decision-making visit with a patient decision aid precedes a preventive service (Moyer, 2014, Centers for Medicare and Medicaid Services, 2015). The requirement for a shared decision-making and patient counseling visit is due in part to recognition of the potential harms associated with lung cancer screening with LDCT, including a high false positive rate, overdiagnosis, complications as the result of invasive diagnostic procedures, and increased risk for new cancers due to increased radiation exposure (Moyer, 2014, Bach et al., 2012, Humphrey et al., 2013). Shared decision making is recommended when the efficacy of an available option, in this case lung cancer screening with LDCT, is not always certain and when there are tradeoffs between benefits and harms associated with the options. CMS further calls for the use of patient decision aids as part of the shared decision-making visit. Patient decision aids can support shared decision making by presenting facts in a balanced manner and encouraging deliberation between patients and health care providers about the tradeoffs (O'Connor et al., 1999). In light of the requirements by CMS for lung cancer screening with LDCT, there is a clear need to have a reliable measure to assess the effectiveness of decision aids on patients' knowledge.

This paper reports the development and psychometric properties of a brief knowledge measure about lung cancer screening with LDCT for use with smokers. This measure development study (ClinicalTrials.gov ID: NCT02282969) was conducted as a precursor to a larger randomized trial evaluating the effectiveness of a video-based patient decision aid (Clinicaltrials.gov ID: NCT02286713).

Materials and methods

Knowledge item generation

In order to generate a pool of knowledge items, we asked tobacco control experts, cancer prevention experts, and a diagnostic radiologist to generate a list of key facts a patient should know in making an informed decision about lung cancer screening with LDCT. We supplemented the expert-generated lists with information about lung cancer screening presented in professional guidelines (Moyer, 2014, Wender et al., 2013, National Comprehensive Cancer Network, 2011), patient and physician fact sheets from professional organizations (American Lung Association, 2012, American Society of Clinical Oncology, 2012, American Cancer Society, 2013, U.S. Preventive Services Task Force, 2013, U.S. Preventive Services Task Force, 2014a, U.S. Preventive Services Task Force, 2014b), the latest evidence synthesis from the USPSTF (Humphrey et al., 2013), and knowledge items from a prior study by the research team (Volk et al., 2014).

The resulting candidate set of 107 fact-based knowledge items (key facts) was broadly grouped into 35 domains (e.g., mortality, incidence, diagnosis, risk factors). The list was reviewed and sorted by the research team, and redundant domains and key facts were collapsed or removed. The resulting 16 domains with 29 key facts were edited for literacy level in advance of cognitive testing and entered into an online survey. Medical content experts (N = 3), from cancer prevention, community oncology practice, and radiology, and research team members rated each item as “essential,” “optional,” or “not necessary” for patients to make an informed decision. From these ratings, 13 domains represented in 16 key facts about lung cancer screening were included in the knowledge measure and evaluated for face validity.

For cognitive testing, participants were recruited from the Tobacco Treatment Program (TTP) at The University of Texas MD Anderson Cancer Center. Eligible participants were current or former smokers 55 to 80 years old with no prior history of lung cancer. After obtaining written consent, research assistants conducted cognitive testing with 10 participants using a “thinking-out-loud” technique, where the participant is asked a series of questions to understand his or her interpretation of what each item is asking and what the response choices mean (Dillman, 1978). We iteratively refined the measure (items and responses) based upon cognitive testing results.

Data collection procedures

Data were collected between December 2014, and September 2015. A multipronged strategy was used to recruit participants, including contacting patients from the TTP at MD Anderson, placing advertisements in local newspapers, and relying on referrals from existing participants in the study. Eligible participants included English-speaking men and women ages 55 to 80 years who were either current smokers or had quit within the past 15 years. Individuals who had been diagnosed with lung cancer were ineligible. These criteria mirrored USPSTF criteria for lung cancer screening eligibility, except pack-year smoking history. Although pack-year smoking history was not an eligibility criterion for our study, we did assess pack-year smoking history and compared results for participants who did or did not have a minimum of a 30 pack-year smoking history.

After consenting to the study, participants completed a baseline questionnaire, and again completed a follow-up questionnaire one month later. The one month time lag was selected, instead of the standard 10 days to 2 weeks interval, to minimize priming effects on the participants' knowledge of lung cancer screening. The order of the knowledge questions was randomized at the follow-up. Participants also completed demographic questions at baseline. This study was approved by MD Anderson's Institutional Review Board.

Analysis

We tested the psychometric properties of the knowledge measure, including item difficulty, item discrimination, reliability, and validity. Not all subjects completed the follow-up questionnaire. Sensitivity analysis of data from the participants who only completed the baseline compared to participants who had completed baseline and follow-up revealed that the item characteristics did not differ significantly. Data analysis was conducted with SPSS.

Item difficulty and uncertainty

Item difficulty was determined by examining the proportion of correct responses. Generally, items with 50% correct responses tend to increase the reliability of a multi-item measure (Crocker & Algina, 1986), although highly difficult items may be retained in a measure if they assess constructs potentially amenable to educational interventions. Item uncertainty was defined as the proportion of “I don't know” responses.

Index of discrimination

The index of discrimination was used to assess the validity of the items in discriminating among high and low performers on the scale. The index of discrimination is the difference between the proportion of correct responses of participants in the upper and lower 25th percentiles. Traditionally, items with an index of discrimination of 40% are considered acceptable, and those with an index less than 20% are considered inadequate (Ebel, 1965).

Reliability

We assessed the reliability of the overall knowledge measure using the Kuder-Richardson 20 (KR-20) formula, an indicator of internal consistency reliability for scales with dichotomous responses (i.e., correct or incorrect responses) (Cronbach, 1951, Kuder and Richardson, 1937). Characteristics of the individual items were examined using item-total correlations, and internal consistency of the overall scale if the item was deleted (alpha if item deleted). Test-retest reliability of the individual items was calculated with kappa coefficients, and test-retest reliability of the overall knowledge measure was estimated by the intraclass correlation.

Differences between groups

One-way analysis of variance was used to explore group differences. It was expected that knowledge would differ between education levels, which would reflect concurrent validity, a type of criterion validity. It is expected that those with higher education (graduated high school/GED or less, some college/trade school, graduated college or more) would score better on any given knowledge measure. Additionally, analysis was conducted to explore whether current smokers and participants who met eligibility criteria for lung cancer screening had different knowledge scores.

Item selection

We considered the performance of the scale items across a number of indicators in making decisions about which items to retain or delete. Decisions to retain or delete an item were not based on a single indicator. Instead, we sought to retain items that had a potential to 1) be responsive to an intervention, 2) discriminate between high and low performers, and 3) represented a necessary fact to make an informed decision. For example, a difficulty item may be kept if the level of uncertainty was moderate to high, indicating that the item has the potential to be responsive to an intervention. Generally, items with a ceiling effect would be deleted, unless they represented an essential fact and there was not an alternative item that could replace it in the knowledge measure.

Results

Subject characteristics

The participants' ages ranged from 55 to 77 years, and slightly more than half were female (Table 1). The majority of participants were White, not Hispanic, and had more than a high school degree. There were no statistically significant differences between those who did and did not complete the follow-up questionnaire, one month after baseline.

Table 1

Characteristics of the subjects.

	Baseline (n, %)N = 120	1 month (n, %)N = 108	p-valuea
Age (mean, sd)	63.5 (5.9)	63.59 (6.0)	0.94
Gender			0.28
Male	57 (47.5)	50 (46.3)
Female	63 (52.5)	58 (53.7)
Race			0.91
Asian/Pacific Islander	3 (2.5)	3 (2.8)
Black or African-American	29 (24.2)	26 (24.1)
White	85 (70.8)	76 (70.4)
Not specified	2 (1.7)	2 (1.9)
Refused to answer	1 (0.8)	1 (0.9)
Ethnicity			0.64
Hispanic	6 (5.0)	5 (4.6)
Not Hispanic	113 (94.2)	102 (94.4)
Education			0.22
Less than high school	3 (2.5)	3 (2.8)
Graduated high school/GED	17 (16.7)	17 (15.7)
Some college/trade school	41 (40.8)	42 (38.9)
Graduated college or more	46 (40.0)	46 (42.6)
Current smoker	61 (50.8)	53 (49.1)	0.38
Years smoked cigarettes (mean, sd)	40.3 (16.8)	37.2 (12.0)	0.08
Avg. cigarettes smoked per day (mean, sd)	18.09 (9.3)	18.37 (9.4)	0.62
Pack-year history (mean, sd)	36.5 (23.3)	36.8 (23.8)	0.57

Abbreviations: sd = standard deviation, GED = general educational development, Avg. = average

All data were collected from 12/2014 to 09/2015 in Houston, TX.

The p-values are based upon one-way ANOVA for continuous variables and chi-square tests comparing those who completed baseline and follow-up and those who did not.

Item characteristics

Item characteristics of the initial 16-item measure are given in Table 2. The item difficulty ranged from 1.7% to 88.3% correct, and item uncertainty ranged from 2.5% to 66.7%. The two items assessing knowledge about the association between smoking and lung cancer (Items # 2 and #3) had very high proportions of correct responses. Interestingly, both items had low levels of uncertainty (10.8% and 2.5%), suggesting that participants were confident in their responses. These items were not included in the final measure. We incorporated two items on the predictive value of screening (Items #7 and #8). Although only a few participants answered Item #7 correctly (2.5%), we felt that it would be more responsive to an intervention than Item #8. For the final measure, Item #7 was retained and Item #8 was dropped.

Table 2

Knowledge Items, Domains, and Item Characteristics for the Lung Cancer Screening Knowledge Measure.

Item	Domain	Item difficultyb	Item uncertaintyc	Index of discriminationd	Item-total correlatione	α if item deletedf	Test-retest reliabilityg
1. What percentage of lung cancer deaths are caused by smoking?	Attributable risk, smoking	20.8%	50.8%	23.4%	0.12	0.70	0.29
2. Is smoking the leading cause of lung cancer?a	Risk factors, smoking	84.2%	10.8%	40.0%	0.23	0.69	0.33
3. The most important thing smokers can do to lower their risk of lung cancer is to…a	Risk reduction, smoking	88.3%	2.5%	7.4%	-0.01	0.71	0.13
4. Where does lung cancer rank as a cause of cancer death in the US?	Risk, cause of death	25.8%	45.0%	53.1%	0.30	0.68	0.41
5. When should someone stop being screened for lung cancer? (Check all that apply)	Screening eligibility	47.5%	41.7%	52.6%	0.25	0.69	0.44
6. Should all current and former smokers be screened for lung cancer?a	Screening eligibility	2.5%	5.8%	2.9%	0.10	0.69	0.66
7. How many people with an abnormal CT scan will have lung cancer?	Predictive value of screening	2.5%	66.7%	2.9%	0.07	0.70	0.38
8. Does an abnormal CT scan mean a person has lung cancer?a	Predictive value of screening	63.3%	35.0%	75.4%	0.49	0.65	0.41
9. Can a CT scan suggest that you have lung cancer when you do not?	Screening accuracy, false positives	37.5%	50.8%	67.4%	0.40	0.66	0.36
10. Can a CT scan miss a tumor in your lungs?	Screening accuracy, false negatives	52.5%	37.5%	68.0%	0.38	0.67	0.49
11. Will all tumors found in the lungs grow to be life threatening?	Natural history, “over-diagnosis”	57.5%	30.0%	83.4%	0.52	0.65	0.45
12. Without screening, is lung cancer often found at a later stage when cure is less likely?	Screening benefit, mortality	72.5%	24.2%	52.6%	0.32	0.68	0.22
13. How much does screening for lung cancer with a CT scan lower your chances of dying from lung cancer?	Screening benefit, mortality reduction	1.7%	59.2%	-1.1%	0.00	0.70	0.20
14. Can a CT scan find lung disease that is not cancer?	Screening, secondary findings	70.0%	29.2%	77.1%	0.52	0.65	0.57
15. Can a CT scan find heart disease?	Screening, secondary findings	44.2%	50.0%	64.0%	0.29	0.68	0.28
16. Is radiation exposure one of the harms of lung cancer screening?	Screening risks	29.2%	42.5%	56.0%	0.33	0.67	0.39

Bolded items were included in the final lung cancer screening knowledge measure.

Abbreviations: CT = Computed tomography, α = alpha. All data were collected from 12/2014 to 09/2015 in Houston, TX.

Items were dropped from final lung cancer screening knowledge measure.

Item difficulty is the percentage correct responses.

Item uncertainty is the percentage of “I don't know” responses.

The index of discrimination is the difference between percentage of respondents with correct responses for an item among those who scored in the upper versus lower quartiles of correct responses to the full scale. Items with an index of discrimination greater than 40% are considered acceptable for discrimination, and those with an index less than 20% are considered inadequate.

Item-Total Correlation is the correlation between the response on an individual question and the overall score on the survey.

α if item deleted represents the internal consistency of the scale without that item.

Kappa coefficients are reported for baseline and 1-month, and represent average intercorrelation of items with dichotomous responses.

Several items were deemed very difficult, with only a small proportion of participants responding correctly. Only a few participants chose all three correct responses for the multi-choice screening eligibility question, “When should someone stop being screened for lung cancer?” (Item #5). We therefore considered this item as being “correct” if the subject endorsed any of the correct responses without also selecting “I don't know.” With this modification, the proportion of correct responses increased from 4.2% to 47.6%, and Item #5 with the alternative scoring system was retained. Item #6 (“Should all current and former smokers be screened for lung cancer?”) appeared to reflect more of a personal belief given that very few people were uncertain about their response (5.8%), despite 91.7% being incorrect. Therefore, Item #6 was dropped from the final measure. Knowledge of the magnitude of the mortality benefit of lung cancer screening with LDCT (Item #13) was poor. However, we retained this item because it was viewed as essential by the content experts and should be amendable to educational interventions.

Examination of the discrimination between lower and higher performers generally supported the findings for item difficulty and uncertainty. The index of discrimination for Items #7 (predictive value of screening) and #13 (mortality benefit of screening) was low, but because of high item uncertainty and importance assigned by the content experts these items were retained.

Reliability

The alpha if item deleted remained fairly stable ranging from 0.65 to 0.71, and suggested that none of the items detracted from the overall scale's reliability. The test-retest reliability estimates ranged from 0.13 to 0.66. For the overall scale, the KR-20 was 0.69, suggesting acceptable internal consistency.

After dropping the poor performing or redundant items, the resulting 12-item knowledge measure (Table 3) had a similar internal consistency as the 16-item knowledge measure (KR-20 = 0.66) and a high test-retest reliability (ICC = 0.84). The mean score at baseline was 4.59 with a standard deviation of 2.38 (using the modified scoring for Item #5). The mean knowledge score was statistically different across education levels (high school or less: mean ± standard deviation = 3.57 ± 1.80; some college: 4.58 ± 2.41; college or more 5.10 ± 2.48; F = 3.36, p = 0.04). Knowledge scores did not differ between current and former smokers (current smoker: 4.31 ± 2.53; former smoker: 4.91 ± 2.18; F = 1.43, p = 0.24) nor participants who met or did not meet the 30 pack-year lung cancer screening eligibility criterion (met criterion: 4.75 ± 2.33; did not meet criterion: 4.41 ± 2.42; F = 0.57, p = 0.45).

Table 3

Final 12-item lung cancer screening knowledge measure.

Discussion

We developed a brief, self-report measure of smokers' understanding of key concepts central to making an informed decision about lung cancer screening with LDCT. A careful development process was used to identify key facts smokers should understand in making an informed screening decision, test the items for clarity and face validity, and select a final set of items with acceptable item characteristics and reliability. The process was multidisciplinary and patients/smokers played a significant role in refining the measure.

Sepucha and colleagues (Sepucha et al., 2014, Sepucha et al., 2013, Sepucha et al., 2011, Sepucha et al., 2008, Sepucha et al., 2007) have argued that high quality decisions are those where the patient understands the options and associated harms and benefits, and makes decisions consistent with one's values. The brief knowledge measure developed in this study can provide an indicator of patients' comprehension of these key facts, and be part of a broader assessment of decision-making quality about lung cancer screening.

It is noteworthy that participants in the study had difficulty with the two questions about eligibility criteria for screening. One question asked whether all current and former smokers should be screened, and over 90% of participants answered affirmatively. This may, in part, represent patients' enthusiasm for cancer early detection in general (Schwartz et al., 2004) and wanting all smokers to be screened for lung cancer regardless of their smoking behavior. This item may have captured the concept more accurately if it was framed as “According to the guidelines, should all current and former smokers be screened for lung cancer?” The question assessing criteria for discontinuing lung cancer screening was challenging as well, which may be due in part to the “check all that apply” response format. Yet, this concept is important because of concerns about “eligibility slippage” where patients not meeting all requirements for lung cancer screening may receive services (Centers for Medicare & Medicaid Services, 2014), and we therefore retained this item in the final measure. Additional item refinement may be needed to better assess patients' understanding of exclusion criteria for screening.

Ceiling effects were observed for two items related to knowledge about the role of smoking in lung cancer. This finding is not surprising given long-standing public awareness campaigns focused on reducing lung cancer deaths through tobacco cessation. Two additional items showed high difficulty and uncertainty. The predictive value of an abnormal CT scan was poorly understood. Many participants felt that an abnormal finding was likely lung cancer, and few participants knew the magnitude of the benefit of lung cancer screening in reducing lung cancer deaths. We argue that these items represent central facts required in making an informed screening decision and should be retained. They may also be highly amenable to educational interventions targeted to patients, where large gains in knowledge might be expected.

This study has several limitations. Overall, the sample included a majority of participants with at least some college education, while national statistics suggest a higher smoking prevalence among people with a high school education or less (Centers for Disease Control and Prevention, 2010, Centers for Disease Control and Prevention, 2004). The responsiveness of the measure to educational interventions was not assessed in this study and will be important to demonstrate in future research. We included in the sample some individuals who would not meet screening eligibility criteria based on their smoking history. However, the knowledge scores did not differ for smokers meeting the 30 pack-year smoking threshold compared to those who did not. The “select all that apply” response structure for the item on when to stop screening may have been too difficult. Further testing would be needed to determine the best way to assess the screening eligibility construct/domain. However, we did propose an alternative scoring method which improved the psychometric properties of this item.

The brief knowledge measure developed in this study can be used to characterize the degree of awareness of facts about lung cancer and lung cancer screening in specific populations, set a baseline for intervention studies, and serve as an outcome measure for educational interventions. Future research might consider identifying an abbreviated set of core indicators that can be used in the clinical setting to identify participants at greatest need for educational interventions and decision support about lung cancer screening.

Disclaimer

All statements in this article, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Authors' contributions

Conception and design: Volk, Leal, Linder.

Development of methodology: Volk, Leal, Linder, Cofta-Woerpel.

Identification of key messages for item development: Bevers, Escoto, Fisch, Godoy, Munden, Cantor.

Acquisition of data: (collection: Richards, Hempstead; access to patients: Godoy, Cinciripini).

Analysis and interpretation of data: Lowenstein, Richards, Housten, Volk.

Writing, revising, or reviewing the manuscript: All authors.

Study supervision: Volk.

Conflict of interest

The authors declare there is no conflict of interest.