My Cancer Genome: Evaluating an Educational Model to Introduce Patients and Caregivers to Precision Medicine Information.

This study tested an innovative model for creating consumer-level content about precision medicine based on health literacy and learning style principles. "Knowledge pearl" videos, incorporating multiple learning modalities, were created to explain genetic and cancer medicine concepts. Cancer patients and caregivers (n=117) were randomized to view professional-level content directly from the My Cancer Genome (MCG) website (Group A; control), content from MCG with knowledge pearls embedded (Group B), or a consumer translation, targeted at the sixth grade level, with knowledge pearls embedded (Group C). A multivariate analysis showed that Group C, but not Group B, showed greater knowledge gains immediately after viewing the educational material than Group A. Statistically significant group differences in test performance were no longer observed three weeks later. These findings suggest that adherence to health literacy and learning style principles facilitates comprehension of precision medicine concepts and that ongoing review of the educational information is necessary.

RCT Entities:   Population Interventions Outcomes

Introduction

New initiatives for precision medicine come with greater opportunities for patients to receive truly individualized care. Oncology is at the forefront of precision medicine[1], as new molecular targeted therapies and immunotherapies are increasingly used to treat cancer patients, including patients with melanoma[2], lung cancer[3], and renal cell carcinoma[4]. However, for the goals of precision cancer medicine to fully be realized, more genomic and clinical trial data must be collected, and the resulting knowledge must be properly communicated[1]. Patients will also increasingly need to be able to understand genetic testing results to fully comprehend their treatment options.

One barrier to the implementation of precision cancer medicine is a lack of understanding about basic genetic concepts[5]. Adults often have difficulty understanding genes, chromosomes, and other aspects of genetics[6,7]. National K-12 genetics standards do not currently cover important genetic concepts, such as mutations or gene expression and regulation, and interpretation of these standards may lead to wide variation in how and to what extent educators teach the included concepts[8]. Genetic knowledge deficits can be particularly detrimental to patients’ ability to understand and give informed consent for genetic testing[9,10] or genomic sequencing[11].

In light of these widespread deficits, easy-to-understand resources about precision medicine are needed. My Cancer Genome (MCG; mycancergenome.org) is a premier online informatics resource for precision cancer medicine knowledge providing information on therapies for specific tumor mutations[12]. Although it is written at the health professional-level, patients and caregivers still access the website. Other online resources available to patients include the National Library of Medicine’s Genetics Home Reference[13], Massachusetts General’s Targeted Cancer Care website[14], and 23andMe’s “Genetics 101” videos[15]. Approximately 36% of American adults have basic or inadequate health literacy. A lack of understanding about basic genetic concepts and low health literacy, defined as “the ability to use literacy skills to read and understand written health-related information encountered in everyday life”[16], may make these resources difficult to understand[17].

We conducted a set of studies to evaluate a model, incorporating health literacy and learning style principles, for delivering information about precision cancer medicine therapies to patients and caregivers. In the first study, conducted in fall of 201418 we developed a consumer translation of information from My Cancer Genome regarding the BRAF V600E mutation in melanoma. The content was adapted to the sixth grade reading level and designed to incorporate preferences for learning via several learning style modalities, including preferences for learning via reading, listening, and watching. The material contained hyperlinks to videos, called knowledge pearls, that were developed to provide easy-to-understand explanations of genetic and cancer medicine concepts. Using a randomized, controlled study design, we found that the consumer version of the information was more effective in educating melanoma patients about the BRAF V600E mutation in melanoma than the My Cancer Genome website version with or without the knowledge pearls embedded. Learning was assessed by administering a knowledge questionnaire before and immediately after viewing the educational content.

In this study, building on the above-mentioned findings, we tested the generalizability of the model for delivering consumer-friendly information about precision medicine. We also assessed whether the changes in knowledge were long-lasting. Content from MCG about targeted therapies and immunotherapies was used for the evaluation because of these topics’ broad applicability to multiple cancer types. The information was translated into a consumer-friendly format and hyperlinks to knowledge pearl videos were added to both the consumer and professional-level versions. A randomized, controlled study design with cancer patients and caregivers was used to evaluate differences in educational outcomes. We hypothesized that participants who received the consumer-friendly content with knowledge pearls would show greater improvements in knowledge test scores than participants who received the professional-level information with or without links to the knowledge pearls.

Methods

The study protocol was approved by the Vanderbilt University Institutional Review Board and the Vanderbilt-Ingram Cancer Center Scientific Review Committee.

Setting

This study was conducted at the Vanderbilt-Ingram Cancer Center (VICC). A member of the National Comprehensive Cancer Network and designated by the National Cancer Institute as a Comprehensive Cancer Center, the VICC is on the cutting edge of new cancer therapies. The VICC is ranked in the top 10 nationally for grant support of cancer research and regularly has access to over 150 clinical trials[19]. Members of the VICC team developed the My Cancer Genome website and collaborated in the creation of the consumer-level content.

Participants

Patients and caregivers from the melanoma, lung cancer, and renal cancer clinics at the Vanderbilt-Ingram Cancer Center were recruited from April 16 - May 15, 2015. To be eligible, participants had to be 18 or older, be able to read and speak English, and either be a patient with melanoma, lung cancer, or renal cancer or be the caregiver for a participating patient. Exclusion criteria were imprisonment or cognitive impairment as determined by the clinical provider.

Study Design

After obtaining informed consent, participants were randomized equally to one of three groups (Figure 1): Group A (control) received information on targeted therapies and immunotherapy in cancer taken directly from My Cancer Genome (http://www.mc.vanderbilt.edu/km/gl/pm/pmc-mcg.html); Group B received the same information from MCG embedded with hyperlinks to the knowledge pearl videos (http://www.mc.vanderbilt.edu/km/gl/pm/pmc-mcg-p.html); Group C received a consumer-level version of the information from MCG, including hyperlinks to the knowledge pearls (http://www. mc.vanderbilt.edu/km/gl/pm/pmc -c. html).

Figure 1.

Overview of Study Design

Randomization was carried out using stratified permuted-block randomization with a block size of 3. Stratification factors included “time since diagnosis disease” (within 12 months, 13 to 24 months, or over 24 months) and “type of study participant” (patient or caregiver). Treatment assignments were implemented using the randomization module in REDCap (Research Electronic Data Capture), a secure, web-based research data management tool developed at Vanderbilt University[20].

Participants were asked to provide information about age, race, gender, education level, occupation status, participant status (cancer patient or caregiver), and contact information. They were also asked to complete a subjective health literacy assessment, consisting of 3 questions[21] and to answer whether they would recall how to do something a year from now if they learned it by reading, listening, watching, and/or doing. Additionally, information about time since diagnosis, cancer stage, whether or not tumor profiling had been done, length of time as a Vanderbilt patient, and any current precision medicine therapies was determined via medical record review in StarPanel, VUMC’s electronic medical record system[22].

To assess changes in knowledge, we developed a 10-item knowledge questionnaire, which included multiple choice and true/false questions about precision medicine therapies (http://tiny.mc.vanderbilt.edu/questionnaire). The information needed to answer questionnaire items was addressed in all three versions of the educational material. Participants were initially asked to complete the questionnaire prior to viewing the educational material. They were then given the educational materials to view on an iPad. Web analytics data were collected during this session to assess intervention fidelity. Immediately after participants viewed the educational information, we re-administered the knowledge questionnaire.

After completing the post-test, participants were asked to rate their level of agreement, with answer choices strongly disagree, disagree, neither agree or disagree, agree, or strongly agree, with the following statements: the information I received was easy to understand; some of the information I received was confusing; I am satisfied with the information I received; the information I received taught me something new. Participants in Groups B and C were additionally asked to rate their agreement about whether the videos (“knowledge pearls”) helped them understand the information they received, were easy to understand, or were confusing.

Participants were given a print copy of the materials they viewed to take home. Each handout also included a link to the online version of the content they viewed to allow participants to review the material after the clinic visit.

To assess participants’ retention of any knowledge gains over time, we re-administered the knowledge questionnaire during a follow-up phone call about three weeks later. During the follow-up call, participants were also asked if they read the handout they were given or visited the website address provided on the handout since leaving the clinic. Finally, they were given an opportunity to share any additional thoughts about the information they received. The phone calls were conducted from May 7 - June 11, 2015. Participants who completed the study received a $25 gift card.

Development of Intervention Materials

Information from MCG, which was written at the college level, regarding the use of precision medicine in cancer was translated into a consumer-friendly format using an iterative process developed and refined in previous research phases[18,23]. Briefly, the consumer-level content was aimed at the sixth grade reading level and informed by health literacy best practices. The scripts for the knowledge pearl videos were also targeted at the sixth grade reading level. The MCG knowledge pearl videos were each approximately one to two minutes long and incorporated multiple learning styles through the use of video, images, text, and narration. In creating the knowledge pearls, we chose to focus on fundamental genetic or cancer medicine concepts so that the pearls could be used in multiple applications and would not need as frequent updating as the consumer-translated text. The development of the knowledge pearls was informed by focus group studies, which included cancer patients and caregivers[18,23]. The materials reflect the diverse health literacy[24] and content knowledge expertise of librarians, information scientists, oncologists, and members of the My Cancer Genome team.

Sample Size Determination

The primary objective of the study was to evaluate the educational effectiveness of consumer information about precision medicine therapies in cancer, which was assessed by comparing pre-, post-and follow-up knowledge test scores among the three intervention groups. With a two-sided significance level of 5% and 80% power, the study required a sample size of 27 individuals per study arm. To adjust for a 30% drop-out rate, the planned recruitment goal was 117 participants. The sample size calculation provided sufficient power to detect a 0.91x standard deviation (SD) difference between groups and was based on the assumption that the effect size is equal to the mean difference between groups divided by the standard deviation.

Statistical Analysis

All missing values were imputed using the multiple imputation approach with all relevant study variables: baseline score, post score, follow up score, group, cancer type, months since diagnosis, gender, education, age, and health literacy score. Multivariable linear regression was used to estimate the intervention effect on (1) post-knowledge test score and (2) follow-up knowledge test score, adjusted for pre-knowledge test score (baseline assessment), as well as other covariates (i.e. age, education and health literacy). Residual analysis was used to check the linear regression assumptions of homogeneity for variance, normality and linearity. The final model was decided based on likelihood ratio tests for comparing three models (with and without education or health literacy) as well as adjusted R[2]. All tests were significant at the two-sided 5% level. Analyses were performed in R 3.1.0, and the study statisticians were blinded to the group assignment.

Results

One-hundred and seventeen participants were randomized in a 1:1:1 ratio to one of the three groups (Figure 2). Eight participants withdrew during the clinic visit, and data for two participants were excluded from analysis as they did not meet eligibility criteria. Pre-and post-knowledge questionnaire data were analyzed for the remaining 107 participants (n = 34, 37, and 36 for Group A, B, and C, respectively). Thirteen participants were lost to follow-up.

Figure 2.

Participant flow diagram

Most of the participants were white and had at least a high school education (Table 1). A higher percentage of participants were cancer patients in Group A (65%) than in Groups B and C, (59% and 58%, respectively). A higher percentage of participants in Group B (52%) reported a household income greater than $75,000 than in Group A (23%) or Group C (35%). The percentage of participants with adequate health literacy in the three groups was 82% in Group A, 89% in Group B, and 83% in Group C (Table 2). Nearly half the participants in each group indicated preferences for learning via multiple modalities (Table 2).

Table 1.

Participant characteristics

Characteristic	N	Group A (n = 34)	Group B (n = 37)	Group C (n = 36)
Participant type, n (%)	107
Patient		22 (65)	22 (59)	21 (58)
Caregiver		12 (35)	15 (41)	15 (42)
Cancer type, n (%)	107
Lung		14 (41)	11 (30)	15 (42)
Melanoma		16 (47)	16 (43)	16 (44)
Renal		4 (12)	10 (27)	5 (14)
Diagnosis range, n (%)	107
Within 12 months		11 (32)	10 (27)	11 (31)
13-24 months		3 (8.8)	5 (14)	5 (14)
Over 24 months		20 (59)	22 (59)	20 (56)
Cancer stage, n (%)
Stage I	107	1 (2.9)	0 (0)	0 (0)
Stage II		0 (0)	3 (8.1)	2 (5.6)
Stage III		4 (12)	5 (14)	5 (14)
Stage IV		17 (50)	24 (65)	19 (53)
Not available		7 (21)	4 (11)	8 (22)
Other		5 (15)	1 (2.7)	2 (5.6)
Tumor sequencing conducted, n (%)	107
Yes		21 (62)	17 (46)	17 (47)
No/data was not available		13 (38)	20 (54)	19 (53)
Received precision medicine therapy, n (%)	107
Yes		16 (47)	14 (38)	17 (47)
No/data was not available		18 (53)	23 (62)	19 (53)
Days since first seen at Vanderbilt, median (Q1, Q3)	106	714 (255, 1662)	1288 (337, 3322)	1339 (237, 2823)
Age in years, mean (SD)	105	57 (15)	55 (15)	58 (9.7)
Gender, n (%)	106
Male		15 (44)	17 (47)	14 (39)
Female		19 (56)	19 (53)	22 (61)
Race, n (%)	104
White		33 (97)	35 (100)	33 (94)
Black or African American		1 (2.9)	0 (0)	0 (0)
American Indian or Alaska Native		0 (0)	0 (0)	2 (5.7)
Hispanic, Latino, or Spanish origin, n (%)	104
Yes		0 (0)	1 (2.9)	1 (2.8)
No		33 (100)	34 (97)	35 (97)
Education, n (%)	107
Between 9th and 12th grade		1 (2.9)	1 (2.7)	3 (8.3)
High school graduate, GED or equivalent		4 (12)	6 (16)	8 (22)
Some college - no degree		8 (24)	4 (11)	4 (11)
Vocational/technical degree		6 (18)	2 (5.4)	7 (19)
Bachelor’s degree		10 (29)	12 (32)	6 (17)
Master’s degree		4 (12)	9 (24)	7 (19)
Professional school degree		0 (0)	2 (5.4)	1 (2.8)
Doctoral degree		1 (2.9)	1 (2.7)	0 (0)
Household income, n (%)	89
$10,000 or less		1 (3.3)	4 (12)	3 (12)
$10,001 - $20,000		4 (13)	2 (6.1)	3 (12)
$20,001 - $35,000		6 (20)	2 (6.1)	4 (15)
$35,001 - $55,000		9 (30)	4 (12)	3 (12)
$55,001 - $75,000		3 (10)	4 (12)	4 (15)
more than $75,000		7 (23)	17 (52)	9 (35)

Table 2.

Health literacy level and learning style preferences

Characteristic	N	Group A (n = 34)	Group B (n = 37)	Group C (n = 36)
Health literacy level, n (%)	106
Adequate		28 (82)	33 (89)	29 (83)
Marginal		2 (5.9)	3 (8.1)	4 (11)
Inadequate		4 (12)	1 (2.7)	2 (5.7)
Learning style, n (%) a	106
Reading		1 (2.9)	6 (17)	1 (2.8)
Listening		2 (5.9)	0 (0)	4 (11)
Watching		2 (5.9)	4 (11)	1 (2.8)
Doing		13 (38)	8 (22)	14 (39)
Multimodal		16 (47)	18 (50)	16 (44)

Note:

Participants who selected more than one learning style were classified as multimodal learners.

The mean post-and follow-up test scores were higher for all groups compared to the pre-test scores (Table 3). A multivariable linear regression revealed that Group C showed a significantly greater improvement in post-test score (p = 0.0308) compared to Group A (Table 4). There was no significant difference detected between Groups A and B in post-test score (Table 4). Group C showed a greater improvement in three week follow-up test score compared to Group A; however, the multivariable linear regression showed that the group difference was not statistically significant at the two-sided 5% level (Table 5). There was also no significant difference detected between Group B and Group A in follow-up test score (Table 5).

Table 3.

Mean scores on the knowledge assessment questionnaire

Mean number of correct answers (SD)	N	Group A (n = 34)	Group B (n = 37)	Group C (n = 36)
Pre-assessment score, mean (SD)	107	4 (2.5)	5.4 (2.0)	4.2 (2.3)
Post-assessment score, mean (SD)	107	6.2 (1.7)	6.5 (1.7)	7.1 (1.8)
Follow-up score, mean (SD)	94	5.7 (1.8)	6.8 (1.6)	6.7 (1.6)
Post-test minus pre-test score, mean (SD)	107	2.2 (1.9)	1.1 (2.0)	2.9 (2.3)
Follow-up minus post-test score, mean (SD)	94	1.7 (2.0)	1.2 (2.0)	2.1 (2.1)

Abbreviations: SD = standard deviation

Table 4.

Multivariate linear regression of post-test score

	Posttest score (N = 107)
Variable	B	(SE)	t	p (two-tailed)
Intercept	4.78	(0.93)	5.15	<0.0001
Pretest score	0.34	(0.069)	4.85	<0.0001
Group B	−0.29	(0.38)	−0.75	0.453
Group C	0.83	(0.38)	2.19	0.0308
Age	−0.0053	(0.011)	−0.47	0.639
College-level education	0.53	(0.42)	1.25	0.215
Master or Doctoral degree	0.82	(0.50)	1.66	0.0998
Health literacy score	−0.03	(0.068)	−0.44	0.662

Abbreviations: SE = standard error

Table 5.

Multivariate linear regression of follow-up test score

	Follow-up test score (N = 107)
Variable	B	(SE)	t	p (two-tailed)
Intercept	4.59	(0.94)	4.9	<0.0001
Pretest score	0.36	(0.072)	5.01	<0.0001
Group B	0.34	(0.41)	0.83	0.411
Group C	0.65	(0.38)	1.72	0.0881
Age	−0.0074	(0.011)	−0.66	0.512
College-level education	0.083	(0.45)	0.18	0.856
Master or Doctoral degree	0.48	(0.53)	0.9	0.372
Health literacy score	0.001	(0.080)	0.01	0.99

There were significant group differences (Figure 3), as assessed by Fisher’s exact tests, in whether the participants thought the information was easy to understand (p = 0.01), was confusing (p = 0.014), and if they were satisfied with the information (p = 0.03). The percentage of participants who either agreed or strongly agreed that the information was easy to understand was highest in Group C (77.3%) compared to Group B (54.1%) and Group A (31%). The percentage of participants who agreed or strongly agreed that they were satisfied with the information was also highest in Group C (75%) compared to Group B (70.1%) and Group A (41%). There were no statistically significant group differences in participant responses regarding whether they thought that they learned something from the information (p = 0.063). There were also no statistically significant differences between Groups B and C regarding responses to whether the videos were useful (p = 0.052), easy to understand (p = 0.44), or confusing (p = 0.215). The percentage of participants in Group B who agreed or strongly agreed that the videos were useful, easy to understand, and confusing were 96%, 87%, and 0%, respectively. For Group C, the percentage of participants who agreed or strongly agreed that the videos were useful, easy to understand, and confusing were 89%, 96%, and 7.8%, respectively.

Figure 3.

Graphical representation of the percentage of participants who strongly disagreed, disagreed, neither agreed or disagreed, agreed, or strongly agreed to statements about whether the information was easy to understand, if they were satisfied with the information, if some of the information was confusing, and if they learned something (A) and whether the knowledge pearl videos helped them understand the information they received (labeled “useful” in the figure), were easy to understand, or were confusing (B). Note that the legend in A also applies to B.

Web analytics data revealed that 78% of participants in Group B and 86% of participants in Group C viewed at least one of the knowledge pearl videos. The knowledge pearls viewed by at least half of participants were antibody, protein, and targeted therapy for Group B and genetic testing and targeted therapy for Group C (Table 6). The mean duration participants spent viewing the intervention materials was 12.1 (SD = 7.03 minutes; n = 30), 21.6 (SD = 17.6 minutes; n = 34), and 17.3 (SD = 35.3 minutes; n = 36) minutes, for Groups A, B, and C, respectively.

Table 6.

Knowledge pearl views

	Group B (n = 36)	Group C (n = 36)
Pearl, n (%)
Antibody	22 (61)	6 (17)
Chromosome	2 (5.6)	N/A
Expression	10 (28)	N/A
Fusion	4 (11)	N/A
Genetic testing	N/A	23 (64)
Immune system	12 (33)	11 (31)
Exon	4 (11)	N/A
Kinase inhibitor	12 (33)	N/A
Mutation	4 (11)	N/A
Protein	19 (53)	5 (14)
Receptor	16 (44)	N/A
Targeted therapy	24 (67)	23 (64)
Wildtype	2 (5.6)	N/A

The median follow-up phone call interval was 22 days for all three groups (Table 7). At follow-up, 32 participants (94%) in Group A, 25 participants (86%) in Group B, and 28 participants (90%) in Group C reported that they kept the educational information handout they were given during their clinic visit. Fifteen participants (44%) in Group A, 7 participants (24%) in Group B, and 12 participants (39%) in Group C indicated that they read the handout since leaving the clinic. Only one participant reported viewing the website version of the material after leaving the clinic.

Table 7.

Follow-up interview

Characteristic	N	Group A (n = 34)	Group B (n = 29)	Group C (n = 31)
Follow-up interval in days, median (Q1, Q3)	94	22 (21, 24)	22 (21, 23.5)	22 (21, 24)
Kept handout, n (%)	94
Yes		32 (94)	25 (86)	28 (90)
No		2 (5.9)	3 (10)	3 (9.7)
Don’t know		0 (0)	1 (3.5)	0 (0)
Read handout, n (%)	94
Yes		15 (44)	7 (24)	12 (39)
No		19 (56)	22 (76)	18 (58)
Don’t know		0 (0)	0 (0)	1 (3.2)
Viewed website, n (%)	94
Yes		0 (0)	1 (3.5)	0 (0)
No		34 (100)	28 (97)	31 (100)

Discussion

In this study, we found that participants who viewed the consumer version of the information about precision medicine with the knowledge pearls embedded (Group C) showed significantly greater knowledge gains immediately after viewing the information than the control group (Group A). Participants who viewed knowledge pearls (Groups B and C) showed a high level of satisfaction with the videos. These findings were consistent with our previous study[18] in which we evaluated understanding of information about a specific tumor mutation in melanoma. Taken together, these studies suggest that adherence to health literacy and learning style principles when designing consumer educational information can facilitate comprehension of complex information regarding genetics and cancer medicine.

While Group C showed greater gains in knowledge immediately after viewing the information relative to Group A, these differences were no longer apparent at the three week follow-up. The majority of the participants in the study indicated that they had not read the information given to them or visited the website version of the information since their clinic visit. These findings suggest that patients may need reminders to re-examine the information periodically and that patients would benefit from ongoing review of the information.

One limitation of the study is the lack of racial diversity among the study population. Most the participants were white. It will be important in future studies to determine whether the findings from this study are still applicable in non-white populations. Another limitation is that a majority of the participants had adequate health literacy. Given that the consumer information was written at the sixth grade level and developed by adhering to health literacy and suitability guidelines, we would predict that our findings would still be applicable in more diverse populations, particularly those with lower health literacy.

Research has shown that cancer patients want to participate in decisions about their care[25]. Although healthcare providers are often cancer patients’ main source of information, patients increasingly seek information from other sources, especially the Internet[26]. Patients looking for genetic information may be more likely to turn to the Internet instead of a healthcare provider[6]. However, evaluations of the state of online consumer health information indicate that attempts to keep it current are often a losing battle. To help meet the need for reliable, patient-friendly cancer information, the consumer content that we have developed will be made freely available online.

Several studies of popular online resources for various cancers have found that much of the content available has not been recently updated or does not indicate the last date of update[27 –32]. Failure to revise online materials to reflect new developments can lead to significant inaccuracies in the information patients access. For instance, one study[29] found that 12 of 38 websites (32%) with bladder cancer information included inaccuracies, largely due to outdated information. It has been suggested that one way to circumvent this problem is to redirect patients to clinicaltrials.gov and similar resources so they can stay abreast of emerging therapies[28]. However, many patients who discover a new precision medicine therapy in this manner may still have difficulty fully understanding its implications for their care. The MCG website was created using a template structure to facilitate updating. By adhering to a similar template design, the consumer versions may more easily be revised concurrently with the professional-level text.

Conclusion and Future Directions

As precision medicine moves to the forefront of patient care, it is becoming increasingly important to ensure that patients are readily able to understand their genetic results and communicate with their care team members. The knowledge pearls created through this project may be reused by oncologists and others to help patients understand complex precision medicine concepts and facilitate patient engagement in the healthcare process. Future opportunities for broader scalability could easily include the integration of the knowledge pearls into a variety of information systems, such as online patient portals designed to communicate to patients their health information. Informed by awareness of the difficulties and limitations of translating constantly evolving information to the consumer level, our approach of providing freely available consumer-friendly explanations of fundamental concepts offers a reusable and scalable alternative to traditional patient education efforts that can be applied to expand consumer knowledge in a variety of medical fields.