Assessing the Accuracy, Quality, and Readability of Patient Accessible Online Resources Regarding Ocular Gene Therapy and Voretigene Neparvovec.

PURPOSE: To evaluate the accuracy, quality, and readability of online information regarding the Food and Drug Administration (FDA) approved ocular gene therapy voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA).
METHODS: Ten online resources about voretigene neparvovec were assessed in this cross-sectional study. A novel 25-question assessment was created to evaluate the information most relevant to patients. Each article was assessed by independent graders using the assessment and the DISCERN instrument. An online readability tool, Readable, was used to assess readability. Accountability was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.
RESULTS: The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points with significant variation in the content accuracy and quality between the articles (P=0.017). EyeWiki achieved the highest score and MedicineNet the lowest. The mean reading grade for all articles was 12.88 (SD 1.93, CI 95% ±1.19) with significant variation between articles (P=0.001). Wikipedia was the most readable, and the FDA website was the least. None of the articles achieved all four JAMA benchmarks, and only one of the ten articles, EyeWiki, achieved three of the four JAMA benchmarks.
CONCLUSION: The information available online regarding this FDA-approved ocular gene therapy is generally of low quality, above the average reading level of the general population, and varies significantly between sources. The articles provide incomplete information that is not entirely accurate or easy to read, and as a result, the material would not support patients adequately in their medical decisions and questions about this new therapeutic option.

Introduction

Gene augmentation therapy is a technique in which genetic material is introduced into a patient’s cells to treat or prevent disease.1 Gene therapy was introduced in human subjects in the early 1990s, and currently, it is being recognized as a promising treatment option for multiple diseases.2 Several trials have been conducted or are ongoing to evaluate the efficacy and safety of gene therapy, including but not limited to hematopoietic stem cell based gene therapy for β-thalassemia major, liver-directed gene therapy for hemophilia B, and T-cell immunotherapy for acute lymphoblastic leukemia.2,3

Gene therapy has been of particular interest in ophthalmology. It has been studied in years of experimentation to determine its use to treat several ocular conditions including neovascularization, retinitis pigmentosa, Stargardt disease, retinoblastoma, and more.4 In 2017, voretigene neparvovec (Luxturna, Spark Therapeutics, Philadelphia, PA, USA) became the third gene therapy to be approved by the Food and Drug Administration (FDA).5 Voretigene neparvovec was unanimously approved by the FDA for patients with the biallelic RPE65 gene mutation form of hereditary retinal dystrophy.5 In the pivotal study, patients with this inherited retinal dystrophy were treated with subretinal injections of voretigene neparvovec and showed improvement in their functional vision which persisted for at least 4 years.6,7

With the approval of voretigene and the subsequent press coverage, patients with eye conditions are likely to learn about treatment options on the internet. Millions of Americans search health-related information and seek medical advice online every day.8 Previous research shows about 62% of Internet users search the internet for health information.8 The content acquired online influences patients’ health decisions.9 The content patients encounter would ideally be easy to comprehend. Most patients read at an eighth grade level; however, most sources of medical information are written at a tenth grade level or higher.10 Low health literacy has been associated with poor medication adherence and adverse outcomes such as uncontrolled chronic disease and increased hospitalizations.11 For these reasons, it is critical that the information patients obtain is accurate, complete, and easy to understand. As such, the goal of this study is to assess the accuracy, quality, and readability of online information regarding ocular applications of this gene therapy.

Methods

Article Selection and Content Analysis

The keywords “eye gene therapy” or “voretigene neparvovec” or “Luxturna” were entered in Google.com and in major medical websites such as the American Academy of Ophthalmology (AAO) website, and relevant articles were selected for analysis from the top search results with exclusion of similar results from the same organization. Preference was given to articles of formal as well as commercial organizations trusted by ophthalmologists and those believed to be regularly accessed by patients such as Wikipedia. The chosen articles included ones from the AAO,12 American Optometric Association (AOA),13 EyeWiki,14 US Food and Drug Administration (FDA),15 Spark Therapeutics,16 MedicineNet,17 National Eye Institute (NEI),18 Novartis,19 WebMD,20 and Wikipedia.21 A twenty-five question grading tool was designed by two vitreoretinal surgeons (JS and AEK) to assess the accuracy and quality of patient-relevant information in each article (Table 1). The questions were chosen to match presumed typical questions a patient or patient family member would ask and potential topic discussion when considering treatment with voretigene neparvovec. The evaluation of each website was completed independently by two vitreoretinal surgeons (JS and AEK) and one vitreoretinal surgery fellow (NY). A grading scale from 0–4, with 4 as a maximum, was utilized to evaluate each question as detailed in Table 1. A score of 0 was given if there was no information provided regarding the question. A score of 1 indicated the response was unclear, had inaccurate information, omitted significant information, and was very poorly organized. A score of 2 was for partial answers that somewhat addressed the concept but had gaps in the information and organization. A score of 3 meant the article provided essential elements to answer the question, addressed the most relevant points, and was focus and organized. A maximum score of 4 indicated the answer was accurate and thorough; the article provided a clear answer that was well explained, focused, and organized. Each article was independently graded by each observer, and interobserver reproducibility was assessed with a Spearman correlation. The average score between the three graders was used to compare the quality of the articles.

Table 1

Questionnaire to Assess Online Resources Describing Ocular Gene Therapy with Mean and Total Scores

	Mean Pointsa
Questions	AAOb	AOAc	EyeWiki	FDAd	MedicineNet	NEIe	Novartis	Spark Therapeutics	WebMD	Wikipedia
What is a gene?	0.33	0.00	1.67	1.33	0.33	1.33	0.67	3.33	0.67	1.33
What is gene therapy?	2.33	1.33	4.00	2.67	0.67	1.67	1.67	3.67	0.67	1.33
How does gene therapy work?	2.33	2.00	4.00	3.00	0.67	3.00	2.67	3.33	0.67	1.33
Are there any FDA approved gene therapy treatments?	3.33	2.67	4.00	4.00	3.33	0.00	1.33	4.00	4.00	4.00
What conditions can be treated with gene therapy?	3.33	3.00	4.00	4.00	2.00	3.00	3.67	3.33	3.33	3.67
What is retinal dystrophy?	1.33	0.67	1.33	3.33	1.00	1.00	3.00	1.67	1.67	1.00
What is Leber’s Congenital Amaurosis?	2.67	1.33	2.00	1.00	0.67	3.00	3.00	3.00	1.33	3.33
Are all patients with Leber’s Congenital Amaurosis eligible for gene therapy?	3.00	2.67	3.00	1.00	1.33	2.00	3.00	3.33	2.00	2.33
What is RPE65?	1.67	1.33	4.00	3.33	1.33	4.00	3.33	3.67	2.33	2.67
Does gene therapy cure the conditions it is used to treat?	2.00	0.67	0.67	1.00	0.67	3.33	1.67	1.00	1.00	3.67
At what age may gene therapy be administered?	0.00	0.00	0.00	1.00	1.67	0.00	0.00	3.67	2.67	0.67
What is a vector?	0.00	0.00	4.00	1.33	0.00	2.67	0.00	1.00	0.00	2.67
What vectors can be used for gene therapy?	0.00	0.00	4.00	0.67	0.00	0.67	0.00	1.00	0.00	2.33
What are the potential risks associated with gene therapy?	0.00	0.00	3.33	3.67	1.00	0.00	0.00	4.00	2.33	1.00
What is the price of gene therapy?	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	2.67
Does insurance approve gene therapy procedures?	0.33	0.00	0.00	0.00	0.00	0.00	0.33	0.33	0.00	0.00
What is Luxturna?	3.33	0.00	3.67	4.00	2.00	1.33	3.33	4.00	3.67	4.00
What is an intravitreal injection?	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Are there FDA approved gene therapies delivered through intravitreal injection?	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
What is a pars plana vitrectomy with a subretinal injection?	0.00	1.00	2.67	1.67	0.00	0.00	0.00	1.33	0.00	1.00
Are there FDA approved gene therapies delivered through pars plana vitrectomy with a subretinal injection?	0.00	1.33	4.00	2.67	0.00	0.00	0.33	2.67	1.33	0.67
How long can results of treatment last?	2.00	0.00	0.00	0.33	0.00	2.33	0.67	0.00	0.67	0.00
Are repeat treatments of gene therapy necessary?	0.00	0.00	0.00	0.00	0.00	0.00	0.67	0.67	1.00	0.00
Is there gene therapy available for retinitis pigmentosa?	2.33	0.00	0.67	0.33	0.00	0.00	0.67	0.67	0.00	1.33
Is there gene therapy available for Stargardt’s disease?	0.67	0.00	0.33	0.33	0.00	0.00	0.00	0.00	0.00	0.33
Total Points, Max 100	32.00	18.00	51.33	40.67	16.67	29.33	30.00	49.67	30.33	41.33
Points, Mean	1.28	0.72	2.05	1.63	0.67	1.17	1.20	1.99	1.21	1.65
SD	1.25	0.96	1.71	1.44	0.85	1.33	1.31	1.53	1.21	1.34
95% CI	0.49	0.38	0.67	0.56	0.33	0.52	0.51	0.60	0.48	0.53

Notes: aScores of individual questions scaled from 0–4. bAmerican Academy of Ophthalmology. cAmerican Optometric Association. dU.S. Food and Drug Administration. eNational Eye Institute. Table 1 displays the 25 questions designed to assess accuracy and quality of each article along with the mean number of points each article received for each question. A grading scale from 0–4, with 4 as a maximum, was used to evaluate each question. Each score in the table is the average of three scores given by three independent graders.

The articles were further evaluated by two graders (JS and NY) using the DISCERN instrument. DISCERN is widely used to determine the quality of health information provided to patients.22 Each article was independently graded using the 16-question tool. Each question is rated using a 5-point scale ranging from “No” to “Yes.” A score of 1 is given if the answer to the question is “No.” A score of 5 is given if the answer is a definite “Yes.” Scores of 2 through 4 are given if the article meets the criterion of the question partially. The interobserver reproducibility was determined with a Spearman correlation. The average score between the two graders was used to compare the quality of the articles.

Accountability Analysis

The accountability of each article was evaluated using the Journal of the American Medical Association (JAMA) benchmarks.23 Each article was assessed for the 4 standards: Authorship, attributions, disclosure, and currency. Each article should include the authors and contributors along with their affiliations and relevant credentials to meet the authorship requirements outlined by JAMA. Attributions, or references, should be reported, and disclosures and currency, or date of update, should be specified.

Readability Analysis

The online tool, Readable, was used to analyze readability of the articles.24 Each article was assessed using the Flesch Reading Ease Score (FRE), Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and Simple Measure of Gobbledygook (SMOG) Index. The Flesch Reading Ease Score awards each article a score from 0–100 based on total words, sentences, and syllables in the text. Higher scores translate to better readability. A score between 70 to 80 is comparable to an eighth-grade reading level. The Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and SMOG Index are separate readability formulas which report the reading level required to understand the text. Each index states the US grade level of education reflected by the article. The mean score of these indices corresponds to the numerical US grade level. Therefore, a mean score of 12 indicates a 12th grade reading level while a score above 12 corresponds to a college reading level.

Statistical Analysis

Statistical analysis was completed using IBM SPSS Statistics for Mac, version 25.0, released 2017 (Armonk, NY: IBM Corp). Content analysis using the 25-question assessment and the DISCERN instrument was performed by treating the data as ordinal variables and analyzing with the Kruskal–Wallis test. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. Statistical significance was set at P≤0.05 for the main comparisons and the pairwise comparisons. The readability analysis was conducted using a Kruskal–Wallis test to compare the mean reading grade level for each article. A post-hoc Dunn-Bonferroni test was used to determine pairwise comparisons. A Spearman correlation test was carried out to evaluate correlation between accuracy and readability. Statistical significance was set at P≤0.05 for the main comparisons, Spearman correlation, and pairwise comparisons.

Results

Ten articles were analyzed for the study. The inter-observer reproducibility was statistically significant between JS and AEK (r=0.66, P=0.038) and between JS and NY (r=0.84, P=0.003) and approached statistical significance between AEK and NY (r=0.62, P=0.058). The average questionnaire score for all the articles was 33.93 (SD 11.21, CI 95% ±6.95) out of 100 possible points. There was a statistically significant difference in the content accuracy and quality between the articles (P=0.017). The top scoring article was EyeWiki with an average score of 51.33 points. MedicineNet was the lowest scoring article with an average score of 16.67 points (Table 2). There was a statistically significant difference in the articles’ scores on each of the 25 questions (P<0.001).

Table 2

Mean and Total Scores for Grading Online Resources Describing Ocular Gene Therapy

	Total Pointsa	Points, Mean	SD	CI 95%
EyeWiki	51.33	2.05	1.71	0.67
Spark Therapeutics	49.67	1.99	1.53	0.60
Wikipedia	41.33	1.65	1.34	0.53
FDAb	40.67	1.63	1.44	0.56
AAOc	32.00	1.28	1.25	0.49
WebMD	30.33	1.21	1.21	0.48
Novartis	30.00	1.20	1.31	0.51
NEId	29.33	1.17	1.33	0.52
AOAe	18.00	0.72	0.96	0.38
MedicineNet	16.67	0.67	0.85	0.33

Notes: aOut of a possible 100 points. bU.S. Food and Drug Administration. cAmerican Academy of Ophthalmology. dNational Eye Institute. eAmerican Optometric Association. Table 2 displays the total scores each of the 10 articles received on the 25-question assessment in order from highest to lowest score. These scores were obtained using the average score between three independent graders.

For the DISCERN instrument, the inter-observer reproducibility was statistically significant between the two graders (r=0.89, P<0.001). The average score for all of the articles with this grading tool was 45.20 (SD 8.40, CI 95% ±5.21) out of 80 possible points. Results of the instrument also showed a statistically significant difference in the content quality between the articles (P=0.043), with EyeWiki scoring highest with an average score of 60.00 points and MedicineNet scoring lowest with 30.00 points (Table 3). There was a statistically significant correlation between the articles’ scores on the 25-question grading scale and on the DISCERN instrument (r=0.90, P<0.001).

Table 3

Mean and Total Scores for Online Sources Describing Ocular Gene Therapy Graded Using the DISCERN Instrument

	Total Pointsa	Points, Mean	SD	CI 95%
EyeWiki	60.00	3.75	1.27	0.62
Spark Therapeutics	54.50	3.41	1.34	0.66
FDAb	53.00	3.31	1.21	0.59
Wikipedia	47.50	2.97	1.18	0.58
AAOc	46.00	2.88	1.02	0.50
NEId	41.50	2.59	1.06	0.52
Novartis	41.00	2.56	1.12	0.55
AOAe	39.50	2.47	0.89	0.44
WebMD	39.00	2.44	0.97	0.47
MedicineNet	30.00	1.88	0.89	0.44

Notes: aOut of a possible 80 points. bU.S. Food and Drug Administration. cAmerican Academy of Ophthalmology. dNational Eye Institute. eAmerican Optometric Association. Table 3 displays the total scores each article received on the 16-question DISCERN instrument in order from highest to lowest score. These scores were obtained using the average score between two independent graders.

None of the articles achieved all 4 JAMA benchmarks. One of the 10 (10%) articles, EyeWiki, achieved 3 of the 4 JAMA benchmarks (Table 4). Currency was the most displayed benchmark followed by attribution and authorship. There was no correlation between the accuracy of the articles and JAMA benchmarks (r=−0.229, P=0.524).

Table 4

JAMA Benchmarks Achieved by Online Sources Regarding Ocular Gene Therapy

JAMA Benchmarks	N (%)
4 Benchmarks	0 (0%)
3 Benchmarksa	1 (10%)
2 Benchmarks	3 (30%)
1 Benchmark	5 (50%)
0 Benchmarksb	1 (10%)
Attribution	4 (40%)
Authorship	4 (40%)
Currency	6 (60%)
Disclosure	0 (0%)

Notes: aEyeWiki achieved 3 of the 4 benchmarks. bSpark Therapeutics achieved 0 of the 4 benchmarks. Table 4 displays the number of articles that achieved JAMA benchmarks and the most common benchmarks displayed by the articles. Each article was assessed for the 4 standards as determined by JAMA: Authorship, attributions, disclosure, and currency.

The mean Flesch Reading Ease Score for all the articles was 39.51 (SD 9.98, CI 95% ±6.19). The mean reading grade for all websites was 12.88 (SD 1.93, CI 95% ±1.19). There was no significant correlation between the FRE score and mean reading grade level (r=−0.340, P=0.336). There was a significant difference between the mean reading grade level of the websites (P=0.001). Wikipedia had the lowest mean reading grade level of 9.63, and the FDA website had the highest mean reading grade level of 16.73. Wikipedia (P=0.005) and the Spark Therapeutics product website (P=0.015) had significantly lower reading levels than the FDA website (Table 5). There was no significant correlation between website accuracy and the mean reading grade (r=−0.444, p=0.199).

Table 5

Readability Analysis of Online Sources Describing Ocular Gene Therapy

Readability	Wikipedia	Spark Therapeutics	AAOa	EyeWiki	MedicineNet	NEIb	AOAc	WebMD	Novartis	FDAd
Flesch Reading Ease	37.6	53.5	50.4	21.8	42.5	41.2	46.2	45.2	32	24.7
Mean Reading Grade	9.63	10.40	11.45	12.85	13.03	13.03	13.63	13.75	14.35	16.73
Mean Reading Grade SD	1.22	1.05	0.90	1.03	0.93	0.92	1.91	1.39	0.96	1.76
Mean Reading Grade CI (95%)	1.20	1.03	0.88	1.01	0.91	0.90	1.87	1.36	0.94	1.72

Notes: aAmerican Academy of Ophthalmology. bNational Eye Institute. cAmerican Optometric Association. dU.S. Food and Drug Administration. Table 5 displays the mean readability scores as determined for each article by the online tool, Readable, and reported in increasing mean reading grade order. The Flesch Kincaid Grade Level, Gunning Fox Index, Coleman Liau Index, and SMOG Index state a US reading grade level. The mean score of these indices was used to determine the numerical grade level of the article. A mean reading grade of 12 indicates a 12th grade reading level while a score above 12 corresponds to a college reading level.

Discussion

The results (Table 6) demonstrated that there was significant variation in the quality, accuracy, and readability of the information available to patients about the ocular gene therapy voretigene neparvovec. EyeWiki provided the most complete and accurate information of the articles analyzed and achieved the most benchmarks of all the articles fulfilling 3 of the 4 benchmarks. However, EyeWiki had a mean reading grade level of 12.85, and is difficult to read for a layperson, as it is primarily designed for ocular provider education rather than patient consumption. Still, given that it is freely available and accessible by the public, it was included in the analysis.

Table 6

Summary Table of Grading Indices Used to Assess Online Sources Describing Ocular Gene Therapy

Article Source	25-Question Assessment Total Pointsa	DISCERN Instrument Total Pointsb	Number of JAMA Benchmarks Achievedc	Mean Reading Graded
AAOe	32.00	46.00	1	11.45
AOAf	18.00	39.50	1	13.63
EyeWiki	51.33	60.00	3	12.85
FDAg	40.67	53.00	1	16.73
MedicineNet	16.67	30.00	2	13.03
NEIh	29.33	41.50	2	13.03
Novartis	30.00	41.00	2	14.35
Spark Therapeutics	49.67	54.50	0	10.40
WebMD	30.33	39.00	1	13.75
Wikipedia	41.33	47.50	1	9.63

Notes: aOut of a possible 100 points. bOut of a possible 80 points. cOut of a possible 4 benchmarks. dAverage of 4 indices reporting US reading grade levels. eAmerican Academy of Ophthalmology. fAmerican Optometric Association. gU.S. Food and Drug Administration. hNational Eye Institute. Table 6 provides a summary of all the grading indices used in assessing the articles including the 25-question assessment, the DISCERN instrument, the number of JAMA benchmarks achieved, and the mean reading grade. The total points on the 25-question assessment is the average score between three independent graders. The total points on the DISCERN instrument is the average score between two independent graders. The number of JAMA benchmarks achieved quantifies the number each article met of the 4 JAMA benchmarks: Authorship, attributions, disclosure, and currency. The mean reading grade states a US reading grade level with a mean score of 12 reflecting a 12th grade reading level while a score above 12 corresponds to a college reading level.

The second-highest scoring resource on both grading tools was the Spark Therapeutics patient informational website regarding voretigene neparvovec; it was the second most readable article as well with a mean reading grade level of 10.40. Nevertheless, it scored only 54.50 out of 80 points on the DISCERN grading tool and below 50 out of 100 points on average on the 25-question assessment. It did not achieve any of the JAMA benchmarks; moreover, despite being the highest scoring resource designed for reading by patients, it is not an ideal source of reference given the potential inherent conflict of interest present in the manufacturing company’s discussion of its own product.25

In this study, the information available from formal organizations such as the FDA, AAO, and NEI was of overall low quality. Ideally, these major organizations would have accurate and complete information that is easy to understand as many patients rely on these institutions for medical knowledge. Prior studies have shown that educated patients have greater trust in federal agencies and community organizations as sources of medical information.26 Overall, patients prefer non-commercial websites especially from established organizations such as the FDA and NEI over other sources.27,28 The FDA website received the highest content and DISCERN average score of these three resources; however, it had a mean reading grade level of 16.73, the highest level of all the articles. Although the AAO and NEI websites were easier to read with mean reading levels of 11.45 and 13.03, respectively, both resources scored poorly on both scales. Furthermore, the AAO and FDA websites only achieved 1 of the 4 JAMA benchmarks while NEI achieved 2 of the 4 benchmarks demonstrating the overall poor accountability of the articles, although one could argue that the organization as a whole may be taking accountability for the content, which would not be identified by the JAMA benchmark scoring.

This study was limited in several aspects. The 25-question assessment was based on two retina providers' experience and assumptions regarding typical patient questions and was not validated in a patient population. Furthermore, the inter-observer correlation between two pairs of the graders resulted in moderate strength of correlation demonstrating variability in interpretation of the articles or of the grading scale. To further test the online resources being evaluated, the standardized DISCERN instrument was also incorporated, and there was a significant and strong positive correlation between the DISCERN scores and the content assessment scores. Inter-observer reproducibility had greater strength in the DISCERN instrument as well. Patients may use search engines or keywords that vary from the ones used to determine websites included in this study. Some resources included in this study were found beyond the second results page on Google, but patients often only use the first two results pages for their research.28 These resources were still included given their importance as prominent, public organizations (eg NEI, FDA). The resources evaluated were non-peer reviewed sources of varied websites including commercial, government, and non-government organizations. These articles were chosen as they were readily available during search, but patients may prefer to read peer reviewed sources or information from other organizations. Resources were not directly assessed by patients; future studies could further study online resource quality and content by directly evaluating patients’ comprehension.

In conclusion, this study demonstrated that patient accessible online resources regarding the ocular gene therapy voretigene neparvovec are generally incomplete, difficult to read, and of poor quality. The available material varied significantly by source and did not provide adequate information to aid patients in their medical decisions. This indicates an unfilled need to create patient-friendly online content regarding this ocular gene therapy as available research and therapeutic options continue to evolve. Future articles can be improved by assuring accuracy of the information using peer reviewed research papers as reference and by maintaining a reading level sufficient for patients’ understanding.