The Quality and Readability of Information Available on the Internet Regarding Lumbar Fusion.

Study Design An Internet-based evaluation of Web sites regarding lumbar fusion. Objective The Internet has become a major resource for patients; however, the quality and readability of Internet information regarding lumbar fusion is unclear. The objective of this study is to evaluate the quality and readability of Internet information regarding lumbar fusion and to determine whether these measures changed with Web site modality, complexity of the search term, or Health on the Net Code of Conduct certification. Methods Using five search engines and three different search terms of varying complexity ("low back fusion," "lumbar fusion," and "lumbar arthrodesis"), we identified and reviewed 153 unique Web site hits for information quality and readability. Web sites were specifically analyzed by search term and Web site modality. Information quality was evaluated on a 5-point scale. Information readability was assessed using the Flesch-Kincaid score for reading grade level. Results The average quality score was low. The average reading grade level was nearly six grade levels above that recommended by National Work Group on Literacy and Health. The quality and readability of Internet information was significantly dependent on Web site modality. The use of more complex search terms yielded information of higher reading grade level but not higher quality. Conclusions Higher-quality information about lumbar fusion conveyed using language that is more readable by the general public is needed on the Internet. It is important for health care providers to be aware of the information accessible to patients, as it likely influences their decision making regarding care.

Introduction

The Internet has become a major resource for health-related issues for the general public. The vast majority of the patients seeing an orthopedic surgeon now have access to the Internet, and most of these patients have researched their conditions on the Internet.1 In one study, 75% of patients in an outpatient spine clinic had access to the Internet and nearly a quarter of patients used the Internet to research their condition.2 As the accessibility and familiarity of the Internet continues to grow, the contents of the Internet will become increasingly important in shaping patients' understandings of their disease, and ultimately, in their health decision making. Close scrutiny of the contents of the Internet is necessary as we move forward in an age of shared decision making.

Patients use the Internet to gain better understanding of their conditions and subsequently to frame their perceptions of their disease. Furthermore, after retrieving and digesting the information from the Internet, patients will begin to weigh their treatment options prior to seeing their provider. The Internet information regarding scoliosis,3 cervical disk herniation,4 cervical disk replacement,5 lumbar disk herniation,6 lumbar spinal stenosis,7 and cauda equina syndrome8 has been previously reported in the literature; however, to date, there have been no studies focused on the Internet information regarding lumbar fusion. Considering the large amount of lay press regarding lumbar fusion in recent years, increasing rates of the procedure, as well as continued controversies about indications and insurance coverage about lumbar fusion, the accuracy of online information available to patients is important.9 10 11

The objectives of our study were to evaluate the quality and readability of information available on the Internet regarding lumbar fusion, whether the quality and readability of this information varied with Web site modality (academic, organizational, commercial, etc.), whether the quality and readability varied with search term, and whether the quality and readability varied with Health on the Net Code of Conduct (HONcode) certification.

Materials and Methods

We chose three search terms of varying levels of medical complexity to simulate different possible search scenarios when patients attempt to find information regarding lumbar fusion on the Internet. The search terms we used, in order of increasing complexity, were “low back fusion,” “lumbar fusion,” and “lumbar arthrodesis.”

We entered the search terms into five search engines on June 15, 2014, resulting in a total of 15 unique searches. The search engines we employed were Google, Bing, Yahoo!, Ask, and AOL, which were the five most popular search engines at the time of the investigation.12 We collected the first 30 results from each of the 15 searches for a total of 450 resultant Web sites. Duplicate Web sites, nonfunctional Web sites, Web sites clearly unrelated to patient information regarding lumbar fusion, and video Web sites were excluded from our analysis. Application of our exclusion criteria resulted in a list of 153 unique Web site hits for final review.

Resultant Web sites were further categorized by Web site modality for the first analysis. Modalities included academic, organizational, physician, nonphysician, commercial, medical billing, medicolegal, media, social networking, and miscellaneous. Academic Web sites were defined as those affiliated with or published by a university or a medical center. Organizational Web sites were defined as those affiliated with or published by a nonprofit or professional organization. Physician Web sites were defined as those published by a physician or physician group not affiliated with an academic institution. Nonphysician Web sites were defined as those published by nonphysician health care providers including chiropractors, physical therapists, or alternative medicine practitioners. Commercial Web sites were defined as those employing advertisement or other profit-generating devices. Medical billing Web sites were defined as focused on health care reimbursement. Medicolegal Web sites were defined as those focused on the legal or regulatory aspects of care. Media Web sites were defined as nonacademic news agencies. Social networking Web sites included forums, blogs, and other user-driven discussion platforms. Web sites not included in the above categories were classified as miscellaneous. Resultant Web sites were also categorized according to search term for the second analysis. Our search terms ranged from low complexity (“low back fusion”), to intermediate complexity (“lumbar fusion”), to high complexity (“lumbar arthrodesis”). Finally, resultant Web sites were categorized according to HONcode certification. The Health on the Net Foundation is a nongovernmental organization, accredited to the United Nations, founded in 1995, with the aim of helping citizens gain access to quality health information on the Internet. HONcode certification displayed on the Web site signifies fulfillment of the ethical standards set forth by the Health on the Net Foundation.

The quality of Web site information was assessed using a scoring system based on the elements of informed consent (Table 1),13 which we take to be appropriate because lumbar fusion is an operative intervention and the decision to operate is conducive to shared decision making. This system places quality of information on a scale from 1 to 5, from unacceptable to excellent. An unacceptable score denotes an omission of indication, benefit, or description of the relevant operation, whereas an excellent score denotes discussion of at least one indication, benefit, risk, description, alternative, and peer-reviewed literature of the operation. For further binary analysis, Web sites with a score of 3 or greater were said to have satisfactory quality, whereas Web sites with a score of 1 or 2 were said to have unsatisfactory quality.

Table 1

Information quality scoring system

Score	Assessment	Criteria
5	Excellent	Indication, benefit, risk, alternative, description, peer-reviewed literature
4	High	Indication, benefit, risk, alternative, description
3	Moderate	Indication, benefit, risk, description
2	Low	Indication, benefit, description
1	Unacceptable	Omission of indication, benefit, or description

The readability of Web site information was assessed using the Flesch-Kincaid score for reading grade level, which has been previously used in evaluating Internet information across many fields.8 14 15 16 A higher Flesch-Kincaid grade level indicates the need for the completion of a higher academic grade level to read and comprehend the material. A higher Flesch-Kincaid grade level is correlated with lower readability.

Descriptive statistics for quality and readability were calculated for all Web sites as well as for Web sites grouped by modality and search term. Unpaired comparisons of data among different Web site types were made using the analysis of variance (ANOVA) test for parametric readability data and the Kruskal-Wallis test for nonparametric quality data. Similarly, unpaired comparisons of data among all three different search terms were made using the ANOVA test for readability and the Kruskal-Wallis test for quality. For unpaired comparisons between the two groups, the Student t test was used for parametric readability data, and the Mann-Whitney U test was used for nonparametric quality data. The standard significance criterion of α = 0.05 was employed for all statistical tests.

Results

The average quality score for all unique Web sites found in our study, given as mean ± standard deviation, was 2.1 ± 1.4; the average Flesch-Kincaid reading grade level was 11.9 ± 3.2. Of the final list of 153 Web site hits, 41 were commercial, 40 were academic, 23 were physician, 16 were medical billing, 10 were organizational, 9 were social networking, 7 were media, 3 were nonphysician, 3 were miscellaneous, and 1 was medicolegal (Fig. 1). Nonphysician, miscellaneous, and medicolegal Web sites were excluded from further descriptive analysis due to an insufficient number of hits. The search term “low back fusion” yielded 65 unique Web sites, “lumbar fusion” yielded 63 unique Web sites, and “lumbar arthrodesis” yielded 70 unique Web sites.

Fig. 1

A pie chart of the various modalities of the unique Web site hits in this study.

For academic Web sites, the average quality score was 2.3 ± 1.5 and the average Flesch-Kincaid reading grade level was 13.0 ± 3.3. For organizational Web sites, the average quality score was 3.5 ± 1.5 and the average Flesch-Kincaid reading grade level was 10.5 ± 2.8. For physician Web sites, the average quality score was 2.1 ± 1.2 and the average Flesch-Kincaid reading grade level was 11.6 ± 2.8. For commercial Web sites, the average quality score was 2.2 ± 1.3 and the average Flesch-Kincaid reading grade level was 11.4 ± 2.8. For medical billing Web sites, the average quality score was 1.3 ± 1.0 and the average Flesch-Kincaid reading grade level was 14.0 ± 3.0. For media Web sites, the average quality score was 1.0 ± 0.0 and the average Flesch-Kincaid reading grade level was 12.8 ± 3.8. For social networking Web sites, the average quality score was 1.2 ± 0.7 and the average Flesch-Kincaid reading grade level was 7.7 ± 2.6. The Kruskal-Wallis test demonstrated a significant difference in quality of information among the various Web site modalities (p = 0.0002; Fig. 2). The Mann-Whitney U test showed that organizational Web sites had significantly higher-quality information (p = 0.003), whereas medical billing, social networking, and media Web sites had significantly lower-quality information (p = 0.008, 0.04, 0.02, respectively). The ANOVA test demonstrated a significant difference in readability among the various Web site types (p = 0.0002; Fig. 3). The Student t test showed that medical billing and academic Web sites were of significantly higher reading grade level (p = 0.007, 0.02, respectively), whereas social networking Web sites were of significantly lower reading grade level (p = 0.00004).

Fig. 2

The Internet information quality score for various modalities of Web sites. The average quality score for each Web site type is shown with error bars.

Fig. 3

The Flesch-Kincaid reading grade level is shown for various modalities of Web sites. The average reading grade level for each Web site type is shown with error bars.

Using “low back fusion” as the search term, the average quality score for Web site hits was 2.1 ± 1.3 and the average Flesch-Kincaid reading grade level was 10.8 ± 3.0. Using “lumbar fusion” as the search term, the average quality score for Web site hits was 2.5 ± 1.3 and the average Flesch-Kincaid reading grade level was 11.2 ± 2.8. Using “lumbar arthrodesis” as the search term, the average quality score for Web site hits was 2.1 ± 1.5 and the average Flesch-Kincaid reading grade level was 12.8 ± 3.2. The Kruskal-Wallis test demonstrated no significant difference in quality of information among the three search terms (p = 0.07; Fig. 4). The ANOVA test demonstrated a significant difference in readability among the three search terms (p = 0.0002; Fig. 5). In particular, the Student t test showed a significant increase in reading grade level of the information found by searching “lumbar arthrodesis” compared with “low back fusion” (p = 0.0002) or “lumbar fusion” (p = 0.002).

Fig. 4

The Internet information quality score is shown for various search terms. The average quality score for each search term is shown with error bars.

Fig. 5

The Flesch-Kincaid reading grade level is shown for various search terms. The average reading grade level for each search term is shown with error bars.

For Web sites with the HONcode certification, the average quality score was 1.7 ± 1.1 and the average Flesch-Kincaid reading grade level was 8.9 ± 2.2. For Web sites without the HONcode certification, the average quality score was 2.1 ± 1.4 and the average Flesch-Kincaid reading grade level was 12.4 ± 3.1. The Mann-Whitney U test showed no significant difference in quality based on the presence or absence of the HONcode certification (p = 0.3); however, the presence of the HONcode certification was associated with significantly improved readability (p = 5.5 × 10−6).

Only 34% of the unique Web sites analyzed were of satisfactory quality (quality score of 3 or greater) and only 4% were of satisfactory readability (sixth-grade reading level or below). Only one Web site found was deemed to have both satisfactory quality and satisfactory readability (Fig. 6).

Fig. 6

A scatter plot is shown for quality versus readability, where each marker represents a unique Web site. Satisfactory quality denotes a quality score of 3 or greater, whereas unsatisfactory quality denotes a quality score of 1 or 2. Satisfactory readability denotes a reading grade level of 6 or less, whereas unsatisfactory readability denotes a reading grade level of greater than 6.

In summary, the average quality score for Web site hits was low, and the average reading grade level of the Internet information was nearly twelfth grade. Organizational Web sites tended to have the highest-quality information, followed, in descending order, by academic, commercial, and physician Web sites; finally, medical billing, social networking, and media Web sites tended to have the lowest-quality information. Medical billing and academic Web sites tended to be written at the highest reading grade levels, whereas social networking Web sites were written at the most accessible reading grade levels. More complex search terms yielded information of higher reading grade level but not of higher quality.

Discussion

Prior studies show that the quality of Internet information about orthopedic injuries is widely variable and even misleading in some cases.14 17 18 For certain spine conditions, common Internet search hits have been shown to emphasize benefits more than risks and indications more than contraindications or complications when discussing operative versus nonoperative treatments, which can potentially result in a misleading representation of expected surgical outcomes.14 18 Studies of the readability of orthopedic patient education information on the Internet show that these materials are consistently written at a reading grade level above that recommended for the average patient.7 15 16 To date, there have been no studies assessing the information available to the general public on the Internet regarding lumbar fusion. Often performed electively, lumbar fusion is a procedure that patients are able to research on the Internet prior to presentation to clinic. The informed patient is more prepared to participate in a discussion about treatment options, and recent publications have shown that when patients share in the decision-making process, they experience better subjective outcomes.19

In the present study, we demonstrated that the average quality of information on the Internet regarding lumbar fusion is low based on the grading scale used. Web sites from professional and nonprofit organizations had the highest-quality information, followed by academic, commercial, and physician Web sites, and finally, medical billing, social networking, and media Web sites had the lowest-quality information. Despite evidence that health education information is best written at a fifth- or sixth-grade level,20 21 our data consistently demonstrated that information regarding lumbar fusion on the Internet is on average written at nearly a twelfth-grade reading level. Medical billing and academic Web sites were written at the highest reading grade levels, whereas social networking Web sites were written at the lowest reading grade levels. Social networking Web sites (forums, blogs, threads, etc.) are generally written by the lay public, and the readability of their information should serve as an example. The selection of a specific search term did not affect the quality of information found, but more complex search terms did result in information written at a higher reading grade level. The presence of the HONcode certification did not affect the quality of information found, but was significantly associated with information of lower reading grade level.

One limitation of our study is that we have used only one quality measure for health information. Our quality scoring system is based on a previous scoring system used to assess Internet information regarding vertebroplasty and emphasizes the elements of an informed consent.13 We believe our scoring system is appropriate to the topic of lumbar fusion because the crux of the shared decision making in this setting is generally between operative and nonoperative treatment. Other authors have used the DISCERN scoring system to assess consumer health information22; however, end-user testing of the DISCERN scoring system showed poor interrater agreement for questions where subjective judgment was required, particularly among untrained users.23 A second limitation of our study is that we used only one readability measure, the Flesch-Kincaid reading grade level. Although other formulas for reading grade level are available, the main concern for the Flesh-Kincaid is that it may actually underestimate the reading grade level compared with other measures.24 This potential bias only strengthens our conclusion that Internet information is written at too high a reading grade level. A third limitation of our study is our inability to tailor recommendations to specific segments of the patient population. Recent research has shown that access to the Internet varies not only by age, but also by ethnicity, income, and level of education.25 Future research should focus on Internet access, search habits, and search term complexity within segments of the patient population. A final limitation of our study is that the reliability of displays of HONcode certification itself has been called into question,26 and our study was unable to parse out fraudulent uses of the HONcode logo.

Although there is a wide spectrum of information on the Internet regarding lumbar fusion in terms of quality and readability, the information is on average of low quality and low readability. Authors of academic and organizational Web sites face the challenge of presenting more complete and accurate information in more accessible terms. It is important for health care providers to be aware of the information already accessible to patients on the Internet when they present to clinic. We have found that more complex search terms were not associated with better-quality information, but only information written at a higher reading grade level. Patients who tend to use more complex search terms may tend to be more comfortable reading at a higher grade level, but this hypothesis warrants further investigation. Potential recommendations to patients are to visit Web sites published by professional organizations and to avoid medical billing, media, and social networking Web sites to receive the highest-quality information. Further research is warranted on the quality and readability of the spine surgery information available on the Internet.