Availability and Readability of Online Patient Information on Osteosarcoma: Assessment of Pediatric Hospital and National Cancer Institute-Designated Cancer Center (NCIDCC) Osteosarcoma Web Pages.

Online patient information (OPI) plays an important role in pediatric orthopaedic patient/caregiver education and decision-making. We assessed the availability and readability of OPI about osteosarcoma found at pediatric hospital and U.S. National Cancer Institute-designated cancer center (NCIDCC) websites.
METHODS: The websites of all NCIDCCs and the top 50 pediatric hospitals identified using U.S. News & World Report were included. The names of NCIDCCs and pediatric hospitals along with the terms "osteosarcoma," "bone sarcoma," and "sarcoma" were entered into Google.com, and were classified according to the availability of osteosarcoma-specific web pages. Unpaid monthly visits were assessed using the Ahrefs.com Organic Traffic Score (OTS) metric. Readability was assessed using 5 validated metrics and the composite grade level (CGL), the rounded mean of the 5 metrics.
RESULTS: Of the 71 NCIDCCs and 50 pediatric hospitals, 48 (67.6%) and 18 (36.0%), respectively, did not have at least 1 web page dedicated to osteosarcoma-specific OPI. The mean OTS for all 116 NCIDCC and pediatric hospital osteosarcoma-specific web pages assessed was 177 estimated visits per month, which was less than the mean OTS for the top 10 Google.com "osteosarcoma" search results (3,287.9; p < 0.001). The 52 NCIDCC web pages with osteosarcoma OPI (representing 23 centers) had an average CGL of 12.9, representing a readability level of at least a high school degree. The mean CGL for the 64 pediatric hospital web pages with osteosarcoma OPI (representing 32 hospitals) was 12.8, also representing readability of at least a high school degree. Only 8 (12.5%) of the 64 web pages were written at a seventh or eighth-grade level.
CONCLUSIONS: Taken together, the majority of NCIDCCs and pediatric hospitals did not have a dedicated page of OPI for osteosarcoma. Of those that did, NCIDCC and pediatric hospital sites were visited much less frequently than sites visited through the most common Google.com searches. None of the osteosarcoma web pages offering OPI from NCIDCCs and pediatric hospitals met the American Medical Association readability recommendation (sixth-grade reading level). Therefore, greater effort must be made to create and direct patients and parents toward high-quality OPI of the appropriate level. CLINICAL RELEVANCE: The clinical relevance of this study lies in its evaluation of OPI and its ability to impact the patient experience of clinical care.

The prevalence of the internet has democratized health information access[1]. Over three-quarters of Americans seek health information online[2]. Many patients turn to the internet as their initial source of online patient information (OPI) and can be influenced by what they find online before seeking providers[1-4]. This is also true in pediatric care, where the vast majority of parents use the internet to learn more about their children’s conditions[5,6].

Despite the prevalence and influence of OPI use among patients and caregivers, studies in myriad fields across medicine[7-10] have found the quality of OPI to be wanting. The findings of many studies suggest that providers circumvent this quality deficit by suggesting academic or hospital-associated websites to patients[11,12]. Specifically, some recommend websites created by educational institutions, often those ending in “.edu”[13], or websites created by U.S. National Cancer Institute-designated cancer centers (NCIDCCs), as prior studies have identified them as providing some of the most accurate and evidence-based OPI[14].

Despite the potential quality of these resources, less is known about the availability and readability of these OPI resources. OPI access is an important area of study, as only OPI that users encounter may influence and benefit them. An evaluation of readability, a metric that approximates the American grade level required to comprehend a text adequately, could identify OPI that may be too complex for most users[14,15]. The American Medical Association (AMA) recommends OPI written at the level of sixth grade to facilitate its utility for most U.S. users[16].

Osteosarcoma is the most common primary bone malignancy in children and adolescents[17], and among adolescents, it is one of the most common cancers[18]. Osteosarcoma care is often complex and multidisciplinary, highlighting the importance of appropriate patient information in treatment decisions. Therefore, we sought to assess the availability and readability of OPI about osteosarcoma made available by pediatric hospitals and NCIDCCs, not only as a test case for OPI that is accessible through NCIDCCs or pediatric hospitals, but also as one of the first assessments of OPI for this serious pediatric condition.

Materials and Methods

Institutional Identification

All NCIDCCs were included in the study and were identified using the “Find a Cancer Center” section of the NCI website, at: https://www.cancer.gov/research/nci-role/cancer-centers/find. Pediatric hospitals were identified using the U.S. News & World Report ranking system for cancer care; the top 50 pediatric cancer centers were identified using https://health.usnews.com/best-hospitals/pediatric-rankings/cancer. All institutions identified in the study are presented in Appendix Supplemental Tables 1 and 2.

Website Search Methodology

The names of NCIDCCs and pediatric hospitals along with the terms “osteosarcoma,” “bone sarcoma,” and “sarcoma” were entered into the Google search engine between December 29, 2019, and February 22, 2020. A single author (J.Y.) manually searched through website content provided by each NCIDCC and pediatric hospital to verify web page content, until a maximum of 5 pages per site were included, as done by Ammanuel et al.[19] Findings were then verified by a second author (E.C.D.) for completeness. Pages that were not scientific papers were assumed to be patient information unless they explicitly specified that they were for providers. If an NCIDCC, a pediatric hospital, or its affiliated university or health-care system did not have at least 1 dedicated web page on the topic of osteosarcoma, it was excluded from further analyses. If an NCIDCC was affiliated with a pediatric hospital that was included in the analysis, or vice versa, the institutions were considered separately, given that—in all but 1 case—the NCIDCCs and pediatric hospitals hosted different websites despite having a common academic affiliation. Pages listing only osteosarcoma provider names or treatment locations were excluded. Patient stories and blog posts were also excluded. Web pages on osteosarcoma from websites associated with identified NCIDCCs and pediatric hospitals were evaluated for readability and website traffic.

OPI Availability Assessment

Websites of NCIDCCs and pediatric hospitals were assessed for the availability of osteosarcoma OPI. Websites were classified as 1 of the following: (1) having at least 1 osteosarcoma-specific web page, (2) having at least 1 paragraph but <1 dedicated page on osteosarcoma, (3) having at least 1 paragraph on sarcoma or bone cancer in general, but <1 paragraph on osteosarcoma, (4) having <1 paragraph of general sarcoma or bone cancer information, or (5) having no information at all on osteosarcoma, general sarcoma, or bone cancer.

Website Traffic Assessment

Website traffic was assessed using website analysis tools from Ahrefs, a website auditing and analytics company (ahrefs.com). Website traffic was assessed using the Organic Traffic Score (OTS) metric, which assigns an estimate of the number of organic (unpaid) page visits per month, based on a composite estimate of the traffic generated by the top 100 keywords associated with searches leading to the particular site over a 30-day average. By way of comparison, organic website traffic was also calculated for the top 10 web pages listed on Google.com from a search of the term “osteosarcoma” on March 5, 2020. The top 10 web pages were chosen because these represent the first page of Google.com results, and prior work has suggested that patients rarely search beyond the first Google.com page[20]. All explicit advertisements were excluded from this search.

Readability Assessment

Readability was assessed using the WebFX online tool (https://www.webfx.com/tools/read-able/), which assesses the readability of websites on the basis of the text presented. Readability was assessed using 5 validated metrics that make use of a combination of total words, sentences, and syllables to estimate the minimum grade level required to comprehend the text from each website. These included the Flesch-Kincaid Grade Level (FKGL), the Gunning Fog Score (GFS), the Simple Measure of Gobbledygook (SMOG) Index, the Coleman-Liau Index (CLI), and the Automated Readability Index (ARI). A higher grade level is correlated with an increased level of reading skill needed for text comprehension, where grade levels are equivalent to American grade levels[16]. Equations for each of these metrics are presented in Appendix Supplemental Figure 1.

To account for the variability among readability metrics, the mean of all 5 metrics, rounded to the nearest grade level, constituted the composite grade level (CGL), which has been reported previously[8].

Statistical Analyses

OTS values were compared between NCIDCC and pediatric hospital websites, and between NCIDCC and pediatric hospital websites and the top 10 Google.com search results, using Student t tests. For a more conservative estimate of the OTS difference, NCIDCCs and pediatric hospitals that were also present in the top 10 Google.com search were included in both groups. T tests were also used to assess whether readability indices were at or below the sixth-grade level, the recommended OPI reading level per the AMA. We also used t tests to compare readability between NCIDCC websites and pediatric hospital websites across all 5 metrics and the CGL. Pearson correlation coefficients were calculated to assess the association between OPI readability metrics and OTS values for all NCIDCC and pediatric hospital websites. Statistical analyses were performed using Stata/SE (StataCorp), with 2-sided α = 0.05.

Results

Assessment of OPI Availability

Of the 71 NCIDCCs, 48 (67.6%) did not have at least 1 osteosarcoma-specific web page, and of the 50 pediatric hospitals, 18 (36.0%) did not have at least 1 web page dedicated to osteosarcoma OPI (Fig. 1). Of these 66 NCIDCCs and pediatric hospitals without a dedicated osteosarcoma-specific OPI web page, 8 (12.1%) had no information at all on osteosarcoma, general sarcoma, or bone cancer; 24 (36.4%) had some (<1 paragraph) general sarcoma or bone cancer information; 26 (39.4%) had at least 1 paragraph on sarcoma or bone cancer; and 8 (12.1%) had at least 1 paragraph but <1 page on osteosarcoma.

Fig. 1

The availability of online patient information (OPI) for NCIDCCs and pediatric hospitals. “With osteosarcoma OPI” = houses at least 1 page dedicated to osteosarcoma, “At least one paragraph on osteosarcoma” = houses at least 1 paragraph but <1 page on osteosarcoma, “At least one paragraph for sarcoma or bone cancer” = houses at least 1 paragraph for sarcoma or bone cancer, but <1 paragraph on osteosarcoma, “General sarcoma information” = houses <1 paragraph on sarcoma or general bone cancer, and “No information” = no information provided.

The mean OTS for all 116 NCIDCC and pediatric hospital osteosarcoma-specific web pages assessed was 177.1, suggesting that approximately 177 visits per month were made to each web page, on average. The mean OTS for the NCIDCC websites was 207.9, and the mean OTS for the pediatric hospital websites was 152.2; there was no significant difference between the two (p = 0.69). In contrast, the mean OTS for the top 10 Google.com “osteosarcoma” search results was 3,287.9, which was significantly greater than that for the NCIDCC and pediatric hospital websites (p < 0.001) (Fig. 2). Of these 10, four were from NCIDCCs or pediatric hospitals (number 2: https://www.mayoclinic.org/diseases-conditions/osteosarcoma/symptoms-causes/syc-20351052, OTS = 6,500; number 4: https://www.stjude.org/disease/osteosarcoma.html, OTS = 2,800; number 9: https://www.hopkinsmedicine.org/health/conditions-and-diseases/sarcoma/osteosarcoma, OTS = 1,200; and number 10: https://www.dana-farber.org/bone-cancer-osteosarcoma/, OTS = 479).

Fig. 2

Comparison of mean monthly traffic estimates (and standard deviation) for the top 10 Google.com “osteosarcoma” search results and National Cancer Institute-designated cancer center (NCIDCC) and pediatric hospital (PH) web pages dedicated to osteosarcoma. Asterisks indicate a significant comparison (p = 0.05).

Of the NCIDCC web pages, 39 (75%) of 52 had an OTS of <10. Of the pediatric hospital web pages, 48 (75%) of 64 had an OTS of <10.

A total of 116 osteosarcoma-specific web pages representing 55 NCIDCCs and pediatric hospitals were assessed. The 52 NCIDCC web pages (from 23 centers) had a mean CGL of 12.9, representing at least a high school degree. The lowest reading level was tenth grade, for 8 (15.4%) of the 52 pages. For the 64 pediatric hospital web pages (from 32 hospitals), the mean CGL was 12.8, also representing at least a high school degree. Only 8 (12.5%) of the 64 pages were written at a seventh or eighth-grade level, and 65.6% required at least a high school degree to comprehend (Fig. 3). Among the websites identified in our top 10 Google.com search, the average CGL was 11.8. Reading metrics did not differ significantly from those of the NCIDCCs or pediatric hospitals (see Appendix Supplemental Table 3).

Fig. 3

Grade-level distributions of osteosarcoma-specific web pages. NCIDCC = National Cancer Institute-designated cancer center, and PH = pediatric hospital.

Across all 5 metrics and the CGL, the readability of the NCIDCC websites and the pediatric hospital websites was significantly greater than the AMA-recommended sixth-grade level (p < 0.001 for all) (Fig. 4).

Fig. 4

The readability of osteosarcoma online patient information found on National Cancer Institute (NCI)-designated cancer center and pediatric hospital (PH) web pages. Mean values (and standard deviation) are shown. The asterisk indicates a significant difference (p = 0.05). The bolded horizontal line indicates the AMA-recommended sixth grade reading level.

Only for the Coleman-Liau index (CLI) was a significant difference between NCIDCC and pediatric hospital websites demonstrated (NCIDCC CLI = 14.8, and pediatric hospital CLI = 13.7; p = 0.008); there were no significant differences across all other metrics (see Appendix Supplemental Table 4). Pearson correlation coefficients did not demonstrate significant associations between OTS and NCIDCC or pediatric hospital web page readability for any metric (see Appendix Supplemental Table 5).

Discussion

OPI plays an important role in patient education, and helps patients and their families to learn more about their conditions and make treatment decisions[4-6,11]. However, the availability and readability of OPI for pediatric cancers, including OPI provided by health-care institutions, remain poorly understood[11]. Consequently, we sought to assess the availability and readability of OPI provided by NCIDCCs and pediatric hospitals for osteosarcoma, given the complexity of care plans and the condition’s importance in pediatric orthopaedic oncology.

OPI Availability and Utilization

Our results suggest that OPI provided by NCIDCCs and pediatric hospitals remains limited in both availability and utilization. The websites of fewer than half of all NCIDCCs and pediatric hospitals provided at least 1 dedicated web page of osteosarcoma OPI; approximately half of the NCIDCCs and pediatric hospitals provided, at most, <1 paragraph of OPI specific to osteosarcoma. Additionally, organic traffic to the web pages of NCIDCCs and pediatric hospitals (OTS, 207.9 and 152.2, respectively) was significantly lower than traffic to sites populating the first page of search results on Google.com (3,287.9; p < 0.001), with 75% of the web pages from NCIDCCs and pediatric hospitals attracting <10 estimated organic visits per month. Prior studies have identified a wide variability in OPI quality and degree of bias but have found that information from academic and nonprofit websites is more accurate than OPI from media or private websites[3]. However, our results suggest that dedicated osteosarcoma OPI is not widely available from NCIDCCs or pediatric hospitals, and that when it is available, it is not frequently utilized compared with popular sites identified through a search on Google.com.

As noted in prior studies, the algorithms dictating what information appears on Google.com searches are not well understood by either physicians or patients[11], and patients do not routinely search beyond the first page of results provided by a Google.com search[20]. Consequently, the current OPI that patients search for and access most frequently may not be screened for clinical accuracy or impartiality and may prove to be misleading or harmful to the patient’s treatment goals and expectations[3]. Web pages associated with NCIDCCs, pediatric hospitals, and their affiliated academic centers represent a privileged space where high-quality OPI can be provided in a way that bridges the care system, physicians, and clinical care received through information available online. Our findings suggest that one method of addressing this challenge would be for health-care institutions and physicians to expand access to, and increase utilization of, quality OPI provided on institutional websites in order to better empower patients in seeking care.

OPI Readability

Our study also demonstrated that, for institutions with at least 1 dedicated osteosarcoma web page, the level of OPI readability for both NCIDCCs and pediatric hospitals is too high, often requiring a reading level equivalent to a high school degree (mean CGL, 12.9 for NCIDCCs and 12.8 for pediatric hospitals). None of the web pages evaluated met the AMA-recommended sixth-grade reading level[16]. This finding is consistent with prior studies that demonstrated that OPI across many specialties and sources is too complex in terms of readability[3,8,14,15]. Additionally, the findings of this study are concordant with a prior evaluation of osteosarcoma OPI that demonstrated a high level of OPI complexity in the most popular web pages identified via search engines[21]; our study builds on this and other studies in highlighting that OPI from osteosarcoma treatment centers is also overly complex.

Health literacy in the U.S. remains low, with only about 1 in 10 Americans possessing proficient health literacy[22]. Prior studies have indicated that health information remains at a level that exceeds the literacy levels of most adults[15]. Understanding appropriate OPI is integral to its utility for patients; patients trust and adhere more to recommendations that they comprehend[23,24]. In contrast, poor understanding engenders dissatisfaction and compromises outcomes[23,25], and OPI that is overly complex can negatively impact patients’ care-seeking and treatment decisions[26,27] and may also perpetuate disparities[8,28,29]. Our results indicate that the challenges posed by overly complex OPI are relevant to, and affect, online information provided by hospitals and academic centers, suggesting that addressing this problem will require, at least in part, solutions coming from clinical and academic institutions.

Our findings highlight the possibility that the need for improvement in OPI availability and readability may extend to other pediatric conditions. The onus lies with health-care providers to point patients and parents toward OPI that is at the appropriate level and of good quality[11]. Additional studies should ascertain whether or not these findings apply to other relevant conditions.

Limitations

Our study is not without limitations. First, our reported OTS represents an estimate based on the Ahrefs algorithm and not actual counted page visits. It remains unclear how its keyword-based algorithm may or may not bias estimated page visits; however, we are not aware of any directional bias. Furthermore, we would not expect differential bias in our comparison of NCIDCC and pediatric hospital OPI and OPI identified in our Google.com search, and consequently, any bias would not change our findings that OPI provided by NCIDCCs and pediatric hospitals was much less viewed compared with that of the sites listed on the first Google.com page. Second, readability is an imperfect proxy for information comprehension[15]. In the absence of other validated metrics, however, readability remains a well-studied, objective measure of reading difficulty that can inform OPI design and identify areas of further study. Third, we did not perform an assessment of the accuracy of OPI provided by the NCIDCCs and pediatric hospitals. However, prior work has indicated that OPI provided by hospitals and academic institutions is among the highest quality available[3]. Fourth, comparing the OTS of the top 10 Google.com results and that of the NCIDCCs and pediatric hospitals is an imperfect measure, as the former is subject to change on the basis of targeted advertising as well as date, prior search history, and saved browser cookies[30]. However, we present the comparison between NCIDCC and pediatric hospital OPI and that of the top 10 Google.com sites to demonstrate that most NCIDCC and pediatric hospital OPI is not being utilized frequently relative to what is commonly being accessed by patients. We did not seek to assess the quality or origins of the top 10 Google.com sites themselves. Finally, it remains unclear whether findings related to osteosarcoma OPI translate to other conditions, and additional work will need to assess how these findings fit within a broader context of orthopaedic OPI.

Conclusions

In conclusion, our study highlights several challenges that educational and health-care institutions face in providing OPI on pediatric osteosarcoma. While there have been calls for educational and health-care institutions to play a more active role in providing quality OPI[13,14], our study indicates that such OPI is not readily available, is poorly utilized, and is not easily readable by patients. Bolstering the availability and utilization of institutional OPI will be critical to properly addressing this problem, either through individual institutional efforts or through a coordinated, national-level program. However, additional work is needed, not only to evaluate the impact of such an intervention but also to improve the quality of OPI accessed by patients and caregivers and ultimately help better empower them to make informed decisions about their care.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A200).