Readability and Quality Assessment of Online Materials for Syndactyly Release.

Syndactyly is one of the most common congenital hand malformations, involving an abnormal fusion of digits and with treatment varying according to its complexity. The internet has become a primary source of information for both families and patients with congenital hand anomalies. The purpose of this study was to evaluate both the readability and quality of available web content for syndactyly, using validated instruments.
METHODS: Two independent reviewers conducted searches for "Syndactyly" using three of the largest online search engines: Bing, Google, and Yahoo. The top 10 websites for each search engine, along with any webpage within one click of the parent website, were analyzed. Readability was assessed using seven established quantitative tests. The quality of the web pages was analyzed using the Discern questionnaire and handbook.
RESULTS: A total of 15 websites were included in the analysis. The average readability of all websites was equivalent to comprehension at a grade 11.3 level. The average Flesch reading ease score was 49.3 out of 100, which is considered difficult to read. Quality was assessed using Discern, a brief questionnaire consisting of 16 questions with five points attributed per question. The mean quality score using Discern was 33.3 points out of a maximum of 80 points.
CONCLUSIONS: Online materials pertaining to the treatment of syndactyly far exceed the recommended sixth-grade reading level, and lack in terms of quality and comprehensiveness of information. Health care professionals should be cognizant of the paucity of available online information and provide patients with more appropriate resources.

Takeaways

Question: The internet has become a primary source of information for both families and patients with congenital hand anomalies. The purpose of this study was to evaluate both the readability and quality of available web content for syndactyly, using validated instruments.

Findings: The fifteen most easily accessible websites across three search engines were analyzed. The average grade level of all websites was 11.33, while the quality assessment according to the Discern questionnaire was 33.26/80.

Meaning: Online materials for treatment of syndactyly are overly verbose, and lack in quality and comprehensiveness of information.

INTRODUCTION

The use of the internet as a source of medical information has drastically increased over the last two decades.[1-6] The accessibility of online search engines has resulted in a significant percentage of patients utilizing the internet as their primary source of medical information. A National trends survey indicated that 49% of people first search online for medically related concerns, whereas only 11% seek advice from their physicians.[1,6] Despite this trend, physicians still remain the most highly trusted source of information for 62% of patients.[6]

With the knowledge that many of their patients rely on the World Wide Web, physicians should be responsible for providing appropriate, complimentary online educational material to improve the shared decision-making (SDM) process with their patients.

Limited health literacy has been associated with increased mortality, poorer health status, inadequate use of medical resources, longer hospitalizations and larger health care costs.[4,5,7] As a result, The American Medical Association and National Institute of Health have recommended that health related information be at or below a sixth to eighth grade reading level to minimize the sequelae of poor health literacy.[1,4,5] However, previous studies evaluating the readability of various surgical subspecialty topics reported grade levels ranging from 10th to 15th grade.[1,3,4,7]

In addition to being of a suitable readability level, online scientific resources should be of adequate quality. Multiple metrics, such as the Discern handbook, have been developed to critically appraise various online resources. Patients, unlike physicians, often lack formal training to evaluate data.[1] A tool providing objective quality scores is therefore essential to ensure that they are directed toward suitable websites.

Syndactyly is one of the most common congenital upper limb anomalies. It is defined as the fusion of one or multiple digits.[8-10] Syndactyly can be classified based on several distinguishing factors. It may be simple or complex based on bone and soft tissue involvement. It may also be complete or incomplete depending on whether the entire length of the digit is involved. Finally, complicated syndactyly refers to a collection of synostoses of abnormal bones. Depending on the complexity of the syndactyly, the treatment, timing of surgery, and additional investigations vary. Although many methods and techniques have been reported, incisional separation of the digits followed by local flaps with or without full thickness skin grafts is the basis of surgical repair.[10] Parents seeking medical care for their child can easily be overwhelmed by the multitude of treatment plans that exist. The primary aim of this study was to evaluate, using trusted search engines, the readability and quality of online materials for digital syndactyly.

MATERIALS AND METHODS

The term “syndactyly” was searched in three separate engines by two independent evaluators (JS & JG) on their personal computers. Location and cookies were disabled using a private browser to minimize bias. Google, Yahoo, and Bing were chosen, as they have been shown to represent 90% of all searches performed on the internet.[5] Only the top 10 websites from each engine were selected and analyzed, as data suggest that 90% of online trafficking is limited to the search engine’s first page.[11] A total of 15 website pages were selected for analysis (Tables 1–2).

Table 1.

Comparison of Mean Readability Grade Levels, FRESs, and Mean Discern Quality Scores of 15 Analyzed Websites

Website	Mean Readability Grade Level	FRES (x/100)	Mean Discern Quality Score (x/80)
Boston Children’s Hospital	12	49	21.5
Cedars Sinai	11	45	22.5
Children’s Hospital of Philadelphia	11	47	20.5
ePainAssist	12	50	35.5
Great Ormond Street Hospital	9	64	52
Great Ormond Street Hospital 2	9	66	61
Kids Health	6	71	21.5
Medical News Today	11	52	37
Nationwide Children’s	9	59	35.5
Orthobullets	28	0	46.5
Seattle Children’s	9	64	28.5
Washington University Orthopedics	10	53	23.5
Washington University Orthopedics FAQ	9	53	34
Wikipedia	11	48	43
Wikipedia 2	13	25	31.5

Table 2.

Included Websites and Corresponding URLs

Website	URL
Boston Children’s Hospital	http://www.childrenshospital.org/conditions-and-treatments/conditions/s/syndactyly/diagnosis-and-treatment
Cedars Sinai	https://www.cedars-sinai.org/programs/plastic-surgery/treatments/hand-surgery/syndactyly.html
Children’s Hospital of Philadelphia	https://www.chop.edu/conditions-diseases/syndactyly
ePainAssist	https://www.epainassist.com/hands/syndactyly
Great Ormond Street Hospital	https://www.gosh.nhs.uk/conditions-and-treatments/conditions-we-treat/syndactyly
Great Ormond Street Hospital 2	file:///Users/yossisaleh/Downloads/Syndactyly%20F0412%20A4%20bw%20FINAL%20Jul12%20(1).pdf
Kids Health	https://kidshealth.org/en/parents/syndactyly.html
Medical News Today	https://www.medicalnewstoday.com/articles/322111
Nationwide Children’s	https://www.nationwidechildrens.org/conditions/syndactyly
Orthobullets	https://www.orthobullets.com/hand/6076/syndactyly
Seattle Children’s	https://www.seattlechildrens.org/conditions/syndactyly/
Washington University Orthopedics	https://www.ortho.wustl.edu/content/Patient-Care/3221/Services/Hand-Microsurgery/Overview/Congenital-Hand-Disorders/Syndactyly.aspx
Washington University Orthopedics FAQ	https://www.ortho.wustl.edu/content/Patient-Care/3222/Services/Hand-Microsurgery/Overview/Congenital-Hand-Disorders/Frequently-asked-questions-about-syndactyly.aspx
Wikipedia	https://en.wikipedia.org/wiki/Syndactyly
Wikipedia 2	https://en.wikipedia.org/wiki/Greig_cephalopolysyndactyly_syndrome

All functional websites appearing in the first 10 search results and containing information related to syndactyly treatment were included for data extraction. Furthermore, any website available within one click of the parent website was analyzed and included. The exclusion criteria included any duplicate website, websites not including information specific to syndactyly, websites for which a subscription or payment was necessary, and websites that were affiliated with paid advertisements.

Readability was assessed using the www.readabilityformulas.com website for all 15 selected pages. The text from each website was inserted into the readability engine and was given scores using the Flesch reading Ease score, the Gunning fog index, the Flesch-Kincaid grade level, the Coleman-Liau index, the Smog index, the automated readability index, and the Linsear Write formula. With these established readability tests, a readability consensus was generated for each website. The Flesch reading ease score (FRES) evaluates texts on a scale from 0 to 100, with higher scores indicating easier comprehension level. Scores of 90–100 are considered an 11-year-old level, whereas scores between 60 and 70 indicate a 13- to 15-year-old reading level. The remaining scores generate a grade level using various predefined formulas (Table 3).[12]

Table 3.

Readability Formulas

Test	Formula
Automated Readability Index	4.71 (characters/words) + 0.5 (words/sentences) − 21.43
Coleman-Liau Index	0.0588L-0.296S-15.8
Flesch-Kincaid Grade Level	(0.39 × ASL) + (11.8 × ASW) − 15.59
Flesch Reading Ease	206.835 – (1.015 × ASL) – (84.6 × ASW)
Gunning’s Fog Index	0.4 (ASL + PHW)
Linsear Write Readability Formula	1) Find 100 word sample 2) ((No. easy words X 1) + (No. hard words X 3)) / total number of sentences 3a) If number is greater than 20 --> Divide by 2 3b) If number is less than or equal to 20 --> Subtract 2 and divided by 2
Smog Index	3 + Square Root of Polysyllable Count
ASL: Average sentence length
ASW: Average syllables per word
PHW: Percentage of hard words
L: Average letters per 100 words
S: Average number of sentences per 100 words

Quality assessment was evaluated by each independent observer using the Discern handbook, which consists of a series of questions designed to allow health consumers and information providers to assess the quality of written information and treatment choices for a given health issue.[13] The evaluators assessed each website and gave a score from one through five for each of the 16 questions in the Discern questionnaire. Then, a mean score for each website was calculated and used. The website’s quality was rated as either excellent, good, fair, poor, or very poor if their scores were between 63–80, 51–62, 39–50, 27–38, and 15–26, respectively.[3,14] A Pearson’s correlation analysis was used to assess agreement between both observers’ quality scores.

RESULTS

Fifteen websites were included for readability and quality assessment. Using the readability consensus, the overall grade level for the 15 websites was 11.3 (Table 1). The average FRES was 49.6, on a scale of 0–100, with 100 being the easiest (Table 1). The average grade level using the Gunning fog index, Flesch-Kincaid grade level, Coleman-Liau index, Smog index, automated readability index, and Linsear Write formula were 14th, 11th, 11th, 10th, 11th, and 13th, respectively (Fig. 1). Using the readability consensus, Kid’s Health was evaluated to have the lowest readability score at a sixth grade level. Orthobullets had the highest score, evaluated at a 28th grade level, corresponding to a college graduate.

Fig. 1.

Grade-level comparison of included websites using six readability metrics.

The mean score for quality assessment, using the Discern criteria, was 36.3 for the first evaluator and 30.2 for the second (Table 1). Quality assessment was then further divided into reliability and treatment sections. The first eight questions pertaining to reliability had an average of 18.3 and 18.1 out of 40 for evaluator one and two, respectively. For the treatment section, which includes seven questions, the average scores were 15.3 and 14.2 out of 35 for evaluator one and two, respectively. A relative percentage was calculated for each section and compared. The relative percentage for the first eight questions was 46% and 45%, and the relative percentage of the second seven questions was 44% and 47% for evaluators one and two, respectively. Of the 15 websites, both observers evaluated five of the websites as being very poor. Observers one and two each evaluated three and six websites as being poor, respectively. Four websites for observer one and two websites for observer two were classified as fair. Finally, three websites for observer one and two websites for observer two were evaluated as good. None of the websites were rated as excellent by either observer. The website with the lowest quality score was Boston Children’s Hospital for evaluator one, which received a score of 19. The website with the highest score from both evaluators was Great Ormond Street Hospital 2, which received a score of 62 and 60 from evaluators one and two, respectively. A Pearson’s correlation analysis between both observers’ quality scores gave a result of 0.93, representing a very high and positive correlation.[15,16] Mean Discern quality scores were plotted against the mean readability grade levels of each website (Fig. 2).

Fig. 2.

Comparison of Discern questionnaire quality scores to mean readability grade levels of each website.

DISCUSSION

Since the development of the internet, the primary source of medical information for patients has shifted from expert opinion to online materials.[1,2] Furthermore, the patriarchal, physician-centered practice of medicine has evolved toward a patient-centered, SDM process.[1,17,18] Access to online information allows an increased participation of patients in the decision-making process, and it is thus imperative that this material be of high quality and comprehensible to the general public.

SDM has been shown to improve the quality of decisions while also decreasing decisional conflict.[1,18] To assure and benefit from SDM, patients must understand the available options and express their preference to make a shared and evidence-based decision.[18] Quality and readability of online materials are key elements for developing these informed preferences. The American Medical Association and the National Institute of Health have recommended a reading level below sixth and eigth grade, respectively.[1] This recommendation is in response to studies demonstrating that a vast majority of Unites States citizens read below an eigth grade level, and that 36% of American adults have basic or inferior to basic literacy skills.[4,5] These low readability levels may ultimately negatively impact SDM and may in turn lead to a worsened overall quality of care, reduce the practice of evidence-based medicine, and increase the overuse or misuse of surgical interventions.[18]

The average grade level for the websites included in this study was 11th grade, far exceeding the current recommended level. Of the 15 websites reviewed, only one website, Kids Health, fell within the sixth to eigth grade range of recommended reading levels. Higher readability grade levels in online materials have also been described in other medical subspecialties. In a study by Basch et al on the readability of materials pertaining to prostate cancer, the average grade levels obtained were between 11.1 and 12.7, using various readability metrics.[7] In a different study assessing readability of colorectal cancer information on the web, the average FRES score was 56.3.[12] A worrisome trend across online medical literature is the worsening readability scores and subsequent decrease in FRES that has been observed over time. One possible explanation is the overuse of scientific jargon without corresponding increase in the quality of the information.[19] Consequently, health care providers must be aware of potential sources that likely surpass the comprehension of the general public and must orient their patients accordingly.

Using the widely validated Discern handbook,[13] the average scores of the websites reviewed in our study were 36.3 and 30.2, respectively. In comparison, a quality and readability analysis study on colorectal cancer had a mean Discern score of 52.2 falling in the good quality range, whereas the results of our study fell within the poor level.[12] The most common information that was lacking in the colorectal cancer study was information regarding complications, risk of treatment, and quality of life information.[3] To further analyze the information that was lacking in online materials for syndactyly, we evaluated the Discern handbook based on its distinct sections. These results show that when partitioning the sections to evaluate the reliability and treatment individually, both were equally poor. Syndactyly presents as a spectrum and as such, there are various surgical techniques used. The goal of surgical correction is to produce a functional and esthetic hand with the fewest number of surgical interventions. Depending on the type of syndactyly, the optimal timing of surgery is debated. In general, release may begin around 6–18 months. The procedure traditionally includes zigzag skin incisions on the volar and dorsal aspects of the hand, followed by separation of the nail or bony connections and the inset of local flaps and full thickness skin grafts. Common complications include hypertrophic scarring and web creep, whereas more serious complications include neurovascular injury.[10] In the included websites, this information was rarely included. Although the aforementioned information is essential to comprehend for informed consent and a SDM process, the reviewed websites were often oversimplified and omitted this critical information.

In a previous study evaluating readability and quality of online materials pertaining to gender affirming surgeries, a trend for higher quality websites being more difficult to read based on FRES was observed.[4] When comparing quality to readability in this study, the website with the highest quality score had an overall readability of ninth grade, representing the second lowest grade level. Therefore, higher readability levels did not seem to positively correlate with higher quality information (Fig. 2). Further studies evaluating the relationship between readability and quality are needed to make concrete conclusions.

Despite being cognizant of the shifting trend toward online access for medical information, the various academic hospitals affiliated with our university offer minimal resources for patient reference. In fact, as plastic surgeons, we are not called upon to provide or edit easily accessible online materials. Studies such as this highlight the importance for surgeons to contribute to their institution’s website to improve the accessibility and quality of information pertaining to their specialty while ensuring it be at an appropriate reading comprehension level for the general public.

Although our study demonstrates the potential shortcomings of online materials related to syndactyly, we must address potential limitations to our quality and readability analysis. Online searches tend to be tailored to the individual based on their browser history and preferences. The evaluators performing the online search attempted to mitigate potential bias using private browsers and disabling location and cookies. Although “syndactyly” might be perceived as too medical for an online search parameter, it is our experience that patients are often familiar with the term. In addition, it was felt that using a multitude of generic terms would overcomplicate the study and detract from the primary aim. Furthermore, standardized readability tests have not yet been established in medical contexts.[20] Information such as diagrams, pictures and other multimedia aids were not included in the readability assessment. As for quality assessment, although the Discern handbook is a well-established quality assessment tool, subjective decisions must occasionally be made. This may explain the slight variability in our data, although a strong coefficient of correlation was obtained between both observers, which reinforces the strength of the Discern tool.

Despite the increase in internet use for medical information, limited quality and readability of these materials continue to be a barrier in optimal patient understanding and subsequent SDM. Online materials pertaining to syndactyly release have shown to be far exceeding the recommended reading level, while also having potentially serious shortcomings in terms of quality of information. Hand surgeons must be aware of the paucity of online resources, and the negative impact that it will have on fostering a patient-centered SDM process.

ACKNOWLEDGMENT

No ethical approval was required for this study.