Quality assessment of patient information on orthognathic surgery on the internet.

OBJECTIVE: The aim of this study was to assess the quality of online information for patients on orthognathic surgery.
MATERIALS AND METHODS: A selection of search terms specific for orthognathic surgery was chosen and 150 websites were identified using the Internet search engines Google, Yahoo and Bing. Irrelevant websites were excluded. The remaining websites were assessed with a modified Ensuring Quality Information for Patients (EQIP) tool. EQIP evaluates the quality of medical patient information by measuring the three key aspects of content, structure, and identification data.
RESULTS: 48 relevant websites were identified. EQIP values ranged between 2 and 28 (median 13.65). While 37 of the 48 websites described details of the surgical procedures, only 13 mentioned possible risks and complications of the surgery. No differences were found between the websites of private practices, dentists and public hospitals, universities, or others (p = 0.66). Websites found by Google had a significantly lower EQIP score compared with Yahoo and Bing (11.12 vs. 16.60 for Yahoo and 16.23 for Bing; p = 0.012). The better the rank of the website, the higher the EQIP score (r = -0.411, p = 0.004).
CONCLUSIONS: The results of this study reflected a large variation of quality of information on orthognathic surgery on the Internet. Therefore, surgeons must be aware that they might be confronted with unrealistic expectations of patients, who may underestimate the potential risks and drawbacks of orthognathic surgery.

Introduction

With the world wide web now widely available, the Internet has become the first-choice source of information, with widespread and growing use of written online information, online videos, Internet discussion groups, and blogs. Increasingly, patients use the Internet to obtain answers to medical questions. According to a recent poll, more than 50% of people consult the Internet regularly to get information on health issues (Hambrock, 2018). It can be assumed that during the COVID-19 pandemic, this number is now significantly higher (Zimmermann and Nkenke, 2020).

Around 60% of the people gather knowledge on the Internet before or after they visit a medical specialist. Approximately 50% of these people believe that they understand the health information they have sourced and are able to rate the quality of the data. However, patients rating the quality and reliability of online information seems to be an arbitrary process. Patients tend to stick to information that confirms their preexisting opinions, prejudices, and fears. Information is preferred that supports aspects that patients have heard or read before (Hambrock, 2018).

On average, people remain on a website for only 69 s. This short interval of time affects their health literacy (Eysenbach et al., 2002). Health literacy is defined as an individual's ability to read, understand, and apply information to make rational health-care decisions and follow treatment instructions (Lee et al., 2019). Studies have shown that insufficient health literacy has an impact on patients' autonomy and ability to make sound decisions regarding their health. The consequence can be a poorer postoperative outcome (Menendez et al., 2017, Halleberg Nyman et al., 2018, Keim-Malpass et al., 2018).

Online health information sources are preferred especially by females, younger individuals, and people with a higher socio-economic status (Zschorlich et al., 2015, Hambrock, 2018). People with a limited scientific background almost always carry out Internet-based searches using one of the major search engines. The dominant leader is Google, with a market share exceeding 92% (Statcounter, 1999–2017).

When patients think about undergoing orthognathic surgery, it is most likely that they will use the Internet to collect additional information and to receive support and reassurance (Bhamrah et al., 2015). For the patients it is most important to gather high-quality information before surgery, because there is a clear correlation with patient satisfaction after surgery (Bailey et al., 1999, Kufta et al., 2016). Aside from online research and the individual's overall expectations, of course the information provided by the doctor during a personal consultation has an important influence on patient satisfaction (Al Kharafi et al., 2014). In order to fully understand patients' motivations and expectations concerning orthognathic surgery, doctors have to be familiar with the information offered on the Internet (Lee et al., 2019). Therefore, the aim of this study was to assess the quality of websites on the Internet that provide information on orthognathic surgery.

Materials and Methods

An Internet search was performed in January 2019 using the three most commonly used search engines — Google, Yahoo, and Bing (Statcounter, 1999–2017). The search was performed by the authors J.E. and C.F. The German terms for orthognathic surgery, operation for mandibular and maxillary advancement and set-back, surgical correction of the jaw. and correction of malocclusion were used as search terms, because these were the most commonly used terms when patients filled in the medical questionnaires during consultation for orthognathic surgery.

In order to guarantee identical search results, the search was performed simultaneously on two personal computers using the same local network and after deleting the cache files. The search was conducted with the latest version of Google Chrome (v. 71.0.3578), because Google Chrome is the most commonly used Internet browser (Statcounter, 1999–2017).

The first 50 results on the search engine results page for each search engine were included in this study in order to achieve adequate statistical power (Aldairy et al., 2012, Karamitros and Kitsos, 2018). Websites not related to orthognathic surgery, duplicates, and websites in languages other than German were excluded. If a website was found by at least two of the three search engines, it was selected for further analysis. The website was attributed to the search engine where it reached the best rank on the search engine results page and was excluded from analysis for the remaining two search engines.

The two investigators assessed all websites separately, with any remaining conflicts being discussed and resolved by consensus.

Further analysis of the selected websites was performed with a modified Ensuring Quality Information for Patients (EQIP) tool (Charvet-Berard et al., 2008; Table 1 ).

Table 1

Modified Ensuring Quality Information for Patients (EQIP) tool.

Item	Criteria	Yes	No
1	Initial definition of which subjects will be covered	20	28
2	Coverage of the previously defined subjects	24	24
3	Description of the medical problem	27	21
4	Definition of the purpose of the surgical intervention	32	16
5	Description of treatment alternatives	5	43
6	Description of the sequence of the surgical procedure pre operation	31	17
7	Description of the sequence of the surgical procedure during operation	26	22
8	Description of the sequence of the surgical procedure post operation	28	20
9	Description of the qualitative benefits to the recipient	30	18
10	Description of the quantitative benefits to the recipient	1	47
11	Description of the qualitative risks and side effects	13	35
12	Description of the quantitative risks and side effects	5	43
13	Addressing quality-of-life issues	18	30
14	Description of how complications are handled	21	27
15	Description of the precautions that the patient may take	9	39
16	Mention of alert signs that the patient may detect	4	44
17	Addressing medical intervention costs and insurance issues	13	35
18	Specific contact details for hospital service	22	26
19	Specific details of other sources of reliable information/support	18	30
20	Date of issue or revision	13	35
21	Logo of the issuing body	43	5
22	Names of the persons or entities that produce the document	21	27
23	Names of the persons or entities that financed the document	2	46
24	Short bibliography of the evidence-based data used in the document	1	47
25	Statement about whether or how patients are involved/consulted in the document's production	9	39
26	Use of everyday language and explanation of complex words or jargon	22	26
27	Use of short sentence (<15 words on average)	16	32
28	Personal address to the reader	27	21
29	Respectful tone	41	7
30	Clear information (no ambiguities or contradictions)	34	14
31	Balanced information on risks and benefits	7	41
32	Presentation of information in a logical order	29	19
33	Satisfactory design and layout (excluding figures and graphs)	17	31
34	Clear and relevant figures and a graphs	11	37
35	Inclusion of an named space for the reader's note or questions	15	33

The item ‘coverage of all relevant issues for the topic’ was removed from the EQIP tool because none of the selected websites fulfilled this criterion. The items ‘use of generic names for all medications or products’ and ‘printed consent form contrary to recommendations’ were not relevant to the study and were also removed. Because of its high relevance for orthognathic surgery, the item ‘description of the sequence of the surgical procedure’ was subdivided into preoperative, intraoperative, and postoperative procedures. The answer ‘partially yes’ has been identified previously as being unreliable and, therefore, was removed from the scoring system (Ademiluyi et al., 2003). All items answered with ‘yes’ were scored equally with 1 point and summed up at the end. A maximum of 35 points for the EQIP score was attainable.

For statistical analysis IBM® SPSS® Statistics v.25 was used. Descriptive statistics, Student's t-test, ANOVA and Pearson's correlation coefficient (PCC) were performed. The level for statistical significance was set at p < 0.05, with all tests two-sided.

Results

Out of the initially 150 identified websites, 55 were excluded due to duplicate retrievals and another 47 were excluded due to irrelevant content. The remaining 48 websites underwent further analysis (Fig. 1 ).

Fig. 1

Flow chart for the selection of websites.

Indications or reasons for the need for orthognathic surgery were given on 32 out of the 48 websites. Planning and sequencing of at least one surgical procedure were included on 37 websites. Risks or side effects were mentioned on only 13 websites. Links and sources for further information were offered on 18 websites. Examples of treated patients were given only on nine websites.

Out of the 48 websites, 22 were provided by private practices or dentists, 13 by public hospitals or universities, 10 by patient blogs, two by online encyclopedias, and one by an online newspaper. For further statistical analysis the latter three were pooled in one group as ‘other online information sources’. 42 websites had their origin in Germany, five websites in Austria, and one in the Czech Republic.

The median EQIP score was 13.65, with a standard deviation of 6.289. The lowest score in the survey was 2, and the maximum score was 28 (Fig. 2 ). The website with the highest EQIP score in our study was www.professor-lindorf.de. This was found by Bing at rank 31.

Fig. 2

Modified Ensuring Quality Information for Patients (EQIP) score frequency. The median EQIP score was 13.65, with a standard deviation of 6.289. The lowest score in the survey was 2 and the maximum score was 28.

Relation to a private practice, hospital, blog, etc. was not correlated with the quality of the website. EQIP scores for the three groups ‘private practices’ (12.77 ± 7.45), ‘public hospitals or universities’ (14.69 ± 4.90), and ‘other online information sources’ (14.08 ± 5.56) showed no statistically significant difference (p = 0.66; Fig. 3 ).

Fig. 3

EQIP scores for private practices (12.7 ± 7.4), public hospitals or universities (14.6 ± 4.9), and other online information sources (14.0 ± 5.5); p = 0.66.

Differences in quality at the search engine level were found. Websites ranked higher on the search engine results page for Google showed a significantly lower mean EQIP score (11.12 ± 6.19) than those identified by Yahoo (16.60 ± 5.71) and Bing (16.23 ± 5.08; p = 0.012).

A significant correlation between the rank on a search engine results page and EQIP score was found. The better a website was ranked on a search engine results page, the higher was the EQIP score (r = −0.411, p = 0.004; Fig. 4 ). The negative sign refers to the fact that a better rank means a lower numeric value.

Fig. 4

Correlation between EQIP score and ranking for the search engine results page (r = −0.411; p = 0.004).

Discussion

This study was the first one to assess online patient information relating to orthognathic surgery using a modified EQIP tool.

The EQIP tool is comparable to the British Medical Association patient information award appraisal form for some criteria, and includes the three key aspects: content, structure, and identification data. The EQIP tool has adequate interrater reliability (Charvet-Berard et al., 2008, Nicholls et al., 2009, Vaona et al., 2011, Melloul et al., 2012, Frueh et al., 2015, Haymes, 2016, Palma et al., 2016, Zuk et al., 2016, Zuk et al., 2017, Karamitros et al., 2017, Raptis et al., 2019). It is often preferred over the DISCERN tool because the latter focuses more on readability than on quality of information (Hargrave et al., 2006). Nevertheless, different working groups have tried to rate the quality of websites on orthognathic surgery based on readability, and therefore used the DISCERN scoring system (Aldairy et al., 2012, Lee et al., 2019). They found Wikipedia to be the source of highest quality, with DISCERN scores of 64 (Aldairy) and 63 (Lee). Interestingly, Wikipedia achieved an EQIP score of 13 in our study, placing it very close to the average score (13.6 ± 6.2). This result supports the assumption that DISCERN is more about readability than scientific evaluation of the content and the quality of information. As a consequence, there is the risk that the lay reader may attribute quality to a well-written web page and thus might be misled.

Previously, the EQIP tool has been applied for the evaluation of online information about the treatment of cleft lip and palate malformations (Karamitros et al., 2018). The mean EQIP score was 19.1 ± 5.4, which is higher than that found in our study for orthognathic surgery (13.6 ± 6.2). Considering the fact that a number of EQIP tool items were removed from the questionnaire because they had no relevance for our study, one might have expected us to find a higher EQIP score. It seems that the quality of online information on orthognathic surgery needs significant improvement before these websites can compete with those on the treatment of cleft lip and palate malformations.

The information provided on the different websites was correct from a medical point of view. However, most of the websites provided only basic information on orthognathic procedures before focusing on fixing an appointment for a face-to-face consultation. For every candidate for orthognathic surgery an individual treatment plan has to be put together depending on the anatomical situation, esthetic demands, and the surgeon's preferences (Thiele et al., 2016). Therefore, it is clearly difficult to set up a website that covers all aspects of orthognathic surgery comprehensively. We assume that a lack of in-depth information possibly led to the low mean EQIP score. Nevertheless, it was surprising to see that only a few websites included in the study described the most common operations, such as Le Fort I osteotomy, bilateral sagittal split osteotomy, and genioplasty, in detail. Obviously, adding this information would immediately increase the quality of the websites.

Patients who had undergone orthognathic surgery generally were satisfied and felt well informed (Al Kharafi et al., 2014). However, they considered information regarding surgical risks and functional limitations following surgery insufficient. Websites of public hospitals and universities were more likely to provide relevant information about the risks and disadvantages of different procedures in cleft lip and palate surgery compared with websites of private practices (Karamitros et al., 2018). However, our study found no significant differences between websites of public hospitals or universities and private practices. Indeed, the website of highest quality came from a private practice. Only 13 websites out of 48 provided information on possible risks or side-effects of orthognathic surgery, although according to the current literature complications occur in approximately 20% of patients (Zaroni et al., 2019).

Regarding search engines, this study can refute the hypothesis that ‘the bigger, the better’. Google has a market share of 92.4%, while Yahoo (2.1%) and Bing (2.4%) are comparatively minor companies (Statcounter, 1999–2017). Although Google was able to identify most of the relevant websites that were included in our study — n = 25 (52%) vs Yahoo with n = 13 (27%) and Bing with n = 10 (21%) — Google produced a significantly lower mean EQIP score. Unfortunately, the design of our study did not enable us to find explanations for why websites identified by Google were of reduced quality.

A limitation of our study was that the number of websites that underwent statistical analysis was low. There are different opinions on how many websites people visit on a search engine results page before they start a new search. Recent studies have shown that it is very unlikely that patients will open more than the first 20 websites on a search engine results page (Eysenbach and Kohler, 2002, Aldairy et al., 2012, Karamitros and Kitsos, 2018). Therefore, increasing the number of websites in this study might strengthen the statistical power, but would change the design of the study to a less realistic scenario. As a consequence, increasing the included websites was not taken into account.

Conclusions

This study showed a pronounced variation in the level of quality of the analyzed websites. Therefore, surgeons must be aware that they might be confronted with unrealistic expectations of patients, who may underestimate the potential risks and drawbacks of orthognathic surgery.

Ethical approval and consent to participate

Not applicable.

Declarations of interest

The authors have declared no conflicts of interest.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.