Evaluation of French-language internet sites dealing with multiple sclerosis.

BACKGROUND: Information available on the internet has changed patient-neurologist relationships. Its evaluation for multiple sclerosis is only partial, regardless of the language used.
OBJECTIVE: We aim to evaluate the content quality and ranking indexes of French-language sites dealing with multiple sclerosis.
METHODS: Two French terms and three search engines were used to identify the sites whose ranking indexes were calculated according to their positions on each page designated by the search engines. Three evaluators used the DISCERN questionnaire to assess the content quality of the 25 selected sites. The sites were classified according to the mean of the evaluators' grades. Grading agreement between evaluators was calculated. Ranking indexes were computed as a rank/100.
RESULTS: Content level was deemed mediocre, with poor referencing of the information provided. The naïve and two expert evaluators' grades differed. Content quality disparity was found within the different website categories, except for institutional sites. No correlation was found between content quality and ranking index.
CONCLUSION: The information available was heterogeneous. Physicians should guide patients in their internet searches for information so that they can benefit from good-quality input which is potentially able to improve their management.

Introduction

New technologies have changed our relationship with information. They have rendered accessible ever larger volumes of more varied content.[1-3] This multitude of information is sometimes difficult to understand, and requires considerable effort to compile, compare and digest before a critical opinion can be formulated. In the absence of control, medical information is no exception to that rule. Attempts to certify medical information by organizations are constrained by the multiplication and rapid, ever-changing sources of information. Exhaustivity and updating of evaluations have proved difficult.[4]

Patients, now more than ever involved in the management of their diseases, frequently turn to information available online. Having access to information about treatments and disease, especially when the disease is chronic, contributes to enforcing disease knowledge, and improves their quality of life.[5-7]

Being given a serious diagnosis and treatment changes are situations that drive patients’ desire for more information.[8] Information needed by patients recently diagnosed with multiple sclerosis (MS) mainly concerns the disease and its management, including the treatment and health resources they could request.[9] Supporting the information provided during a personal consultation with internet content was shown to improve patient knowledge and make decision-making related to the disease easier.[10] Patients with MS considered information found on the internet as useful, but have difficulty assessing the reliability of the information. The physician needs to know the quality of content available online in the language of the country where he/she practises in order to advise their patients in this process.

The evaluation of internet content encompasses several elements, such as quality, the patient's ability to understand and his/her need to be better informed, and its accessibility.[11,12] The DISCERN questionnaire is a tool validated for the evaluation of health-related content quality. At present, establishing a ranking index (RI) is not codified.[13-15]

We analysed the content quality and RIs of French-language internet sites that can be consulted by patients newly diagnosed with MS who want to inform themselves about the different facets of their disease, other than treatment.

Methods

The internet sites to be evaluated were identified with three search engines (Google https://www.google.fr/?gws_rd=ssl, Yahoo https://fr.yahoo.com, Bing https://www.bing.com/?cc=fr) and two French terms, used alternately: “sclérose en plaques” and “SEP”, respectively, multiple sclerosis and MS in French. The search engine site-identification process was done on 9 October 2014 using a computer whose search history had been erased.

Ten sites appeared per page. The first 30 sites without commercial advertisements were retained for each of the six searches. Pharmaceutical industry sites, blogs and forums, sites for MS patients without information on the disease and sites not addressing MS were excluded; repetitions were deleted. The retained sites were classed into six categories: non-proprietary sites constructed with outside contributors (henceforth generic), patient associations, press, regional healthcare network coordinating care (network), learned societies/institutions (institutional) and others.

Each site's RI was calculated (30 sites with 10 per page), taking into account (a) the page number on which the site appeared (1–3), (b) the ordered position of appearance on the page (1–10) and (c) the number of times the site appeared in the six searches: RI = ∑(10(3–/B) × C/6, ranging from 0 to 100, according to an inverse function, taking into consideration internet users’ habits. The sites visited were often limited to the first ones appearing on the first page. Three levels of RI were defined: good: ≤ 100 but > 10; intermediate: ≤ 10 but > 1; and poor: ≤ 1.

Content quality was evaluated by three evaluators: two neurologists specialized in MS and a non-medical “naïve” person, without any particular knowledge of MS.

The DISCERN questionnaire was used to assess content, respecting the grading rules defined by the tool (www.discern.org.uk). This questionnaire is used to evaluate the content of health-related internet sites. It was validated in its totality and for each one of its questions. Indeed, the validation per question enabled us to exclude, without impacting the tool's validity, questions about treatments that were beyond the scope of this study. Nine questions (Q; Q1–Q8 and Q16), each scored with a 5-point Likert scale, enabled us to grade sites with a possible maximum score of 45. Q1 and Q2 address goals; Q3, Q6 and Q8 concern content; Q4 and Q5 deal with referencing the sources of the information reported; Q7 focuses on external sources, that is, where to find additional information; and Q16 targets overall impression. The mean ( ± standard deviation, SD) of the three evaluators’ DISCERN scores was used to class the selected sites, which were then assigned to four quality strata according to: very good: ≥ 36; good: < 36 but ≥ 32; intermediate: < 32 but ≥ 24; or poor: < 24. The distribution of the DISCERN score define the above-mentioned strata: upper 10% of the distribution > 36, first quartile = 32, third quartile = 24.

Statistical analyses

Discrete continuous variables, expressed as median [range], were compared with Kruskal–Wallis tests and post-hoc analysis with the Dunn test. A logistic-regression model was used to determine the probability that content quality was associated with a site category. The Cronbach α-coefficient was calculated to assess between-evaluator concordance, with an α-coefficient > 0.7 defining acceptable agreement. After having excluded outlier values (n = 2) according to the Mahalanobis method, the relationship between continuous variables was calculated with Spearman's correlation coefficient (ρ).

Results

Analysis of the selection of sites

After application of the exclusion criteria, the list of 180 sites without evident commercial advertising was reduced to 112 and, once repeats were eliminated, to 25 sites (Figure 1), Annex 1. The 25 sites were distributed according to their category as follows: 36% generic, 24% association, 16% press, 12% network, 8% institutional and 4% other. Site rankings are reported in Table 1.

Figure 1.

Selection of the internet sites analyzed according to search engine.

Search terms were in French: sclérose en plaques, multiple sclerosis; SEP, MS. Sites excluded were defined as follows: Ads, with commercial advertising; Ind, pharmaceutical industry; Off, off-subject; NID, No Information about Disease, not providing information on the disease itself; Blg, blog.

Table 1.

Sites identified according to search engine and keyword (in French), their individual rankings and global ranking index RI.

Site	Category	Google		Yahoo		Bing		Times	RI
		Sclérose en plaques	SEP	Sclérose en plaques	SEP	Sclérose en plaques	SEP	cited, n	(/100)
wikipedia.fr	Generic	3	1	1	1	1	1	6	88.88
passeportsante.net.fr	Generic	2	7	2	24	2	20	6	27.59
doctissimo.fr	Generic	4	4	3	5	3	6	6	25.55
sclerose-en-plaques.apf.asso.fr	Association	7	8	5	2	5	3	6	25.02
sante-medecine. commentcamarche.net	Generic	10	3	9	7	9	7	6	15.69
afsep.fr	Association	6	6	18	10	19	9	6	9.47
e-sante.fr	Press	8	12	6	–	6	–	4	8.47
inserm.fr	Institutional	5	5		13		15	4	7.55
ligue–sclerose.fr	Association	14	11	14	8	13	10	6	6.57
docteurclic.com	Generic	–	–	8	–	8	–	2	4.17
sindefi.org	Network	29	13	–	–	–	–	2	0.57
sante-guerir.notrefamille.com	Generic	–	–	17	–	15	–	2	0.57
unisep.org	Association	–	–	–	18	–	16	2	0.49
ameli-sante.fr	Institutional	–	–	20	–	18	–	2	0.37
arsep.org	Association	15	–	–	–	–	–	1	0.33
sante.lefigaro.fr	Press	16	–	29	–	–	–	2	0.3
rhone-alpes-sep.org	Network	30	16	–	–	–	–	2	0.3
scleroseenplaques.info	Generic	–	–	21	–	22	–	2	0.25
fr.medipedia.be	Generic	18	–	–	–	–	–	1	0.21
gsep.fr	Network	–	19	–	–	–	–	1	0.18
mssociety.ca/fr	Association	19	–	–	–	–	–	1	0.18
futura-sciences.com	Press	22	–	–	–	–	–	1	0.08
carenity.com	Generic	25	26	–	–	–	–	2	0.06
vidal.fr	Press	–	–	28	–	27	–	2	0.04
mscenter.be/fr	Other	24	–	–	–	–	–	1	0.04

See Materials and methods for definitions of site categories.

SEP: sclérose en plaques.

DISCERN questionnaire-assessed content quality

Concordance of the grades given by the three evaluators to the sites was good (α = 0.73), as was the agreement between expert evaluators (α = 0.81). In contrast, concordance between the naïve evaluator and the experts individually was poor (α = 0.59 and 0.46). Concordance of the grades evaluators’ gave to the different questions was good (α = 0.85), regardless of the evaluator.

The content quality of the French-language sites dealing with MS was: 28.67 [15.33–38.33]. Site classification according to the global DISCERN questionnaire grade and the detailed grades given by the evaluators are given in Table 2 and compared in Figure 2. Q4 and Q5, concerning the quality of referencing the information provided, obtained the lowest grades (p < 0.001) (Figure 3).

Table 2.

Evaluators’ total and individual DISCERN-question grades (means ± SD).

Site name	Category	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q16	MQ ± SD	SQ/45
wikipedia.fr	Generic	5.00 ± 0.00	3.87 ± 1.73	1.56 ± 1.51	5.00 ± 0.00	5.00 ± 0.00	3.33 ± 0.58	4.33 ± 0.58	4.00 ± 0.00	4 .00 ± 1.00	4.26 ± 0.76	38.33
sclerose-en-plaques. apf.asso.fr	Association	4.33 ± 1.15	3.94 ± 1.32	4.33 ± 1.29	4.00 ± 1.00	4 .00 ± 1.00	4.00 ± 0.00	4.33 ± 0.58	3.67 ± 0.58	4.33 ± 0.58	4.15 ± 0.66	37.35
inserm.fr	Institutional	4.33 ± 1.15	3.94 ± 1.32	4.00 ± 1.31	3.67 ± 1.15	3.00 ± 2.00	4.00 ± 0.00	4.33 ± 0.58	4.67 ± 0.58	4.67 ± 0.58	4.11 ± 0.97	36.99
ameli-sante.fr	Institutional	4.67 ± 0.58	3.65 ± 1.47	3.67 ± 1.42	4.33 ± 0.58	4.67 ± 0.58	3.67 ± 1.15	3.67 ± 1.15	3.00 ± 0.00	3.67 ± 0.58	3.85 ± 0.91	34.65
passeportsante.net.fr	Generic	4.33 ± 0.58	3.61 ± 1.34	4.33 ± 1.29	3.00 ± 1.73	3.00 ± 1.73	4.00 ± 0.00	3.67 ± 1.53	4.33 ± 1.15	4.00 ± 0.00	3.81 ± 1.04	34.29
mssociety.ca/fr	Institutional	4.33 ± 0.58	3.99 ± 1.45	4.33 ± 1.42	1.67 ± 0.58	1.67 ± 1.15	4.33 ± 0.58	4.00 ± 1.00	4.00 ± 1.00	4.33 ± 0.58	3.70 ± 1.32	33.30
vidal.fr	Press	4.67 ± 0.58	3.90 ± 1.52	3.00 ± 1.42	4.00 ± 1.00	3.33 ± 1.15	4.00 ± 1.00	2.67 ± 0.58	2.33 ± 1.15	4.00 ± 1.00	3.56 ± 1.09	32.04
ligue-sclerose.fr	Association	4.67 ± 0.58	4.03 ± 1.47	4.00 ± 1.39	2.00 ± 1.73	1.33 ± 0.58	4.00 ± 1.00	3.33 ± 1.53	3.33 ± 2.08	4.33 ± 0.58	3.48 ± 1.48	31.32
doctissimo.fr	Generic	4.67 ± 0.58	3.78 ± 1.45	4.00 ± 1.29	2.00 ± 1.00	1.33 ± 0.58	4.00 ± 1.00	4.33 ± 0.58	2.67 ± 1.53	4.00 ± 1.00	3.41 ± 1.31	30.69
sante-medecine. commentcamarche. net	Generic	4.33 ± 0.58	3.74 ± 1.32	3.33 ± 1.23	2.33 ± 1.15	1.33 ± 0.58	4.00 ± 1.00	4.00 ± 1.00	3.33 ± 1.15	3.67 ± 0.58	3.37 ± 1.15	30.33
carenity.com	Generic	4.33 ± 1.15	3.81 ± 1.36	3.67 ± 1.32	3.33 ± 1.15	3.33 ± 1.53	3.33 ± 1.53	1.33 ± 0.58	3.00 ± 1.73	3.67 ± 1.53	3.33 ± 1.36	29.97
arsep.org	Association	4.67 ± 0.58	3.53 ± 1.58	3.00 ± 1.39	2.00 ± 1.00	2.67 ± 2.08	3.67 ± 0.58	4.33 ± 0.58	3.00 ± 0.00	3.33 ± 0.58	3.30 ± 1.14	29.70
unisep.org	Association	4.67 ± 0.58	3.78 ± 1.64	4.00 ± 1.39	1.67 ± 1.15	1.00 ± 0.00	4.00 ± 1.00	2.67 ± 1.15	3.33 ± 1.15	3.67 ± 0.58	3.19 ± 1.39	28.71
afsep.fr	Association	5.00 ± 0.00	3.87 ± 1.73	4.33 ± 1.49	1.00 ± 0.00	1.00 ± 0.00	4.00 ± 1.00	1.33 ± 0.58	3.67 ± 0.58	4.33 ± 1.15	3.15 ± 1.61	28.35
fr.medipedia.be	Generic	4.33 ± 0.58	3.74 ± 1.43	4.33 ± 1.33	1.00 ± 0.00	1.33 ± 0.58	4.00 ± 0.00	2.33 ± 1.53	2.33 ± 1.15	3.67 ± 0.58	3.04 ± 1.43	27.36
docteurclic.com	Generic	4.67 ± 0.58	3.65 ± 1.48	3.67 ± 1.30	1.00 ± 0.00	1.00 ± 0.00	3.33 ± 0.58	3.00 ± 1.00	2.67 ± 0.58	3.33 ± 0.58	2.89 ± 1.25	26.01
rhone-alpes-sep.org	Network	4.33 ± 0.58	3.61 ± 1.45	3.33 ± 1.28	1.00 ± 0.00	1.00 ± 0.00	3.00 ± 0.00	3.00 ± 1.00	2.67 ± 1.53	3.00 ± 0.00	2.78 ± 1.25	25.02
e-sante.fr	Press	3.67 ± 1.15	2.98 ± 1.12	3.33 ± 1.05	2.33 ± 1.53	2.33 ± 1.53	3.00 ± 0.00	2.33 ± 0.58	2.33 ± 1.15	2.67 ± 0.58	2.74 ± 0.94	24.66
sindefi.org	Network	3.00 ± 1.00	3.00 ± 1.07	4.00 ± 1.10	1.67 ± 1.15	1.33 ± 0.58	3.33 ± 0.58	2.00 ± 1.00	2.33 ± 1.53	3.33 ± 0.58	2.74 ± 1.16	24.66
futura-sciences.com	Press	3.67 ± 1.53	3.02 ± 1.29	2.67 ± 1.19	2.33 ± 2.31	1.00 ± 0.00	3.33 ± 0.58	3.33 ± 2.08	2.33 ± 0.58	2.67 ± 1.15	2.67 ± 1.36	24.03
gsep.fr	Network	4.00 ± 1.00	3.12 ± 1.36	3.00 ± 1.21	1.00 ± 0.00	1.33 ± 0.58	3.33 ± 0.58	3.33 ± 1.53	2.00 ± 1.00	2.67 ± 0.58	2.59 ± 1.22	23.31
sante.lefigaro.fr	Press	4.67 ± 0.58	3.40 ± 1.57	3.00 ± 1.31	1.33 ± 0.58	1.00 ± 0.00	3.3 3 ± 0.58	1.33 ± 0.58	2.67 ± 0.58	3.00 ± 0.00	2.56 ± 1.19	23.31
mscenter.be/fr	Other	3.67 ± 0.58	3.03 ± 1.09	3.00 ± 0.99	1.00 ± 0.00	1.00 ± 0.00	3.00 ± 0.00	2.00 ± 0.00	2.67 ± 0.58	2.67 ± 0.58	2.44 ± 0.93	21.96
sante-guerir. notrefamille.com	Generic	2.67 ± 0.58	2.53 ± 1.02	2.33 ± 1.00	1.67 ± 1.15	1.67 ± 1.15	2.33 ± 1.53	2.33 ± 1.53	2.67 ± 0.58	2.33 ± 0.58	2.33 ± 1.00	20.97
scleroseenplaques.info	Generic	3.00 ± 1.00	2.37 ± 1.06	1.33 ± 1.00	1.67 ± 1.15	1.67 ± 1.15	1.67 ± 1.15	1.33 ± 0.58	1.00 ± 0.00	1.67 ± 1.15	1.70 ± 0.95	15.33

Q: questions from the DISCERN questionnaire; MQ ± SD: mean and standard deviation of DISCERN questions; SQ/45: DISCERN score as the Σ Q. See Materials and methods for definitions of site categories.

Figure 2.

DISCERN questionnaire grades given by the three evaluators for each of the 25 Internet sites.

All the evaluators’ grades given for the nine questions are represented as whisker plots. The internet sites are classed in the order (left to right) of increasing quality content. Each evaluator's (naïve evaluator: —; the experts – – and – – –) median grade is indicated as a circle within the box; the lower and upper limits of the rectangle are the 25th–75th interquartile range; the T-bars correspond to range.

Figure 3.

The three evaluators’ grades given for each of the nine DISCERN questions for the 25 internet sites are represented as whisker plots.

Each evaluator's (naïve evaluator: —; the experts – – and - - -) median grade is indicated as a circle within the box; the lower and upper limits of the rectangle are the 25th–75th interquartile range; the T-bars correspond to range.

Three sites (generic, association or institutional; one each) were considered to have very good quality contents (Figure 4). Globally, institutional sites were of very good and good quality. Association sites had very good to intermediate quality information, while generic sites were of unequal quality, ranging from very good to poor quality. Press sites had good to poor quality contents. Network and other sites were of intermediate and poor quality. The probability of finding quality information for a certain site category was only significant for institutional sites (p = 0.042).

Figure 4.

Mean DISCERN scores, for each site category.

Each plot corresponds to one selected site.

Site categories are represented on the x-axis, and the mean DISCERN scores, divided in four quality strata, on the y-axis.

RIs and content quality

The order of site appearance on each internet page is reported for each of the search engines and the two keywords in Table 1. The potential relationship(s) between RI and site category was analysed. The disparity of RIs within categories contributed to the absence of significant RI differences between categories: generic 4.16 [0.06–88.88], association 3.53 [0.18–25.01], institutional 3.96 [0.37–7.56], press 0.19 [0.04–8.47], network 0.3 [0.18–0.57] and other 0.04 (Kruskal–Wallis test, p = 0.28). No correlation could be established between the content quality and RI (ρ = 0.38; p = 0.08).

Discussion

Evaluation of the information about MS provided by internet sites revealed heterogeneous content quality and poor referencing of the information sources. The French-language Wikipedia site arrived at the head of the class for quality and RI; however, other generic sites exhibited broad quality and RI heterogeneity. The institutional, and association sites also had very good quality contents.

Validity of the quality evaluation using the DISCERN questionnaire was confirmed by the good concordance of the grades among evaluators, despite the withdrawal of questions concerning treatments. The complexity of the site contents and their comprehension by patients were not specifically addressed herein. However, that the non-medical naïve evaluator was not knowledgeable about MS enabled us to partially integrate this component into our analysis. Finally, patient accessibility to the information was assessed with the RIs and the site-selection process. Other parameters, such as site-visit indexes and evaluation of the rapidity of content changes, were not examined. The difficulties inherent in grading internet sites and the absence of defined methodology contributed to these limitations.

The quality of French-language sites dealing with MS was about average with that of other health-related sites.[12,16] A study evaluating the quality of English-language sites linked to MS also found content-quality heterogeneity ranging from good to poor.[17]

The absence of information-referencing contributed to the mediocre quality of the French-language sites. The difference between Wikipedia and some other association or institutional sites depended in large part on this parameter. If we accept that the value of the information can only be assessed when its origin is known, evaluating this parameter differs between experts and naïve readers. Indeed, mastery of the subject matter is needed to judge the pertinence of the references offered and, thus, the validity of the information provided, as confirmed by the divergent grades our experts and naïve evaluator gave referencing quality. Confidence accorded to the information provided is often subjective, and depends, in part, on the reader's habits and experience, independently of all referencing.[15] Moreover, some sites might benefit from quality labels, independently of the information-referencing that they provide, and our results fully supported that premise for institutional sites and some association sites.

RIs are rarely investigated, even though they determine the information that is consulted. Notably, their examination is constrained by a lack of validated tools, performance variability from one search engine to another and notable lability. We used RI as an indicator of accessibility of the information available on the internet. Evaluation of site-visit frequency provided by certain companies, such as Media Metrix, associated with the socioeconomic profile of individuals who visit them would be informative to analyse and compare with RIs.

The absence of correlation between the content quality and RIs highlights the difficulties that patients face. The attempts of health authorities to control internet content have proved illusory since the 1990s. A strategy of content certification would be difficult, if not impossible, to enforce in light of the rapidity of perpetually changing information and multiplication of sites.

The advice provided by treating physicians is essential in guiding access to quality information. It seems important that neurologists guide their patients in their searches for complementary information. The situations which lead patients to search the web have been clearly identified: being given a serious diagnosis, medication choices and changes, and management of a disease symptom;[7,17] these are easily recognized by the physician treating the disease. Advising patients to visit a panel of sites, where they will find quality information, seems necessary in the practice of modern medicine.[18] Healthcare networks involved with MS also have a role to play in the dissemination and popularization of knowledge. Their sites should offer quality information on which patients can rely. This usage can only improve patient management. We hope that our findings provide useful information for healthcare providers involved in managing patients with MS.