The Quality of EBM Sources Perceived By Belgian Family Physicians.

BACKGROUND: Belgian family physicians use several local and international sources for evidence-based medicine (EBM). AIM: This study aims to investigate the quality of these EBM sources according to the Belgian family physicians.
METHODS: A sample of Belgian family physicians completed a digital survey on the quality of EBM sources.
RESULTS: Respondents evaluated the quality of the information for the major part of the local and international EBM sources good to excellent. More than 50% of the respondents found in the major part of the sources an answer to the question. More than half of the respondents found the necessary information in less than 5 minutes in most of the sources. Younger participants self-evaluated their search skills better than older participants.
CONCLUSION: The quality of most frequently used EBM sources in Belgium is evaluated as good and client-friendly. More than half of the respondents found an answer to their questions in most of the sources and this within 5 minutes.

Introduction

Evidence-based medicine (EBM) has grown to one of the most important aspects of clinical decision making during the past decades. EBM is defined in 1996 by Sackett as: “Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [1].

The quality of international sources seems to be guaranteed by international procedures in the development of guidelines and other EBM sources. Thereupon we can assume that the widespread usage of EBM forms an external control for these sources. Besides the international sources, many countries develop local guidelines and other EBM sources for which the quality control is less obvious.

Belgian scientific organisations also developed specific local EBM sources which are adapted to the Belgium healthcare system. Minerva publishes reviews of recent international papers [2]. The Belgian Centre for Pharmacotherapeutic Information (BCFI) provides EBM information on drugs and their indications. They publish the “Commented Drugs Repertory”, the “Folia Pharmacotherapeutica” and the “Transparency sheets” [3]. Recommendations for good practice are published by the French-speaking Scientific Society for General Practice guidelines (SSMG) [4] as well as by their Dutch-speaking counterpart Domus Medica [5].

The Belgian Centre for Evidence-Based Medicine (CEBAM) [6] makes among others two EBM sources available to Belgian healthcare workers: the Digital Library for Health (CDLH) and the Evidence Linker [7]. The Evidence Linker is integrated into the Electronic Medical Records and provides EBM sources based on the clinical complaint or diagnosis. EBMPracticeNet is another portal for easy access to EBM sources [8]. Several other EBM sources exist such as the “Belgian Guide for anti-infectious treatment in the ambulatory practice” (BAPCOC) [9], Guidelines for the use of medical imaging from the Belgian Medical Imaging Platform (BELMIP) [10], the consensus papers from the National Institute for Health and Disability Insurance (RIZIV) [11], a database on medication before and during pregnancy and breastfeeding (Cybele) [12], the project Farmaka. Be [13] providing EBM information to physicians on medication, the “Formularium for the Homes for the Elderly” [14], a website reviewing medical information from the lay press “Gezondheid en Wetenschap” [15], thematic advice from the “Superior Health Council” [16] and the Belgian Healthcare Knowledge Centre (KCE) [17].

Two Dutch EBM sources are widely used in Belgium: the “Dutch College of General Practice guidelines (NHG) [18] and the guidelines from the “Dutch Institute for Healthcare Improvement (CBO)” [19].

The abundance of local and international EBM sources guarantees their quality neither their user-friendliness nor their user-satisfaction. Although EBM is considered by most physicians as positive, there are some aspects that might hamper its use and implementation. Difficulties in finding the right source may cause long search times and consequently loss of time which cannot be remunerated for the physician [20]. This could undermine the future use of EBM or at least of specific sources. The lack of performant search skills is one of the most important barriers to use EBM [21]. Several studies have investigated the access and barriers to EBM in Belgium and several other countries [21] [22] [23] [24] [25] [26]. However, the perceived quality, the user-friendliness and the user-satisfaction have hardly been studied in Belgium.

This study aims to find out (1) how much time was spent to find the source, (2) how the sources were evaluated regarding the quality of the information and (3) how satisfied the physicians were with these sources.

Methods

In Belgium, the government and the Board of Physicians have the (email) addresses of all physicians but for privacy reasons, these addresses can not be used for research and consequently it is difficult or almost impossible to invite all physicians to participate in a survey. We tried to short-circuit this by sending an email to all Dutch-speaking local organisations of family physicians (n = 54) and by asking them to forward our email with an invitation to participate in our study.

Our department of family medicine and chronic care has another 377 email addresses of family physicians who participated previously in training our research and who were willing to receive new invitations. They received an email to participate in the study. Invitations were sent in December 2015, and a reminder was sent in February 2016. Data entry was stopped on February 22, 2016.

It is not known how many local associations forwarded the email neither do we know how many physicians received the email. For these reasons, we can not estimate the response rate.

The study was limited to Dutch-speaking family physicians living in Belgium. They needed a computer to answer the questions. Physicians without a computer, physicians who did not comprehend Dutch or who were not family physicians, were excluded. Surveys who were completed partly were excluded from the analyses.

An online survey containing 27 questions was build after analysing the literature. It enquired about demographics, general questions about EBM and questions about the personal appraisal on EBM sources. A list with commonly used (EBM) sources was developed including the international and Belgian sources. For each source the questionnaire asked about the perceived quality, the time used to find the information and the user-satisfaction.

During the survey, only information related to physicians is gathered, and no medical information from patients neither physicians was registered. For that reason, permission from an ethics committee was not necessary. The anonymity of the participants was guaranteed because no personal identifying characteristics were recorded.

The collected data were stored in a protected database in LimeSurvey that also generated the descriptive statistics. An SPSS file was extracted from LimeSurvey permitting the analyses in IBM-SPSS-v23. T-tests and ANOVA-tests were used for univariate analyses of continuous variables with a normal distribution and Mann-Whitney-Wilcoxon-tests and Kruskal-Wallis for continuous variables without a normal distribution. Chi-square tests were used for categorical variables.

Results

In total 77 men (54%) and 66 women (46%) fully completed the survey (Table 1). Forty other participants did not answer all questions and were excluded. The mean age of the participants was 45 years (SD 14) ranging between 25 and 71 years. The mean age of men was 52.9 years and of women 35.8 years with men significantly older (P < 0.001) (Table 2). The mean time of activity as a family physician was 18.4 years (SD 14). A correlation of 0.953 was observed between age and the years of activity.

Table 1

Demographics of the participants (n = 143)

	N	%
Gender
Men	77	53.8
Women	66	46.2
Type of practice
Solo practice	43	30.1
Duo practice	25	17.5
Group practice	75	52.4
Practice as part of a network
Yes	36	25.2
No	94	65.7
Unknown	13	9.1
Place of practice
Urban	57	39.9
Semi-rural	56	39.2
Rural	30	20.9
Type of family physician
Trainee	15	10.5
Involved in Master after Master programme	27	18.9
Trainee supervisor	53	37.1
None of the above mentioned	66	46.2
The receiver of telematics premium by the government
Yes
No	136	95.1
Unknown	4	2.8
	3	2.1
Usage of technology on a visit
Laptop	34	23.8
Tablet	15	10.5
Smartphone	35	24.5
None of the above mentioned	67	46.9
Not applicable (no visits)	5	3.5
Access to a wireless Internet connection, when using technology
Yes
No	57	80.3
Unknown	13	18.3
	1	1.4

Table 2

Age and time active in family practice for the participants

	Men (N = 77)	Women (N = 66)	P	All (N = 143)
Age
Mean (SD)	52.9 (10.9)	35.8 (11.5)	< 0.001	45.0 (14.0)
Median	56	30		48
Min	26	25		25
Max	71	64		71
Years active
Mean (SD)	26.2 (11.4)	9.7 (11.2)	< 0.001	18.4 (14.0)
Median	30	4		20
Min	1	1		1
Max	47	38		47

More than 50 % of the participants worked in a group practice, 17.5% in a duo practice and 30.1% worked solo. One-quarter of the participants worked in a network with family physicians in other practices. Forty per cent of the physicians worked in urban practice, 39% in a semi-rural setting and 21% in a rural area. Ten per cent of the participants were trainees, and 19% was involved in the master-after-master training.

All physicians used a computer in their practice, 24% used a laptop on housecalls, 11% a tablet and 25% a smartphone. Ninety-five per cent received a telematics premium from the government for the use of medical informatics, and 80% had access to the internet during house calls.

The physicians who worked in solo practice were significantly older (55 years) than those who worked in group practices (40 years) (p < 0.001). There was no significant difference between the age of those who worked in a rural area (48 years) and those who worked in an urban area (43 years) (p = 0.499) (Table 3).

Table 3

Age differences per type of practice and type of area

	GP in solo practice (N = 43)	GP in duo practice (N = 25)	GP in group practice (N = 75)	P
Age
Mean (SD)	54.6 (9.4)	42.4 (14.1)	40.3 (13.6)	< 0.001
Median	57	46	33
Min	27	25	25
Max	71	64	69
	Urban area	Semi rural area	Rural area	P
	(N = 69)	(N = 63)	(N = 31)
Age
Mean (SD)	42.6 (14.0)	45.7 (14.6)	48.1 (12.8)	0.499
Median	39	51.5	53
Min	25	25	25
Max	71	68	64

Most of the respondents (91%) believed that patients might benefit from EBM. Not less than 59% of the respondents attended an information session on EBM, and 55% estimated their search skills good to excellent.

Younger respondents self-evaluated their search skills in EBM sources better than older participants. Similarly, respondents with a newer activity as family physician self-evaluated their search skills in EBM sources better than those with a longer activity (Table 4) (Figure 1).

Table 4

Perception and willingness to learn about EBM sources

			N	%
Will patient benefit from the use of EBM practice
Yes			130	90.9
No			2	1.4
Unknown			11	7.7
Self-evaluation of search skills
Very bad to bad			23	16.1
Neutral			41	28.7
Good to excellent			79	55.2
Previously attended an info session about searching with (an) EBM source(s)
Yes			84	58.7
No			59	41.3
Willingness to improve search skills
Yes			110	76.9
No			33	23.1
Self-evaluation of search skills	Age		Number of years active
	Mean	SD	Mean		SD
Very bad to bad	55.5	6.1	29.8		6.3
Neutral	46.0	13.4	19.7		13.7
Good to excellent	41.5	14.5	14.5		13.9
P value	< 0.001		< 0.001

Figure 1

Error bar plots (95% C.I.) for the perceived search skills, respectively for age and number of years active as GP

More than 70% of the responders used BAPCOC, BCFI, CEBAM, Domus Medica and NHG. Less than 30% used BELMIP, CBO, CIPIQ-S, “Gezondheid en wetenschap”, Google Scholar, Superior Health Council, SSMG and UpToDate (Table 5).

Table 5

Time spent on average to find information

Source	Users N	<1 minute	1 - 5 minutes	5 - 10 minutes	10 - 20 minutes	>20 minutes
BAPCOC	104	50%	44%	5%	0%	1%
BCFI	129	43%	48%	7%	1%	1%
BELMIP	10	10%	30%	40%	20%	0%
CBO	5	0%	20%	40%	40%	0%
CEBAM	106	4%	48%	38%	8%	2%
CIPIQ-S	1	0%	0%	100%	0%	0%
Clinical Evidence	45	0%	40%	42%	14%	4%
Cochrane Library	72	1%	26%	34%	29%	10%
RIZIV	57	2%	23%	54%	14%	7%
Cybele	51	39%	49%	10%	2%	0%
Domus Medica	122	3%	57%	37%	2%	1%
EBMPracticeNet	92	7%	67%	21%	4%	1%
Evidence Linker	50	12%	60%	20%	4%	4%
Farmaka.be	72	7%	60%	28%	5%	0%
Formularium for Elderly	49	8%	69%	17%	4%	2%
Gezondheid en Wetenschap	19	5%	53%	32%	5%	5%
Google	92	20%	52%	24%	3%	1%
Google Scholar	18	6%	50%	32%	6%	6%
Superior Health Council	35	9%	22%	40%	20%	9%
KCE	68	6%	19%	40%	26%	9%
Medline	70	1%	24%	26%	30%	17%
Minerva	75	5%	31%	39%	15%	10%
NHG	108	6%	50%	36%	6%	2%
SSMG	10	0%	50%	10%	30%	10%
UpToDate	42	7%	40%	29%	17%	7%

Fifty per cent or more of the participants could find their information within five minutes in sources such as BAPCOC, BCFI, CEBAM, Cybele, Domus Medica, EBMPracticeNet, Evidence Linker, Farmaka. Be, Formularium for the Elderly, Gezondheid en Wetenschap, Google, Google Scholar and NHG. For BAPCOC and BCFI the information was even found within the first minute for 50% and 43% of the users, respectively.

Searching in Medline takes relatively longer time. Seventeen per cent needs 20 minutes or more in Medline, and 47% searched for 10 minutes or more. Similar search times were found for SSMG and Cochrane Library.

The user-satisfaction was measured by the proportion of participants that reported to have found an answer to their question. In 80% of the cases BAPCOC, BCFI and Cybele could answer. CBO, Cochrane Library, Superior Health Council, KCE, Medline, Minerva and SSMG delivered an answer in half or less than half of the cases when being used (Table 6).

Table 6

User-satisfaction for different EBM sources

Source	Users	Always	Mostly	50/50	Seldom	Never
BAPCOC	104	41%	51%	6%	2%	0%
BCFI	129	37%	59%	4%	0%	0%
BELMIP	10	10%	30%	50%	10%	0%
CBO	5	0%	20%	60%	20%	0%
CEBAM	106	3%	52%	36%	7%	2%
CIPIQ-S	1	0%	0%	100%	0%	0%
Clinical Evidence	45	0%	33%	54%	13%	0%
Cochrane Library	72	1%	26%	49%	21%	3%
RIZIV	57	0%	28%	54%	14%	4%
Cybele	51	39%	53%	6%	2%	0%
Domus Medica	122	7%	60%	23%	10%	0%
EBMPracticeNet	92	9%	58%	25%	9%	0%
Evidence Linker	50	8%	56%	26%	10%	0%
Farmaka.be	72	3%	44%	43%	10%	0%
Formularium Elderly	49	12%	51%	22%	14%	0%
Gezondheid en Wetenschap	19	0%	58%	32%	10%	0%
Google	92	8%	44%	41%	7%	0%
Google Scholar	18	0%	39%	61%	0%	0%
Superior Health Council	35	0%	43%	34%	23%	0%
KCE	68	0%	33%	42%	23%	2%
Medline	70	1%	30%	46%	20%	3%
Minerva	75	1%	36%	43%	17%	3%
NHG	108	13%	61%	23%	2%	1%
SSMG	10	0%	10%	50%	40%	0%
UpToDate	42	7%	43%	38%	10%	2%

More than 70% of the questioned physicians found the quality of most sources good to very good. Only for BELMIP and Google, the quality was evaluated less good: 20% and 17%, respectively, received a poor score which in contrast to most other sources (Table 7).

Table 7

Quality of the EBM information

Source	Users	Very good	Good	Neutral	Poor	Very poor
BAPCOC	104	57%	39%	4%	0%	0%
BCFI	129	55%	41%	3%	1%	0%
BELMIP	10	20%	40%	20%	20%	0%
CBO	5	20%	40%	40%	0%	0%
CEBAM	106	35%	50%	14%	1%	0%
CIPIQ-S	1	0%	100%	0%	0%	0%
Clinical Evidence	45	24%	53%	22%	0%	0%
Cochrane Library	72	24%	50%	26%	0%	0%
RIZIV	57	21%	56%	21%	2%	0%
Cybele	51	49%	39%	10%	2%	0%
Domus Medica	122	34%	52%	14%	1%	0%
EBMPracticeNet	92	27%	51%	22%	0%	0%
Evidence Linker	50	30%	46%	24%	0%	0%
Farmaka.be	72	40%	47%	13%	0%	0%
Formularium Elderly	49	41%	41%	18%	0%	0%
Gezondheid en Wetenschap	19	26%	53%	16%	5%	0%
Google	92	5%	30%	47%	17%	0%
Google Scholar	18	0%	39%	56%	6%	0%
Superior Health Council	35	17%	49%	31%	3%	0%
KCE	68	16%	60%	19%	4%	0%
Medline	70	13%	50%	34%	3%	0%
Minerva	75	36%	40%	23%	1%	0%
NHG	108	46%	48%	5%	1%	0%
SSMG	10	30%	40%	30%	0%	0%
UpToDate	42	21%	55%	21%	2%	0%

Discussion

The problems with the sample population were already mentioned in the methods section. We do not claim to report on a representative sample of the Belgian family physicians because we recruited Dutch-speaking physicians only, the respondents in our study were younger, and they needed a computer to participate [27]. However, our research did not focus on the participants but on the quality of the EBM sources itself. Consequently, the results provide valuable information on the quality and the user-friendliness of the included EBM sources.

We are aware that the methodology of a user survey has its limitations. We could not measure the real time that was spent to find a certain source but the time was estimated by the participant. It was recorded whether an answer was found in the source. We did not evaluate the correctness of the answer. We could neither evaluate the quality of the sources with a validated method, but we asked for the subjective impression of the physicians.

BAPCOC, BCFI, CEBAM, Domus Medica and NHG were the most frequently used sources. They were freely accessible, in Dutch and focused on the Belgian situation, which may explain the high number of users.

Google was also mentioned among the EBM sources because from our pilot questioning we understood that many physicians used Google in their search for medical information although it is not an EBM source. It is possible that Google was used to access real EBM sources by adding the name of the source in the search strategy. However, this was not investigated in our study.

Most of the EBM users thought that there was a need for improvement or clarification of the available sources. This confirms previous studies indicating that there exist significant barriers to work with EBM because of a large number of sources, the English language, the lack of user-friendliness and contradictions between sources [20] [21].

For most of the sources, the information is found within 5 minutes. These EBM sources had some points in common: they provided practical information, the information was of direct clinical interest and specific, the websites were user-friendly permitting to click through to the topic of choice in an easy way, and the information was concise. These sources did not require to read long articles to find the information. These sources also scored high in the frequency of use, which also can explain that the information was quickly found by intuition.

A small proportion of the participants found the information on BAPCOC, BCFI, BELMIP, Cybele, Evidence Linker and Google even within 60 seconds. These sources focussed on specific informations such as antibiotics, medication and radiographic examinations, respectively.

The sources which needed more search time were used less frequently. Some focussed more on policy support, others contained less specific information, were provided in a different language (Clinical Evidence, Cochrane Library) or contained more extensive and detailed information (UpToDate, Medline). These sources with comprehensive information certainly had their value but were less suitable to use online on-the-spot.

More than 50% of the respondents found an answer to their question in most of the EBM sources. However, only 20% of the participants seldom to never found an answer in CBO, Cochrane Library, Superior Health Council, KCE, Medline, Minerva and SSMG. This may indicate a lack of knowledge about these sources or difficulty in finding the information available. Sources, where it was difficult to find the answer, were not necessarily bad source.

Physicians should make a considered choice for which clinical question a particular source may be used. A bad search strategy may have led to a non-response, even though the source was, on itself, of good quality. This is in line with the study by Davies et al., [26] where the importance has been emphasised that physicians know where to find good guidelines.

The respondents found the quality of the information in the majority of the sources good to excellent. Only for BELMIP and Google, the quality was considered poor by at least 17% of the responders. It must be said that Google is not an EBM source but a search engine that gives access to a huge number of sources whose quality is not assured. It is not surprising that physicians evaluate its quality as poor. According to Tang and Ng, Google shows the correct diagnosis in 58% of the searches [28]. This corresponds with a wrong diagnosis in 42% of the cases.

Younger physicians self-evaluated their search skills better than older physicians. Several reasons may underlie these findings.

The recent introduction of some new sources such as EBMPracticeNet and Evidence Linker [22] and the recent incorporation of courses on EBM in the medical training [29] have probably contributed to the fact that younger physicians have better search skills for these sources. Younger physicians have less clinical experience and need to search for more information, even in the presence of the patient. Older physicians may more rely on their experience and do less appeal to EBM [21]. Moreover, a shorter search time makes searching on-the-spot easier. And finally, the more experience a physician has with a resource, the quicker the searching.

The fact that some sources were used more frequently suggests that there was a better understanding of the usefulness of these sources and that experience with a source contributes to better search strategies. This may explain the higher user-satisfaction and higher perceived quality by younger physicians. Physicians should know the specific goals of a source and the kind of information available in the sources to choose the most suitable source. This will increase the perceived user-friendliness and quality of the source.

In conclusion, physicians evaluate the quality of the information on most of the EBM sources as good to very good. Only for BELMIP and Google, the quality was considered as poor by at least 17% of the physicians. More than half of the respondents found an answer to their questions in most of the sources. For some sources of known high quality [for example Cochrane and Medline), the answer to the question was rarely or never found in 20% of the cases.

More than half of the respondents found the necessary information in less than 5 minutes in most Belgian sources but also in Google and NHG. The fact that all these sources were available in Dutch, the language of the participants, may play a role. Thereupon is the information in these sources very concise, of direct clinical relevance and adapted to the local situation. We advise the developers of EBM sources to take these points into account by the development or modification of new EBM sources.