Clinical Evidence: a useful tool for promoting evidence-based practice?

BACKGROUND: Research has shown that many healthcare professionals have problems with guidelines as they would prefer to be given all relevant information relevant to decision-making rather than being told what they should do. This study assesses doctors' judgement of the validity, relevance, clarity and usability of the Italian translation of Clinical Evidence (CE) after its free distribution launched by the Italian Ministry of Health.
METHODS: Opinions elicited using a standardised questionnaire delivered either by mail or during educational or professional meetings.
RESULTS: Twenty percent (n = 1350) doctors participated the study. Most of them found CE's content valid, useful and relevant for their clinical practice, and said CE can foster communications among clinicians, particularly among GPs and specialists. Hospital doctors (63%) more often than GPs (48%) read the detailed presentation of individual chapters. Twenty-nine percent said CE brought changes in their clinical practice. Doctors appreciated CE's nature of an evidence-based information compendium and would have not preferred a collection of practice guidelines.
CONCLUSIONS: Overall, the pilot initiative launched by the Italian Ministry of Health seems to have been well received and to support the subsequent decision to make the Italian edition of Clinical Evidence concise available to all doctors practising in the country. Local implementation initiatives should be warranted to favour doctor's use of CE.

Background

While there is a wide consensus that information on the effectiveness of health care interventions should be, as much as possible, evidence-based, it is common experience that many important questions doctors face in clinical practice do not have such information[1].

Moreover, research on health services and quality of care have consistently demonstrated that the availability of good quality information is an essential yet insufficient condition for enhancing doctors' adherence to evidence-based practices[2]. Some have argued that, in order to be usable, information should be credible and independent from commercial interests, relevant and applicable in clinical practice and be delivered within a context (i.e cultural organisational, etc.) that favours its adoption and implementation.

Some ten years ago the Cochrane Collaboration took the challenge of producing, maintaining and disseminating the results of high quality systematic reviews of the effects of health care interventions[3]. However, Cochrane reviews are sometimes evidence-rather than question-driven and their format is more appropriate for researchers than for those who need practical guidance at the time health care decisions are taken. Evidence based practice guidelines (PGs), as well as other tools (such as clinical pathways, etc) have also attracted substantial interest and investments but research has shown that PGs represent tools many health care professionals have problems with [4-6] and their impact has been shown to be generally modest.

In 1995 the BMJ Publishing Group launched Clinical Evidence (CE), a compendium summarising the best available evidence of the effects of health care interventions, published and updated twice a year[7-9]

Key distinguishing features of Clinical Evidence are that [7]:

• its contents are driven by practical questions rather than by the availability of research evidence;

• it aims not to make recommendations (unlike PGs) but to inform health professionals on the best available evidence;

• it highlights rather than hiding gaps in research evidence, so that physicians know when their uncertainty stems from these gaps rather than from gaps in their own knowledge.

In response to the need for more general, good quality and updated information the General Directorate for Drug Evaluation and Surveillance of the Italian Ministry of Health launched, at the end of the end of the '90s, the "National Program for Independent Information (NPII)" with a funding that since 2001 amounts to approximately 50 million Euro. The NPII has an infrastructure comprising two monitoring centres and foresees the regular the dissemination of various information tools.

The two centres are the "Osservatorio dei Medicinali (OsMed" aimed at providing annual feedback on profiles of drug utilisation and the "Osservatorio Sperimentazioni Cliniche" reporting information on clinical trials and clinical research approved by Italian Research Ethics Committees). The dissemination activity included the translation and free distribution to health professionals of three reference books – British National Formulary; Medicines for Children and Clinical Evidence (CE). The latter was introduced gradually with a first pilot distribution – whose evaluation is the subject of this article – took place in 2001. Still as part of the information dissemination activities a bimonthly journal on drug efficacy and safety, "Bollettino di Informazione sui Farmaci" (BIF) [10], is delivered to all physicians (300.000) and pharmacists (60.000). The implementation of a unified web reference source where different information tools will be made available to all health professionals within the country is currently under discussion.

Methods

CE was freely distributed to 43.000 doctors (i.e. 15% of the total practising in 10 Italian regions). After one or more public presentations organised by the local departments of health, the volume was mailed to a sample of individual doctors chosen as target for the pilot experiment.

The evaluation was carried out using a standardised questionnaire developed by the Italian Cochrane Centre (ICC), with the help of 30 members of the scientific committee of CE's Italian edition, and tested for face validity within a small group of 8 GPs not subsequently involved into the study. The final version of the questionnaire (Appendix, see additional file 1) included 17-items exploring frequency of use as well as opinions about the scientific validity, clarity, relevance and readability of the book. The final section explored whether responders considered CE as a collection of guidelines or a source of evidence based information. The questionnaire was delivered to a total of 6619 physicians (GPs, hospital-based and other specialists, doctors working into hospital management units ("Direzioni Sanitarie") 3–6 months after the delivery of the book. The sample was not selected with a formally randomised method but was drawn from a list of active doctors, stratified by region, provided by regional and local health departments. In three regions questionnaires were mailed by regional health departments, in one by investigators in charge to each local health unit while for the remaining six regions they were mailed centrally by the Italian Cochrane Centre. All questionnaires were anonymous and no follow up mailing was attempted after the first delivery.

Results

Overall 20% (n = 1350) physicians filled out the questionnaire with marked regional variations in response rates (range= 8%-31%). Three-quarters respondents were males, 63% GPs and 23% hospital clinicians. Median time since graduation was 23 years.

Most respondents (84%) used CE at least once in the month prior to the survey (41% once or twice, 23% about five times and 10% at least twice a week). Professional continuing education and search for solutions to specific clinical questions were the main reported reasons for consultation (Table 1).

Table 1

Respondents' opinions about CE (n° responses)

	Very	Fairly	Little	Not at all	Don't know
Are CE contents comprehensible? (1110)	26.0	65.9	6.4	1.6	0.2
In your opinion, are CE contents scientifically valid? (1101)	27.7	68.0	3.4	0.7	0.2
Considering your practice, has CE consultation been generally useful? (1096)	20.4	68.2	8.5	2.3	0.6
useful for keeping updated (992)	38.9	54.7	4.6	1.3	0.4
useful for getting information about specific questions (941)	34.8	57.4	6.2	1.3	0.4
useful for preparing seminars/publications (443)	18.7	34.3	24.2	7.7	15.1

Most respondents considered CE very/fairly understandable and useful, and its scientific contents very/fairly valid (Table 1). Its thoroughness was considered "just right" by 90%. More than half (53%) read the detailed presentation of the chapters, whereas 43% focused on key messages and 64% said that they looked only at the conclusions (Table 2); the chapter on cardiovascular diseases being the most commonly read (Table 3).

Table 2

Sections read by professional setting (% responders)

CE'S SECTIONS READ	PROFESSIONAL CATEGORY
	General practitioners	Hospital doctors	All
Questions and interventions	64.5	61.5	63.8
Key messages	41,0	44.7	43.4
Detailed presentation	47.6 *	63.0*	53.1

* p < 0.01

Table 3

Chapters read (% responders)

Cardiovascular disorders	59,2
Digestive diseases	32,3
Infectious disorders	31,0
Respiratory disorders	30,1
Muscoloskeletal disorders	29,4
Endocrine disorders	22,4
Neurological disorders	21,3
Skin disorders	18,6
Mental health	18,4
Child health	15,4
Kidney disorders	14,9
Ear, nose and throat disorders	14,8
Pregnancy and childbirth	9,6
Poisoning	8,6
Men's health	8,3
Eye disorders	7,1
Sexual health	5,8
Oral health	3,4

* p < 0.01

Twenty-nine percent said they had changed their practice after reading CE; 28% said they learned that several common health care interventions are not evidence-based and 10% said that by consulting the book they realised that some effective interventions may be underused. CE was also seen by 54% as a useful tool for improving inter-professional collaboration between GPs and specialists. Sixty-four percent indicated that they value evidence-based information more than guidelines. However, 38% of the respondents still classified CE as a collection of guidelines.

Years since graduation and perception of CE as a book of guidelines did not predict for differences in opinions. CE was felt more useful by GPs than hospital specialists ("very useful" for 23% vs. 14%, p < 0.01) and GPs also reported more often to have changed their practice after reading CE (33% vs. 23%, p < 0.01). Hospital based doctors reported to have read the detailed presentation of the chapters more often than GPs (63% vs. 47.6%, p < 0.01) (Table 2).

Although more than half (54%) respondents came from one region (Emilia-Romagna) which also yielded the best response rate (31%), results showed the same qualitative directions and did not vary appreciably among different regions

Discussion

Italian clinicians seem to consider CE a clear, valid and useful source of clinical information. Three main findings should be highlighted. First, CE seems to affect doctors' awareness about the efficacy and safety of some health interventions and, most importantly, to make them willing to consider changes in their clinical behaviour. This seemed true especially among GP's. Second, CE is deemed to foster communication between GPs and specialists by creating a common knowledge ground; this might help smoothing the interface between primary and secondary care and improving overall patient care. Third, our data suggest that the original objective of the book – ie providing information rather than making clinical recommendations [5] – may be achieved. The majority of respondents stated a preference for evidence-based summaries over guidelines, confirming that the latter are sometimes not easy to be accepted by doctors.

This pilot, free distribution of CE to Italian doctors seems to have been well received, confirming their preference for problem-driven information and the key-role of a strong endorsement from Health Authorities for the implementation of this information. Local implementation initiatives should be warranted to favour doctors' use of CE (for example through pharmacists outreach visits and Continuing Medical Education programs). Recognising the value of promoting information independent from the drug industry [11] and also thanks to the encouraging results of this survey the Italian Ministy of Health is continuing – though the NPII briefly described in the introduction section of this article – this initiative. Between July and September 2003 all 300.000 Italian doctors have received a free copy of CE's second Italian edition, based on CE issue 7, in the Concise format. This relevant investment, coming to 1,900,000 Euro (the previous translation and distribution evaluated here cost instead 770,000 Euro) indeed pales when compared to the 575 times larger investment (1,093 million Euro) made by pharmaceutical companies for medical information in 2002 only [12].

In interpreting the results of this study some limitations should be considered. As in most surveys, and considering the low response rate, selection bias cannot be ruled out as those more interested in using CE may have been more willing to respond. However, 17% of the total sample (i.e. 84% of the total of 20% responders) used Clinical Evidence "at least once" in the month prior to the survey and 6% (29% of the 20% responders) said they had changed their practice after reading it. This means that the free distribution of CE and the endorsement of the Italian NHS seem justified even in the "worst case scenario" (ie assuming that all non responders did not open the book at all). Further more qualitative evaluations will be carried out in the future to better explore CE's usefulness and these evaluations will hopefully be conducted internationally as Clinical evidence is now available in several languages other than English (Spanish, French, German, Russian, Japanese). Another limiting factor is that more than half respondents came from one region. However, this latter seems unlikely to have affected our results given that the regional subgroup with the highest response rate yielded results similar to those of the whole sample. Finally, although a positive attitude towards CE emerged, still 38% of responders classified it as a collection of guidelines. This may call for a greater effort to clarify the nature of CE in order to improve its acceptability and to decrease the likelihood that its users see it as a prescriptive rather than an informative source of data.

Authors' Contributors

GF helped with the preparation of the questionnaire, co-ordinated the study in one region, analysed the data, wrote the first draft of the manuscript and provided important feedback in the subsequent versions.

LM helped with the preparation of the questionnaire, co-ordinated the study in 9 regions and helped with the preparation of the manuscript.

FN participated to the preparation of the Italian edition of CE retrieving data on the epidemiology of the diseases included in the Italian version of CE' chapters and commented on earlier drafts of the manuscript.

PD helped with the preparation of the questionnaire, provided comments and suggestions in the preparation of the manuscript.

AA provided comments and suggestions in the preparation of the manuscript.

NM reviewed the final version of the manuscript.

AL designed the study, prepared the first draft of the questionnaire and provided substantial comments and feedback on the several drafts of the manuscript. He is the guarantor of the paper.

Competing interests

The Italian Cochrane Centre (ICC) was the recipient and general administrator of the grant from the Italian Ministry of Health to oversee the scientific accuracy of the translation of Clinical Evidence and to set up – assisted by Zadig Editore – the editorial format of the Italian version.

Ce.V.E.A.S. collaborated with the design of the survey but did not receive any grant money for it.

The ICC designed and co-ordinated the survey independently from the Italian Ministry of Health and the BMJ Publishing Group. While in the contract the publication of the Italian version of CE was subject to the approval from both Clinical Evidence (BMJ Publishing Group) and the Italian Ministry of health no prior approval nor scrutiny on the results was warranted before the publication of the results of this survey.

Although the ICC and Editore Zadig did not have any direct financial interest in the outcome of this evaluation they have indeed scientific and cultural interests in the success of the dissemination of the book; this may have influenced their interpretation of the results.

Pre-publication history

The pre-publication history for this paper can be accessed here:

Additional File 1

Appendix – The final version of the questionnaire

Click here for file