A simple heuristic for Internet-based evidence search in primary care: a randomized controlled trial.

BACKGROUND: General practitioners (GPs) are confronted with a wide variety of clinical questions, many of which remain unanswered.
METHODS: In order to assist GPs in finding quick, evidence-based answers, we developed a learning program (LP) with a short interactive workshop based on a simple three-step-heuristic to improve their search and appraisal competence (SAC). We evaluated the LP effectiveness with a randomized controlled trial (RCT). Participants (intervention group [IG] n=20; control group [CG] n=31) rated acceptance and satisfaction and also answered 39 knowledge questions to assess their SAC. We controlled for previous knowledge in content areas covered by the test.
RESULTS: Main outcome - SAC: within both groups, the pre-post test shows significant (P=0.00) improvements in correctness (IG 15% vs CG 11%) and confidence (32% vs 26%) to find evidence-based answers. However, the SAC difference was not significant in the RCT. OTHER MEASURES: Most workshop participants rated "learning atmosphere" (90%), "skills acquired" (90%), and "relevancy to my practice" (86%) as good or very good. The LP-recommendations were implemented by 67% of the IG, whereas 15% of the CG already conformed to LP recommendations spontaneously (odds ratio 9.6, P=0.00). After literature search, the IG showed a (not significantly) higher satisfaction regarding "time spent" (IG 80% vs CG 65%), "quality of information" (65% vs 54%), and "amount of information" (53% vs 47%).
CONCLUSION: Long-standing established GPs have a good SAC. Despite high acceptance, strong learning effects, positive search experience, and significant increase of SAC in the pre-post test, the RCT of our LP showed no significant difference in SAC between IG and CG. However, we suggest that our simple decision heuristic merits further investigation.

RCT Entities:   Population Interventions Outcomes

Introduction

Unanswered questions

In primary care, patients present a wide spectrum of clinical problems to their general practitioners (GPs). The questions that are brought up in these encounters touch upon virtually every clinical topic.1–3 According to a recent systematic review, ten patient visits left GPs with approximately six unanswered questions on average. Only ~50% of GPs attempted to find an answer.4 The main reasons given by GPs for not pursuing their open questions further were lack of time and doubt that relevant information was available at all.

Available information

There is a discrepancy between the situation described earlier and the large amount of medical information available. Structured databases and other sources are accessible via the Internet. Online evidence has the potential to provide quick answers to physicians’ questions.5

Clinicians’ use of electronic resources has been reported to be increasing.6 Examinations of resources used in primary care have revealed that, after textbooks and discussions with other physicians, Internet use represents the third most frequently used resource.4,6–10

Answering questions

Evidence-based medicine (EbM) has provided a framework for accessing, appraising, and using the medical literature relevant to individual patients. One of its pioneers, David Sackett, coined the paradigm of “Five Steps”: question formulation, literature search, critical appraisal, implementation of results, and evaluation.11 However, time constraints and lack of skills prevent most physicians from following this model.1,10,12–16

As an alternative to the classical appraising mode, the McMaster Group already suggested the searching mode, which primarily uses databases of processed information for quick answers to questions arising in everyday care situations. Even physicians familiar with EbM appraisal methods would resort to less sophisticated strategies for their everyday clinical questions. Slawson and Shaughnessy have formulated a need for simple heuristics not just to appraise but to manage medical information.17,18

Methods

Learning program (LP)

We developed an LP that is based on a simple three-step heuristic (Figure 1). The aim of the program is to help GPs use Internet sources for their clinical questions. This can be taught in interactive seminars. All information is also available in the form of a booklet for individual study. A CD-ROM with relevant URLs and background material helps GPs put into practice what they have learned. In order to improve the effectiveness of the program, we used multimedia (workshop [WS], booklet, CD, Internet) and multiple exposures (lectures, exercises, homework).19

Figure 1

The three-step heuristic of decision making.

Simple heuristic

The content of the LP (Figure 2) was structured around the three-step heuristic,20 shown in Figure 1. Its application ensures that GPs search and appraise sources in a methodological way and also helps them decide when to end their search. We regard this as especially important because GPs often feel overwhelmed by the amount of available information.21–23 This set of three questions is not meant to be followed strictly in every literature search. It is also meant to encourage GPs to reflect on their own search and appraisal habits.

Figure 2

Workshop table of contents.

Abbreviations: ARR, absolute risk reduction; RRR, relative risk reduction; NNT, number needed to treat.

Example 1: a commercial manufacturer mentions in the drug-sheet of a lipid-lowering drug that no long-term studies with patient-relevant outcomes have been conducted so far. This information is not in line with the interest of the manufacturer (source); it should therefore be accepted without further search. The claim of a manufacturer that a herbal preparation alleviates tumor pain, however, would not lead to immediate acceptance but should motivate further research.

Example 2: if for a herbal preparation, no formal evaluation, such as a controlled study, can be found, its use should be avoided. Only if the claim is in line with the interest of the source and trials evaluating a particular treatment exist, a more detailed appraisal of study design features is required. Here the most valid and relevant criteria should be checked first, such as:

is this a randomized controlled trial (RCT)?

is the main outcome clinically relevant?

can the choice of the control treatment be justified?

which patients have been recruited into the study (external generalizability)?

Participants

GPs located in five different cities in Germany were invited to take part in the main study. Participants had to be motivated to attend two WS sessions and contribute to the evaluation.

Stratification

GPs were asked whether they had performed an Internet search for a clinical question during the previous year. The answer (yes/no) was used for stratification.

Randomization

The computer-assisted randomization was undertaken by the central clinical trials unit at the University of Marburg. Within the two stratification-groups, permuted blocks of variable length were formed, including one to four participants. According to the order of registration, the GPs were assigned to either the intervention group (IG) or the control group (CG). In order to maintain the allocation concealment, the participants were not given details about the study design or the allocation process. The control arm was a waiting group of GPs who were offered the WS after study completion and evaluation of the main outcome.

Intervention

GPs attended two interactive small group WS in a computer classroom. Figure 2 shows the contents of the WS. Sessions provided information on databases that are useful for GPs. Their scope, advantages, and disadvantages were discussed. There was an emphasis on processed evidence-based information as opposed to original papers, such as drug bulletins, clinical evidence (British Medical Journal) or national and international guidelines. Practical search strategies, bias of sources, and simple criteria to check the validity of information were covered. There was ample opportunity for hands-on exercises and discussion of participants’ experience with the sources provided. WS lasted at least 2 hours each with a 4-week interval in-between. More specific background information can be found in the “Supplementary materials” section (English) and on the Internet (German).24

Study outcomes

The primary outcome of the study was GPs’ search and appraisal competence (SAC). The secondary outcome was to determine the acceptance and satisfaction with the WS.

WS acceptance

We asked GPs to evaluate both WS using a semi-standardized questionnaire each time directly after the event. Fifteen questions in a 6-point Likert scale covered organization, educational process, and relevance of content (1= very good, 6= deficient). Finally, they estimated their own commitment for change regarding content covered by the WS in a standardized way.25

Search satisfaction

GPs were asked to rate the relevance of content and their satisfaction using the strategies learnt. Acceptance and satisfaction were evaluated not only by IG participants but also by control subjects after their participation in the WS.

SAC

We evaluated GPs’ SAC by additionally (after pre–post test) correct answered clinical questions.

Clinical questions

Thirty-nine clinical questions from seven clinical domains were developed by the study team. Questions concerned prevention, therapy, diagnosis, and prognosis. Most questions were in multiple-choice format, but GPs were also asked to elaborate on some open questions. The appropriateness of answers was checked by experts in EbM and clinical areas covered (see the “Acknowledgments” section).

Pre–post test

All participants had to answer each question twice: first, spontaneously in order to establish their baseline knowledge and again after a thorough search. We could thus control for previous knowledge.

If participants improved, ie, gave an incorrect answer at baseline but a correct answer after thorough search, the answer was scored “+1”. In the case of a correct answer at baseline but a wrong answer after the search, the score was “−1”; no change resulted in “0”. We arrived at an overall SAC score by adding up the single question scores for each participating GP. The result can theoretically vary between −39 (−100%) and +39 (+100%) additional correct answers. Besides the objective professional correctness, we checked the subjective confidence (0%–100%).

Pilot study

Before starting the study, we checked the comprehensibility and the level of difficulty of the 39 clinical questions with a group of 12 GPs. Subsequently, we held the workshop (WS) with another 14 GPs to improve methods and content including learning material. These GPs did not take part in the main study.

Statistics

The data analyses were performed with SPSS software (SPSS Inc., Chicago, IL, USA). For baseline comparisons, we report medians, interquartile ranges, and results of the Mann-Whitney U test (P-values) for continuous variables and percentages, and results of asymptotic χ2 tests for categorical variables. Differences in SAC improvement (before vs after; between study arms) were evaluated by the Mann-Whitney U test (P-value). We used the Hodges-Lehmann estimator to obtain the 95% confidence interval for the difference between medians.26

Results

Study participants

Recruitment, randomization, and flow of participants are shown in Figure 3. Twelve percent (n=110 of 905) of the GPs originally approached gave written consent to participate and were randomized thereafter. Seventy-six percent (n=84 of 110) of subscribers attended at least one WS. Finally, complete data were available from 46% (n=51 of 110) of the original group. Consequently, the dropout rate was 64% due to delayed information search (n=30 “did not search”), absence (n=10 “did not attend”), and incomplete WS attendance (n=15 “only one WS”) or incomplete data (n=4).

Figure 3

Flowchart of general practitioners’ participation, and dropout.

Abbreviation: WS, workshop.

Characterization of study participants

Participants’ characteristics are shown in Table 1. In most respects, they resembled the population of GPs in Hessen. The IG and CG did not differ significantly, but in the IG, there were slightly more GPs with Internet search experience and a DocCheck™ password (access to content provided by the pharmaceutical industry).

Table 1

Preconditions of study participants

Skills and preconditions	External comparison: general practitioners				Internal comparison: complete dataa
Group characteristics	Gb	Hc			IG	CG	P*
	n=32,477	n=2,860	n=84	n=51	n=20	n=31
Personal preconditions
Age (years, SD),	52b	52 (7.5)	49 (7.4)	49 (9.8)	50 (7.3)	48 (11.2)	0.56
Sex (female, mean %)	38d	36	30	26	30	23	0.79
Professional preconditions
Years in practice, (median, SD)	–	14 (9.3)	13 (8.5)	14 (8.2)	15 (9.1)	13 (7.6)	0.29
Working in group practice (mean %)	32b	42	58	61	53	68	0.34
Internet preconditions
Internet access: “Yes”	–	–	95%	96%	95%	97%	0.44
Broadband account: “Yes”	–	–	61%	61%	65%	58%	0.58
DocCheck™ password:e “Yes”, %	–	–	44	42	55	32	0.08
Yes: Internet searchf, %	–	–	58	61	75	52	0.1
Skillsg (good or very good, %)
Internet	–	–	45	42	47	42	0.12
Computer	–	–	41	36	47	39	0.2
English language	–	–	31	38	37	39	0.9

Notes:

Complete data: 1: participation and evaluation of both workshops; 2: documentation and evaluation of seven literature searches; 3: answering 39 questions twice.

G = published statistics of the Federal Chamber of Physicians in Germany;37

H = details of the Regional Chamber of Physicians in Hessen (Landesärztekammer Hessen);38

representative sample of the Commonwealth Fund;39

the DocCheck™ password allows licensee to access drug formulary and labeling information;

participants have conducted Internet search for medical topics in the previous year;

skills as judged by participants themselves. Scale: 1 (low) to 5 (high).

P-values for between-group comparison.

Abbreviations: IG, intervention group; CG, control group; SD, standard deviation.

Acceptance

GPs rated the Continuing Medical Education WS as very positive (Figure 4). Over 80% of respondents rated all items as good or very good. We obtained similar ratings referring to WS organization and educational process (data not shown). At the end of the WS, most participants (78% in the IG and 69% in the CG) made a commitment to change their information management according to what they had learnt during the WS (Figure 5).

Figure 4

Participants’ workshop evaluation.

Notes: (N=51, 1= very good, 6= deficient). Please note that GPs in the control group are included in this process evaluation.

Abbreviations: IG, intervention group; GPs, general practitioners; CG, control group.

Figure 5

Commitment to change at the end of the workshop (n=48).

Note: Please note that GPs in the control group are included in this process evaluation.

Abbreviation: GPs, general practitioners.

Literature search experience

In order to find answers to standardized questions, the IG used more sources per topic and spent more time searching than the CG (Table 2). Members of the IG were slightly more satisfied with the quality and amount of information as well as the time spent. Ratings for the sources’ trustworthiness and relevance were almost identical in both study arms.

Table 2

Literature search experience

Group allocation
Intensitiy and satisfiction of search	IG (n=20)	CG++ (n=31)	Difference#	P*
Number of sources used per topic
Median (IQR)	1.9 (0.6)	1.4 (0.5)	36%	0.01
Duration of search per topic
Median min (IQR)	20.9 (9.6)	17.8 (11.5)	17%	0.34
“Yes, I am satisfied with […] […] quality of information”	65%	54%	11%	0.08
“[…] time spent”	80%	65%	15%	0.35
“[…] amount of information”	53%	47%	6%	0.22
Surprised by result	26%	35%	9%	0.28
Trustworthiness of sources
Median (score: 1= low to 5= high) (IQR)	3.7 (0.6)	3.6 (0.7)	0.1	0.47
Results relevant for daily practice
Median (score: 1= low to 5= high) (IQR)	3.1 (0.9)	3.1 (0.8)	0.0	0.88

Notes:

P-value for between-group comparison: Mann-Whitney U test.

Please note that physicians in the control group provided these evaluations before they were offered workshop participation.

Difference in % between IG and CG.

Abbreviations: IG, intervention group; CG, control group; IQR, interquartile range.

Literature search sources

IG and CG preferred different sources to search literature (Table 3). The IG used more Internet sources (+22%) and targeted web sites (+37%). Overall, the WS recommendations on literature review were implemented well (67%, odds ratio 9.6, 95% confidence interval 2.6–36.4; P=0.0008).

Table 3

Sources of information used for literature review

Intervention group (IG)/control group (CG)	IG (n=20)	CG++ (n=31)	Difference# (IG-CG)	OR
Sources of information(n=570)
Number of sources per topic	1.9	1.4	0.5	1.4
1. Internet sources (n=382, =67%)	80.7%	59.4%	21.3%	2.9
(of which n= 119 search engines: Google, Yahoo, etc)	10.2%	26.2%	−16.0%	0.3
(of which n=263 targeted site selection)	70.5%	33.2%	37.3%	4.8
2. Print media (magazines, textbooks)	15.7%	30.5%	−14.8%	0.3
3. Personal interview	3.6%	10.1%	−6.5%	0.4
Internet sources(n=263, without search engines)
1. Pharma-critical drug bulletins	17.1%	8.0%	9.1%	2.4
2. Guidelines (national and international)	38.7%	23.0%	15.7%	2.1
3. Commercial homepages (pharmaceutical, publishing houses, clinics)	1.8%	31.0%	−29.2%	0.2
Adherence to workshop recommendations	67%	15%	52%	9.6

Notes:

Please note that physicians in the control group provided these evaluations before they were offered workshop participation.

Difference in % between IG and CG.

Abbreviation: OR, odds ratio.

The improvement in answers to clinical questions was the main outcome measure. In both study arms, GPs performed better with a thorough search than at baseline (Table 4). Median improvement was 11% in the CG (Wilcoxon test: P<0.0003) and 15% in the IG (Wilcoxon test: P=0.007). Although IG GPs improved slightly more than controls, the between-group comparison was statistically not significant (P=0.29).

Table 4

Effectiveness measures of search and appraisal competence

Correctness and confidence	Pre test	Post test	Difference (95% CI)	P*
Correctnessa (SAC)b
Pre–post trial
IG (n=20)	39.6%	54.5%	14.9% (7.85–21.90)	0.00
CG++ (n=31)	41.9%	52.6%	10.7% (5.98–15.36)	0.00
Randomized controlled trial
IG/CG (n=51)	−2.32%	1.88%	4.2% (−4.26–1.01)	0.29
Confidencec
Pre–post trial
IG (n=20)	50.1%	81.8%	31.7% (20.9–42.5)	0.00
CG++ (n=31)	53.6%	79.4%	25.8% (20.1–31.5)	0.00
Randomized controlled trial
IG/CG (n=51)	−3.5%	2.4%	5.9% (−17.7–5.9)	0.28

Notes:

Correctness: additional correct answers: before vs after search in pre–post test.

SAC corresponds with score of correctness.

Confidence: perceived safety when responding (0 to 100%: 0= completely unsafe/100%= absolutely certain that the answer is correct. P indicates Student’s t-test used, for paired samples, twotailed significance.

Please note that physicians in the control group provided these evaluations before they were offered workshop participation.

Abbreviations: CI, confidence interval; SAC, search and appraisal competence; IG, intervention group; CG, control group.

Discussion

Our study shows good WS acceptance, strong learning effects, positive search experience, and significant increase of SAC in the pre–post test comparison. Also, SAC improved more in the IG than in the CG although the difference was not statistically significant in the RCT.

The potential of online evidence in order to improve SAC skills of GPs has been demonstrated in controlled laboratory settings. Previous pre–post trials have shown an improvement from 10% to 18% in GPs’ answers to typical clinical scenarios following the use of online evidence.5,25,27 Our results correspond to these findings, with the proportion of correct answers increasing by 15% in the IG (Table 4).

The learning effect on the IG could be shown through the implementation of suggestions given in the WS (Table 3, odds ratio 9.6) of using targeted Internet sources (odds ratio 4.8), pharma-critical drug bulletins (odds ratio 2.4) or guidelines (odds ratio 2.1). We encouraged our participants to use targeted sources they know well despite their limitations of scope and possible bias because accessing the Internet via unspecific search engines such as Google or Yahoo is prone to error.28,29

In many areas requiring decision making under time pressure and with limited information, fast and frugal heuristics have been shown to be valid and helpful.30 Our three-step heuristic20 (Figure 2) not only provides rules for search and appraisal of validity but also rules for when to stop a search to save time. This seems important because lack of time is one of the main reasons why questions remain unanswered.4 In our study (Table 2), the IG invested 17% more amount of time but investigated 36% more sources in this time. Moreover, GPs in the IG were more satisfied with the amount of time invested (+16%), the quality (+11%), and the quantity (+6%) of the information thus obtained.

Our recruitment rates of 12% (Figure 3, “consenting to participate”) and 6% (“complete data available”), respectively, seem rather low. However, the experience of other studies with GPs in Germany shows that “it is generally possible to enlist 3%–4% of the initially contacted practices”.31 Thus, our recruitment quota of 12% actually indicates that GPs have a relatively large interest in the topic of our study. The positive comments by participants are in line with this idea.

GPs in the CG had more previous methodological knowledge than we had expected. They more often obtained their information from the Internet (59%) than from printed media (31%) or through consultation of colleagues (10%). Literature has so far listed the relative importance of these sources of information in the reverse order.4,10,32 This may be an indication of a powerful CG.

Strengths and limitations of the study

The main strengths are the RCT study design and the inclusion of experienced GPs in private practice because they are often underrepresented in studies of appraisal skills training.33–35

Only 46% (Figure 3) of those approached actually terminated the study completely. Although this limits the external validity of our findings, it reflects the reality of continuing medical education where physicians are usually free to decide which program they want to take part in. Though, study participants’ characteristics were similar to the population of GPs in Hessen and Germany.

More serious is the high dropout rate (54%) which differed between study arms (IG: 62% vs CG: 47%). As a result, the power of the study was limited and our findings therefore failed to reach statistical significance.

Apart from a lack of power due to the small sample size, a ceiling effect remains as a possible explanation for the negative result. Theoretically, even control physicians may operate at such a high level regarding their SAC that a WS of the kind evaluated in this study would not improve the outcome to a relevant degree. We regard this as unlikely. However, there remains the possibility that we recruited GPs with above-average SAC skills. This would be an alternative explanation for the negative findings of our study.

In any case, it is known that training effects become harder to detect with longer professional experience, more previous knowledge, and higher qualifications.5 Furthermore, a longer and more intense learning experience might be required to achieve a measurable increase in SAC.36

Conclusion

Conclusions regarding our learning project: the present RCT could not prove the superiority of our intervention based on the three-step heuristic. Lack of power and above-average SAC skills of the sample are the most likely explanation. Moreover, longer and more intense teaching efforts are required to improve experienced GPs’ skills and change their behavior. On the other hand, the WS was well accepted and participants were highly satisfied. We therefore suggest that educators and researchers evaluate the three-step heuristic in their respective settings.

The high acceptance and proof of effectiveness have encouraged us to continue offering WSs for GPs and to further develop our LP. However, the success of our LP is made complicated by the fact that only two to three consecutive WSs are possible for a group of GPs. We are currently working on optimizing our material for the adoption of a blended learning LP. The initiative required of each participant in a blended learning scenario could help us manage the natural process of self-selection in a constructive way.

General conclusions regarding educational projects: an RCT is an important tool to test effectiveness. Applying only a pre–post test without a CG can easily lead to an overestimation of the effectiveness of an intervention.