Speed of updating online evidence based point of care summaries: prospective cohort analysis.

OBJECTIVE: To evaluate the ability of international point of care information summaries to update evidence relevant to medical practice.
DESIGN: Prospective cohort bibliometric analysis.
SETTING: Top five point of care information summaries (Clinical Evidence, EBMGuidelines, eMedicine, Dynamed, UpToDate) ranked for coverage of medical conditions, editorial quality, and evidence based methodology. MAIN OUTCOME MEASURES: From June 2009 to May 2010 we measured the incidence of research findings relating to potentially eligible newsworthy evidence. As samples, we chose systematic reviews rated as relevant by international research networks (such as, Evidence-Based Medicine, ACP Journal Club, and the Cochrane Collaboration). Every month we assessed whether each sampled review was cited in at least one chapter of the five summaries. The cumulative updating rate was analysed with Kaplan-Meier curves. Results From April to December 2009, 128 reviews were retrieved; 53% (68) from the literature surveillance journals and 47% (60) from the Cochrane Library. At nine months, Dynamed had cited 87% of the sampled reviews, while the other summaries had cited less than 50%. The updating speed of Dynamed clearly led the others. For instance, the hazard ratios for citations in EBM Guidelines and Clinical Evidence versus the top performer were 0.22 (95% confidence interval 0.17 to 0.29) and 0.03 (0.01 to 0.05). Conclusions Point of care information summaries include evidence relevant to practice at different speeds. A qualitative analysis of updating mechanisms is needed to determine whether greater speed corresponds to more appropriate incorporation of new information.

Introduction

As biomedicine evolves with the accumulation of new research and publications, promising healthcare interventions can emerge while others become out of date or suboptimal.1 2 Sound evidence—together with contextual factors, values, resources, etc—forms the basic framework on which healthcare decisions should rest. Failure to incorporate results of new research into practice can affect individual and population outcomes. This is the main reason for updating any medical information sources such as systematic reviews, guidelines, and clinical summaries.

Comprehensive presentation of new findings from research against the background of what is already available is essential to meet doctors’ needs for evidence during clinical consultations: which interventions work, which don’t work, which are additional or alternative, which need more investigation, and which might be harmful. For internet based information in particular, doctors and health professionals expect to rapidly find the latest knowledge to answer their information needs.

Point of care information summaries are web based compendiums designed to provide health professionals with comprehensive evidence condensed into easily digestible formats. The innovative aspect of these tools relies on how the information is engineered to be used at the point of care, when patient and practitioner interact. Point of care content is logically grouped around common medical scenarios and translated into alternative options related to diagnosis, treatment, and management.3 Publishers encourage physicians to use them during consultations or as a second opinion in their clinical decision making.

To make them attractive to final users, all publishers claim these products are regularly updated. Some even make direct reference to the dynamic incorporation of the latest evidence in their commercial names. To determine how long it takes for the latest research findings to make their way into a point of care summary we conducted a bibliometric analysis to examine the speed of updating—that is, the time between a paper’s publication and its citation in a point of care summary. For this analysis, we considered only papers with implications relevant to practice.

Methods

Out of 18 point of care information services available in 2008, we selected Clinical Evidence, Dynamed, EBM Guidelines, eMedicine, and UpToDate. These were ranked in the top quarter for at least two desirable dimensions: coverage of medical conditions (volume) and editorial quality and evidence based methodology.3 Our reasoning was that updating is a further desirable dimension of point of care summaries on top of others, and it would have been useless to look at the updating speed of products that were suboptimal in other dimensions on the basis of our evaluation. The decision to limit our analysis to the top ranking summaries reflected the aim of our research, which was to help users select one product over others.

For each of the five point of care information summaries we collected data on the updating mechanism by examining the free access web pages and sending emails to the information request service and editorial teams, as needed. This cross sectional qualitative analysis was done only once, in December 2009.

To evaluate updating speed we measured the incidence of research findings cited in point of care information summaries on potentially eligible newsworthy pieces of information. As samples of information relevant to practice we chose systematic reviews, which aim to provide a comprehensive appraisal of evidence. Findings from a single clinical trial are often rapidly contradicted by subsequent studies and low bias systematic reviews could help to get closer to the unknown “true evidence.”4 5 6 Systematic reviews have also gained acceptance as a starting point in the development of evidence based clinical practice guidelines.7 High quality systematic reviews are used more and are rated more highly by physicians in terms of relevance to clinical practice than other designs of articles.8 Four of the five point of care summaries we included clearly give priority to systematic reviews (and in general a cumulative approach to evidence) than other types of publications.3 We were not able to retrieve this information for eMedicine.

We selected all the systematic reviews signalled by the American College of Physicians (ACP) Journal Club and Evidence-Based Medicine Primary Care and Internal Medicine from April to December 2009. These two literature surveillance journals survey a wide range of international medical journals, applying strict criteria for the quality and validity of research articles. Practising clinicians assess studies that meet the basic validity criteria for relevance of clinical implications for practice and newsworthiness and a summary is then produced for the top rated articles. In the same period (April to December 2009) we selected all the Cochrane systematic reviews labelled as “conclusion changed” in the Cochrane Library. These are new citation versions of updated reviews that warrant additional highlighting in the Cochrane Library (for example, with a flag), indicating that they should be read again.9 We assumed that this sampling frame was representative of systematic reviews that meet explicit quality standards and are deemed directly relevant to clinical practice.

To evaluate how fast point of care summaries are updated we used a prospective cohort design over a one year period from June 2009 to May 2010. The follow-up started two months after the collection period to allow the potential citation of the most recent systematic reviews. Two reviewers independently checked whether each sampled systematic review was cited in at least one chapter of the five point of care information summaries. This was done monthly at the same time for each product. Disagreements were resolved by discussion between the two reviewers.

For each systematic review we defined “birth” as the publication date in one of the two literature surveillance journals or in the Cochrane Library and “death” (that is, event) as its citation in the monitored summaries. When the two reviewers agreed on the inclusion of that evidence in a summary the follow-up for that systematic review was terminated by the event. We censored systematic reviews when they had not been cited by the end of follow-up or if there was clear evidence that the topic was not covered by a given summary, similar to losses at follow-up in survival analyses. Two independent reviewers defined loss to follow-up. We excluded citations in additional reference lists, such as further or external readings and alert systems. We kept an archive of all the reference web pages citing the sampled systematic reviews.

We did not attempt any formal sample size calculation because information about the baseline incidence rates of citation was not available. Instead, we conducted an interim analysis after six months to determine the length of the collection period (that is, a small difference would have required an extended collection period and hence more systematic reviews). At the interim analysis we found substantial differences between the top performer and the other summaries, dramatically boosting the power of the study. The collection period was then stopped at nine months (December 2009).

We assessed the cumulative rate of updating using Kaplan-Meier survival analyses. As there were substantial differences between the top performer and the other summaries, we calculated the hazard ratios and 95% confidence intervals for each comparison using a univariate random Cox model. As we conducted an interim analysis to drive the length of the collection period, P≤0.025 was considered significant.

We further explored whether systematic reviews were more likely to be cited by the point of care information summaries on the basis of their source (literature surveillance journals or the Cochrane Library). As we observed different patterns of citation between the two second to top summaries, we compared the proportions of systematic reviews retrieved from literature surveillance journals or the Cochrane Library in these two summaries. Because this exploratory analysis did not aim to compare the citation rates but only the proportions, we used logistic regression and have reported the results as odds ratios.

Results

Table 1 describes the updating mechanism for product. For EBM Guidelines information was obtained after contacting the editors by email, while for eMedicine we were unable to retrieve any details on updating. Clinical Evidence declares a target updating cycle of one year and alerts readers of each specific chapter about potentially relevant new publications, providing links to the full reference (BMJ Updates). These alerts, however, are not inserted in the chapters or evaluated together with the existing body of evidence. EBM Guidelines, UpToDate, and Dynamed refer to “a continuous update,” meaning that new research findings are incorporated into the summaries every time they are published. UpToDate is the only product that clearly reports quantitative data on the topic updated (35% of all contents during a four month cycle).

Table 1

 Description of updating mechanisms reported on website of each point of care information summary

	Description of updating policy
Clinical Evidence (www.clinicalevidence.com)	“We aim to update Clinical Evidence reviews annually. In addition to this cycle, details of clinically important studies are added to the relevant reviews throughout the year using the BMJ Updates service. BMJ Updates is produced by collaboration between the BMJ Group and the internationally acclaimed McMaster University’s Health Information Research Unit to provide clinicians with access to current best evidence from research. All citations (from over 110 premier clinical journals) are rated by trained researchers for quality, and then rated for clinical relevance, importance and interest by at least three members of a worldwide panel of practicing physicians. The final content is indexed by health professionals to allow news of studies to be added to all relevant Clinical Evidence reviews.”
Dynamed (www.ebscohost.com/dynamed/)	“The final step in DynaMed’s evidence-based methodology is changing conclusions when new evidence alters the best available evidence. This step is crucial because new evidence is published every day. Having new evidence summaries handled separately from reviewed content in a manner requiring the clinician to search in two locations to synthesize the entire story would make finding the best available evidence more difficult. As soon as new evidence is evaluated using the 6 steps governing systematic processing, it is added to the appropriate DynaMed topic(s) in context. This process allows immediate and comprehensive access to the best available evidence as it occurs. This process occurs EVERY DAY in DynaMed.”
EBM Guidelines* (www.ebmg.wiley.com)	“Since the first electronic version was published in 1989 the contents of the database have been continuously updated. Over the years the guidelines have been extensively reviewed and even rewritten several times to include mounting evidence from clinical studies, comments by external referees, and feedback that has been collected systematically from clinicians who use the database in their daily practice. There are four updating processes that complement each other: (1) All guidelines are sent to authors and external reviewers every 2 years for systematic updates; (2) The editorial board meets once a month, and at every meeting, one speciality or a group of topics are discussed with 1-3 top experts on the field invited to attend; (3) The editorial team produces and updates evidence summaries continuously, and whenever the evidence summaries give rise to updates to the guidelines, the guidelines are updated; (4) The editorial teams of the translated versions of EBM Guidelines systematically check for updating needs. Updated parts of the text appear in red colour for a minimum of 6 months after the update was made.”
eMedicine (www.emedicine.medscape.com)	No detailed information on updating policy is reported on website or provided by publisher
UpToDate (www.uptodate.com)	“UpToDate performs a continuous comprehensive review of the resources listed above (peer-reviewed journals, clinical databases, etc.) in order to keep the program updated. Topics in UpToDate are revised whenever important new information is published, not according to any specific time schedule. Updates are integrated carefully, with specific statements as to how the new findings should be applied clinically. Each topic has a date indicating when the topic was last reviewed and/or modified. On average, approximately 35% of the topics are updated during each four-month cycle. A subset of those updates can be viewed by searching on What’s New and then selecting your specialty or area of interest. These updates represent, in our editors’ view, the most important new information added during the previous four months. They include Practice Changing UpDates, a compilation of studies with important or immediate implications for how clinicians practice.”

*From editorial team.

From April to December 2009, we retrieved 128 systematic reviews, 68 from the two literature surveillance journals (53%) and 60 (47%) from the Cochrane Library. The complete list is available in the appendix on bmj.com. Overall, 114 systematic reviews (89%) had been cited by at least one point of care summary. The median follow-up time was 33 weeks (range 1-60). Table 2 reports the proportions of citations by summaries over time and the hazard ratio for each summary compared with the top performer. Dynamed has an updating process that markedly led the others (fig 1). For instance, the hazard ratios for citation for EBM Guidelines and Clinical Evidence versus the top performer were 0.22 (95% confidence interval 0.17 to 0.29) and 0.03 (0.01 to 0.05), respectively. This means that the updating speed of Dynamed is 78% and 97% greater than those of EBM Guidelines and Clinical Evidence, respectively. The median time to citation was 7.7 weeks (range 7-8.2) for Dynamed and 42 weeks (range 34-maximum not reached) for EBM Guidelines. Dynamed has a median citation rate of around two months, EBM Guidelines is around 10 months but quite close to the limit of our follow-up. The citation rate of the other three point of care summaries (UpToDate, eMedicine, Clinical Evidence) were so slow that they exceeded the follow-up period and we could not compute the median.

Table 2

  Proportions of citations of 128 systematic reviews by point of care summaries over time (ordered by ranking at nine months) and hazard ratios between top performer (Dynamed) and other summaries

Summary	At 3 months (%)	At 6 months (%)	At 9 months (%)	HR (95% CI)
Dynamed	77	84	87	Reference
EBM Guidelines	18	31	41	0.22 (0.17 to 0.29)
UpToDate	23	27	29	0.14 (0.09 to 0.21)
eMedicine	7	9	12	0.05 (0.03 to 0.09)
Clinical Evidence	0	1	4	0.03 (0.01 to 0.05)

Fig 1 Updating curves for relevant evidence (128 systematic reviews) by point of care information summaries (log rank χ2=404, P<0.001)

Dynamed was also the first when we separately analysed the updating rate for systematic reviews retrieved through the Cochrane Library (fig 2) and the literature surveillance journals (fig 3). The two second point of care summaries (EBM Guidelines and UptoDate) had similar updating rates when we considered the whole sample of systematic reviews but differed when we took the origin of the systematic reviews into account. Cochrane systematic reviews were more likely to be cited by EBM Guidelines than by UpToDate (odds ratio 0.02, 0.01 to 0.10; P<0.001, logistic regression). EBM Guidelines has a formal agreement with the Cochrane Collaboration to use Cochrane contents and label its summaries as “Cochrane inside.”

Fig 2 Updating curves of Cochrane reviews (n=60) by point of care information summaries (log rank χ2=300, P<0.001)

Fig 3 Updating curves of non-Cochrane reviews (n=68) by point of care information summaries (log rank χ2=188, P<0.001)

Discussion

Evidence held to be relevant to clinical practice is inserted at different rates in point of care information summaries, and these products vary widely in their speed at updating content. Our citation analysis showed that Dynamed clearly dominates the other products (Clinical Evidence, EBMGuidelines, eMedicine, and UpToDate). Slowness in updating could mean that new relevant information is ignored and could thus affect the validity of point of care information services. Ultimately, whenever the transfer of relevant information is inappropriately slow, this can affect the care of patients, potentially denying treatments of proved benefit. This happens despite the fact that many of these products promote themselves to the clinical community as being regularly updated with the latest evidence.

When should point of care information content be updated?

A few studies have looked into strategies for updating clinical guidelines2 10 11 12 13 and systematic reviews,14 15 but no definitive conclusions have been reached on the best approach. A bottom line common to these studies was that updating is costly and time consuming. As far as we know, no data are available on how quickly point of care information content is updated and so publishers seem to adopt empirical approaches in managing their updating schedule. Even without an optimal approach, the updating of point of care information summaries should be evaluated bearing in mind that these online tools are largely intended to be used by an audience sensitive to brand new information.

Reasons for different updating speeds

Differences in updating ability are possibly justified by different approaches to content development. According to Shekelle et al, the updating process is based on two phases: identifying important new evidence and assessing whether it offers new information that might change recommendations for clinical practice.2 In addition, a third phase exists in which the new evidence should be included in the “old” body of knowledge. Citing a single trial or a systematic review without appraising and interpreting this new evidence in the light of existing knowledge is not enough.16 In other words, updating is not only a matter of literature surveillance but implies a critical evaluation of what a new item of knowledge adds to other works and what that means for clinical practice.6

Referring to these three phases, do these point of care information summaries differ in their approaches? Some of the products we analysed identify important new evidence by regular systematic searches or active surveillance of published journals and other information sources (such as reports from drug regulatory agencies, public health entities, World Health Organization, etc). In this phase we detected no major differences between products. How this new evidence is deemed relevant and then incorporated into the body of the summary probably largely dictates the different updating speeds. In Dynamed, the top ranked summary, updating is done centrally by the editorial team (supported by McMaster University’s Health Information Research Unit since the end of 2010), and this might make for more prompt inclusion of evidence. In Clinical Evidence, one of the lowest ranked, the authors of chapters are involved and often a new peer review process is required (R Minhas, editor of Clinical Evidence, personal communication). This is time consuming so content is likely to be updated more slowly or, in the worst case, to simply become out of date. In 2009, the BMJ Group launched the BMJ Best Practice product by engineering the contents of Clinical Evidence to fit the purpose of better use at the point of care, but we did not include it as it was not evaluated in our previous work.3 As little information on updating mechanisms was available for some summaries, our ability to further explore possible differences in updating approaches is limited. Publishers should fully elucidate information about their updating mechanisms.

Limitations

We chose a citational approach to measure updating speed, though there are shortcomings with this approach. Firstly, the total number of citations in the point of care information products should have been taken into account. Secondly, citational analysis counts only bibliographic references without going deeply into the content of the citation. This criticism, widely raised when citational analysis is used to evaluate scientific productivity and quality,17 18 also applies to our assessment. We did not attempt to go beyond the empirical number of citations found. In fact we did not judge the appropriateness of the update but simply used the updating speed as its proxy. Qualitative analysis of the updating process and how new evidence is incorporated and affects recommendations should also be taken into account in assessing whether one summary is better than others. Thus we cannot say that Dynamed is superior to the other products in terms of the appropriateness of the updating process or that Clinical Evidence compensates the limitations of its updating speed by offering deeper and more insightful updating.

We did not directly assess how many systematic reviews in our sample called for a change in clinical practice as we assumed that all our sources (ACP Journal Club, Evidence-Based Medicine, and the Cochrane Library) highlight newsworthy and relevant evidence through well established selection processes. Furthermore, these are considered authoritative international networks that close the gap between medical literature and clinical practice. We cannot exclude that relevant messages requiring urgent action might be given priority by the publishers, thus preserving the quality of the point of care summary. We chose Cochrane systematic reviews with “conclusion changed,”—that is, those that should be read again.9 If a point of care information summary still cited the old version of the Cochrane systematic review this was considered not updated, regardless of the nature and impact of the change in conclusions. We believe this conservative approach, which might have partially influenced the citing speed of Cochrane systematic reviews, was appropriate as knowing that a Cochrane systematic review has been updated could be important for readers. We are also aware that we relied on sources of pre-appraised evidence mainly devoted to general practice and internal medicine, though specialty topics are provided too (see appendix on bmj.com). Finally, we did not consider the updating of results from studies with other designs (such as randomised clinical trials) as we think that systematic reviews are preferable to support decision making at the point of care.

Conclusions

Updating is only one aspect of the overall quality of a point of care product. Other studies have assessed other dimensions such as user’s satisfaction, how well different online point of care services answered questions arising in daily clinical work, content development, and evidence based soundness.19 Findings from both user and content centred analyses need to be combined if one has to choose one product rather than another. Readers should be aware that point of care information summaries vary widely in their updating ability, and, in some cases, it might be unsatisfactory in relation to what users expect and what publishers advertise. In the context of an editorial market with rapidly evolving point of care summaries, our specific intent was to provide a snapshot assessment of the updating speed of point of care services with recently published relevant systematic reviews. The quantitative findings should be considered together with a qualitative analysis of updating methods that can be done only if updating mechanisms are described in greater detail by publishers.

The process leading from evidence to clinical recommendation and then to changes in behaviour is affected by many factors besides access to the latest studies.20 21 22 Nevertheless, appropriate promotion of progressing evidence is essential to provide patients with better healthcare.

Few studies have compared the quality of point of care summaries

Most looked at user satisfaction as the outcome measure and none evaluated their speed of updating and appropriateness

Point of care information summaries insert latest evidence relevant to practice at different speeds