Evidence-based medicine for all: what we can learn from a programme providing free access to an online clinical resource to health workers in resource-limited settings.

The rapidly changing landscape of medical knowledge and guidelines requires health professionals to have immediate access to current, reliable clinical resources. Access to evidence is instrumental in reducing diagnostic errors and generating better health outcomes. UpToDate, a leading evidence-based clinical resource is used extensively in the USA and other regions of the world and has been linked to lower mortality and length of stay in US hospitals. In 2009, the Global Health Delivery Project collaborated with UpToDate to provide free subscriptions to qualifying health workers in resource-limited settings. We evaluated the provision of UpToDate access to health workers by analysing their usage patterns. Since 2009, ∼2000 individual physicians and healthcare institutions from 116 countries have received free access to UpToDate through our programme. During 2013-2014, users logged into UpToDate ∼150 000 times; 61% of users logged in at least weekly; users in Africa were responsible for 54% of the total usage. Search patterns reflected local epidemiology with 'clinical manifestations of malaria' as the top search in Africa, and 'management of hepatitis B' as the top search in Asia. Our programme demonstrates that there are barriers to evidence-based clinical knowledge in resource-limited settings we can help remove. Some assumed barriers to its expansion (poor internet connectivity, lack of training and infrastructure) might pose less of a burden than subscription fees.

Evidence-based clinical resources (EBCRs) are widely used by clinicians in high-income countries to answer clinical questions during patient care.

UpToDate, a leading EBCR, has been linked to average length of stay and decreased mortality in US hospitals.

Adoption of EBCRs in resource-limited settings (RLS) has lagged behind.

Removing the cost barrier to accessing UpToDate, a leading EBCR, leads to frequent use of the resource by a large and diverse set of clinicians practising in low-income and middle-income countries.

Clinicians practising in RLS seek evidence-based information across a wide spectrum of topics, with infectious disease and paediatrics capturing the largest share of interest.

Recommendations for policy

Access to evidence based clinical resources for clinicians in all settings is an important component of improved health care delivery. Enhancing access may include removal of cost barriers as well as technological barriers, such as Internet connectivity.

Policy should support the development and production of EBCR content specific to LMIC health priorities to ensure relevancy and utility.

Introduction

All health professionals face the daily challenge of incorporating a vast and rapidly evolving body of medical knowledge into their clinical practice. In 2014, providers were faced with consuming new evidence from over 5000 clinical trials.1 Alarmingly, preventable medical errors, which result partly from lack of access to or understanding of the best available evidence, are common across the world: A 2012 study of over 15 000 medical records from 26 hospitals in Africa and the Middle East showed that 6.8% of all hospitalised patients experienced a medical error and one-third of them died as a result.2 In 2004, investigators in The Lancet asked, ‘Can we achieve health information for all by 2015?’ and, in 2006, the WHO identified access to information as a critical step towards improving population health.3 4

In response to this, private and public entities in high-income countries created numerous online evidence-based clinical resources (EBCRs) over the past decade that distilled research into guidelines for therapeutic choices and care (table 1). Uptake of such resources has been extensive among health professionals in several high-income countries: among 16 122 health professionals surveyed across 181 hospitals in the USA and Canada about clinical resource usage in the past 6 months, 53% of attending physicians and 77% of residents had used UpToDate—one of the leading EBCRs; 59% of physicians used online journals, 20% eMedicine and 24% Micromedex.5

Table 1

Most-used online evidence-based clinical resources (EBCRs) by Global Health Delivery UpToDate (GHD UTD) programme applicants

Resource	Description	Cost per year*
Medscape	WebMD-owned site of original medical content authored by physicians	Free
PubMed	Database of abstracts and citations maintained by the US National Library of Medicine	Free (abstracts only)
HINARI	WHO programme to provide free or low-cost access to over 15 000 journals and 47 000 e-books for health workers in specific countries	Free or $1500 depending on country
Cochrane	Global network of health professionals who compile evidence and author practice guidelines for clinicians	Free 12 months after publication†
New England Journal of Medicine	Medical journal published by the Massachusetts Medical Society	$79‡
Dynamed	Clinical reference tool written by physicians and aimed targeted at health professionals	$395
BMJ	Medical journal published by the British Medical Association	Free
The Lancet	British Medical Journal	$174‡
UpToDate§	Continuously updated, expert-authored clinical content targeted at health professionals. Includes information on drugs and drug interactions	$499

*Indicative cost per year for an individual US-based physician. Source: resource websites, accessed on 1/10/2016.

†New Cochrane Reviews become free 12 months after publication. Entire Cochrane Library to become free by 2020, according to strategic plan.

‡Online-only version.

§UpToDate was not one of the most used online EBCRs by GHD UTD programme applicants but is included here for comparison. A detailed description of the most used EBCRs can be seen in figure 1D.

Figure 1

Description of applicants to GHD UTD programme. (A) Cumulative number of total applications to GHD UTD programme by continent (including individual and institutional applications). ‘Application analysis’ refers to application data collected between March 2013 and December 2015, which was used for the analysis in figure 1C, D. ‘Usage analysis’ refers to usage data collected between January 2013 and December 2014 and was used for the analysis in figures 2 and 3 and table 3. (B) Map of the world showing country chosen by applicants as ‘main location of practice’ and number of associated applications from each country.(C) Location of primary medical education of applicants as specified in application. (D) Use of other evidence-based clinical resources as specified by first-time applicants in application. GHD UTD, Global Health Delivery UpToDate donation programme.

Recent research has demonstrated a connection between use of EBCRs and patient outcomes: Use of UpToDate by physicians practising in the USA was shown to increase performance in standardised examinations,6 and, most importantly, a 2012 US study showed that use of UpToDate at the hospital level reduced patients' average length of stay and mortality.7

Recognising the proven importance of EBCRs, several initiatives have opened access to clinical, scientific and research content via online databases either for free or at a low cost (eg, HINARI to institutions in low-income countries), while others disseminate information from publicly funded research (eg, PubMed), or release full-text articles selected by the publishers or the authors. Open-access publishers, such as BioMed Central, BMJ Open, and organisations such as the Public Library of Science (PLoS) are dedicated to making the biomedical and scientific literature freely accessible and have developed new models of content creation and distribution.8 While important, these efforts offer a literature base that is often not exhaustive, and physicians report that a full article can be difficult to navigate, or even irrelevant, during a clinical encounter.9 In addition, some websites including Wikipedia, Medscape and Epocrates offer free clinical content, but, as others have discussed, the quality, comprehensiveness and timeliness of this content varies.10

In the midst of an ‘information age’, global health providers often lack access to the latest practical, lifesaving information.11 12 Getting information to the people in the places where it is most needed is a delivery challenge. We hypothesise that contributing factors may include lack of reliable internet connections, low rates of internet-capable device ownership and the high subscription costs of many EBCRs.13 14 An individual UpToDate subscription for a US-based medical professional costs $499 per year, while a subscription to the New England Journal of Medicine costs $79 per year (table 1). In 2013, Tanzania spent $49 per person on healthcare (private and public expenditure combined).15 Hence, the country would need to spend its entire healthcare budget for 10 people to purchase one individual UpToDate subscription.

As part of the effort to address this information bottleneck, in 2009, the Global Health Delivery Project at Harvard University (GHD) launched a pilot programme with UpToDate to grant free subscriptions to UpToDate content to physicians and health workers in resource-limited settings (RLS). The programme, called the Global Health Delivery UpToDate donation programme (GHD UTD) grew rapidly over the past 6 years. Analysing UpToDate usage data and information from subscription applications, we provide here an overview of the programme. The aim of this analysis was (1) to explore whether the removal of the cost barrier can catalyse uptake of EBCRs among global health professionals, (2) to examine how frequently global health professionals who obtain free access to UpToDate log in and (3) to describe the medical topics of interest among grant recipients.

Programme overview

Subscription application and screening

GHD used its existing online platform, GHDonline.org—a virtual professional community of over 17 000 healthcare delivery professionals worldwide—for grant administration. In order to be eligible for a 1-year free subscription to UpToDate, applicants had to: (1) deliver medical care, medical education or related services to poor or underserved populations outside the USA, (2) have at least intermittent internet access, (3) be able to provide evidence that they or their organisations could not afford to subscribe otherwise and (4) be proficient in English. The last condition was necessary as UpToDate content is currently available only in English.

Qualified applicants filled out an online application on GHDonline that included questions about themselves, their organisation and the patient population they serve. Applicants submitted a letter stating why they needed free access to UpToDate and a reference to confirm the veracity of their responses. Applicants also chose between an individual subscription (associated with a unique username and password) and an institutional subscription (associated with a unique IP address so that anyone on a particular institution's network could access it). GHD and UpToDate staff reviewed all applications to ensure they were complete, accurate and met eligibility criteria. UpToDate staff then sent an email to all approved applicants containing their username and password (for individual subscriptions) or instruction on how to set up an institutional subscription. All individual subscriptions came with the option to download ‘Mobile Complete’, an offline mobile version of UpToDate that allowed users to browse content without a connection to the internet.

During their 1-year subscription, we asked users to answer 12 questions on the GHDonline online community pertaining to their experience with UpToDate, as well as offer any feedback they had for the programme. At the end of their subscriptions, we invited all users to submit a brief renewal application to extend their subscription if they wished to do so.

Outreach and dissemination

We promoted the GHD UTD collaboration to global health professionals through a page on GHDonline and through an email to all GHDonline users. In addition, UpToDate created a link on the company website (http://www.uptodate.com) informing visitors of the existence of this programme for health professionals in RLS and a promotional video that was also posted on the UpToDate website. Lastly, The Boston Globe wrote an article describing the programme.16

Information collected

Application data: We collected demographic information about participants and their organisations using an online Google Form application as part of their application for a free UpToDate subscription. We anonymised data by deleting all identifying information from the data set before conducting any research activities. Informed consent for the use of these data was not obtained from applicants, as the Harvard University Institutional Review Board (IRB) deemed this project exempt from full review due to the anonymous nature of the data. In this analysis, we present application data from the period 2009–2015. In March 2013, the application was changed to include questions on the location where applicants received their medical education, as well as the names of other EBCRs they regularly used. Hence, data in figure 1B, C represent applications received between March 2013 and December 2015.

Figure 2

Patient case vignettes from GHDonline private community. ARDS, acute respiratory distress syndrome; GHD, global health delivery; US, ultrasound; UTI, urinary tract infection.

Figure 3

Frequency of usage and topics of interest among GHD UTD users. (A) Histogram showing percentage of users accessing UpToDate at specified frequencies. (B) Share of total usage by topic of medical specialty. Each topic accessed was assigned to a medical specialty using UpToDate's table of contents. GHD UTD, Global Health Delivery UpToDate donation programme; GI, gastrointestinal; Ob/Gyn, obstetrics and gynaecology.

Table 3

Most read topics among Global Health Delivery UpToDate (GHD-UTD) users by continent

	Africa		Asia		North America		South America
Rank	Term	Frequency	Term	Frequency	Term	Frequency	Term	Frequency
1	Clinical manifestations of malaria	404	Overview of the management of hepatitis B and case examples	154	Epidemiology, microbiology, clinical manifestations and diagnosis of typhoid fever	161	Pre-eclampsia: clinical features and diagnosis	57
2	Treatment of severe falciparum malaria	394	Overview of hypertension in adults	147	Metronidazole (systemic): drug information	156	Epidemiology, pathogenesis and clinical manifestations of Ebola and Marburg virus disease	41
3	Initial assessment and management of acute stroke	353	Overview of medical care in adults with diabetes mellitus	138	Trimethoprim-sulfamethoxazole (co-trimoxazole): drug information	151	Clinical manifestations and diagnosis of acute pancreatitis	38
4	Overview of the therapy of heart failure due to systolic dysfunction	348	Serologic diagnosis of hepatitis B virus infection	133	Treatment regimens for Helicobacter pylori	143	Shock in adults: types, presentation and diagnostic approach	35
5	Overview of hypertension in adults	347	Treatment and prevention of typhoid fever	127	Tetanus	133	Chronic otitis media, cholesteatoma and mastoiditis in adults	35

Usage data: UpToDate provided the authors de-identified usage logs of all active GHD UTD users during the period 2013–2014. These logs only indicated the country of origin of the user if there were at least five subscriptions from that country to preserve anonymity. Logs tracked all online and offline activity of users on any device (searches, clicks, reading of pages/figures, etc).

Ebola data: To compare the search behaviour of GHD UTD programme participants to that of the general public, we retrieved global Google search data from Google Trends on 11/28/2015 (https://http://www.google.com/trends/explore#q=ebola) and data on the number of confirmed Ebola cases from the US Centers for Disease Control website (http://www.cdc.gov/vhf/ebola/outbreaks/2014-west-africa/cumulative-cases-graphs.html).

Usage vignettes: We reviewed qualitative data from one of the GHDonline.org questions asking users to describe a patient case where UpToDate was helpful in determining the correct diagnosis or treatment plan.

Data analysis

We analysed usage data in terms of ‘sessions’. We defined a ‘session’ as a time period using UpToDate, initiated by a unique log-on of a user to the UpToDate website (on a computer), the UpToDate mobile site, or the UpToDate mobile application (on a mobile device) and terminated when the user actively logged off, closed the application or remained inactive for more than 3 hours. Since of hacking attempts known to UpToDate to download large amounts of UpToDate content using computer software, we excluded from our analysis any sessions in which more than 10 actions were taken in the span of the last 100 ms of the session duration (indicating faster activity than a human can normally initiate). We also excluded from our analysis one institutional account, which exhibited abnormally high activity during 2 months of the study period and was determined by UpToDate to be likely hacked. Average daily usage frequencies per user were calculated as total number of user sessions over study period divided by the number of days with an active account. Subsequently, average usage was assigned to frequency categories as shown in table 2.

Table 2

Explanation of frequency category assignments

Average daily usage (ADU)	Frequency category assigned
ADU>1	Once a day
1≥ADU>1/7	Once a week
1/7≥ADU>1/30	Once a month
1/30≥ADU	Less than once a month

Users who accessed UpToDate once only during their subscription were assigned to the ‘one time-only’ category.

For the topic analysis, we used a table of contents provided by UpToDate, which assigned each topic title to a particular primary medical specialty. We counted topics appearing more than once in the same session once. Medications are not classified into primary medical specialties by UpToDate, so we excluded all articles covering specific medications from this analysis as well as articles that had been deleted from UpToDate or whose title had changed at the time of the analysis. In the end, we included 390 595 article topics out of 494 650 total articles accessed in this analysis.

We used Microsoft Excel and Stata SE V.14 for all data analyses.

Results

Applicant demographic characteristics

Between its inception in 2009 and the end of 2015, the Global Health Delivery UpToDate donation programme (GHD UTD) received 1948 unique applications for access. The programme's growth picked up significantly after 2013, potentially due to the growing awareness of it (figure 1A). Approximately 82% of applications received during the study period met eligibility criteria and were awarded a 1-year free subscription to UpToDate. Of applications considered, 94% came from individuals and 6% from institutions. Institutions had a wide range of affiliated physicians (from 5 to 300 clinicians per institution). Applications represented a total of 116 countries from six continents (figure 1B). The plurality of applicants (36%) completed their medical education in Africa (figure 1C). Almost half of the applicants (48%) indicated that they had access to no other online evidence-based resources, while some mentioned accessing Medscape (22%), PubMed (7%) and HINARI (4%; figure 1D and table 1).

Usage frequency

During the period 2013–2014, 45 institutional and 405 individual GHD UTD users logged on to UpToDate for ∼150 000 unique sessions. Usage rose steadily during the study period. Patient vignettes suggest that UpToDate proved useful as a clinical resource. (With permission from the authors, please see four such clinical vignettes in figure 2). Users from Africa accounted for 54% of the usage. Across the world, the top countries were Rwanda (19% of total usage), Haiti (10%), Zimbabwe (8%) and Mozambique (8%). The 45 institutional accounts were responsible for 55% of total usage. On average, 20% of all users logged on at least once per day over the duration of the study, and 41% of users logged at least once per week (figure 3A).

Topics of interest

Medical specialties that attracted a particularly high interest among our users included infectious disease (14% of total usage), paediatrics (14%), obstetrics and gynaecology (6%), and gastrointestinal medicine (6%; figure 3B). In table 3, we list the five most popular topic titles in each category and the frequency with which they were accessed over the 2-year period of the programme.

Lastly, to assess user interest in Ebola during the global outbreak, we plotted the search volume for the term over 2014. We compared it with the search volume for the same word on Google by the general public (figure 4). Interestingly, searches for ‘Ebola’ peaked earlier on GHD UTD than Google.

Figure 4

Search volume for ‘Ebola’ among GHD UTD users and general public. Cases: number of confirmed cases in Guinea, Sierra Leone and Liberia. Source: US CDC. Google: normalised worldwide search volume of the term ‘Ebola’ on Google. Source: Google Trends. UpToDate: normalised worldwide searches containing the word ‘Ebola’ among GHD UTD subscription recipients. GHD UTD, Global Health Delivery UpToDate donation programme; US CDC, US Centers for Disease Control.

Discussion

Our work demonstrates that removing the subscription fee barrier enabled almost 2000 health workers from 100 countries to access and use UpToDate, a leading EBCR. The fact that the majority of users accessed UpToDate very frequently (more than once a week) and read a wide spectrum of topics supports the hypothesis that there is demand for EBCRs in low-income and middle-income countries (LMICs). While users shared salient examples of how UpToDate helped them improve their clinical practice, it will be important for future research to test more formally the health outcome gains associated with EBCRs in RLS and whether the quality of care provided by global health professionals using EBCRs is better than the quality of those who do not. Measuring care outcomes, surveying patients and providers, and using clinical vignettes and audit patients to understand providers' knowledge and practice are possible ways in which these questions could be addressed.17

There are several limitations to our analysis that merit discussion: First, according to WHO estimates, there are more than 100 000 physicians working in low-income countries;18 yet, only 1948 health workers applied for access to our programme. Our programme did not address the demand side of the information bottleneck. A lack of training on use of EBCRs, absence of an evidence-driven culture in medical education and potential lack of relevance of existing EBCRs to local clinical challenges, are important to note and to explore in future research (table 4). The relevance, or lack thereof, of EBCRs in general, and UpToDate in particular, to LMIC clinical needs has not been evaluated: It is possible that UpToDate's recommendations on clinical care are different from, and even in conflict with, local practice and guidelines. It is also possible that UpToDate recommends the use of diagnostic procedures and therapeutic interventions that are unavailable in LMICs due to high cost or other constraints. In those cases, use of UpToDate might not be helpful to the practising clinician. As access to mobile and broadband internet across Africa and other developing regions increases and the prices of internet-capable devices fall, demand generation will be a more significant barrier. Potential interventions to increase demand could include introducing EBCRs in the context of medical education, using incentives or ‘nudges’ to promote use of EBCRs by clinicians, or creating locally specific EBCR content to increase its relevance.

Table 4

Our working hypothesis on contributing factors to the disparity in use of evidence-based clinical resources (EBCRs) between high-income and resource-limited settings11–14

Disparities in supply	Disparities in demand generation
Lack of reliable internet connections	Lack of training on use of EBCRs
Low rates of internet-capable device ownership	De-emphasizing of evidence-based medicine in medical education
High subscription cost of many EBCRs	Lack of relevance of existing EBCRs to local clinical challenges

Exploring what our users viewed on UpToDate can provide useful insights in the knowledge gaps of professionals in RLS and guide future attempts to create EBCR content targeted to those professionals. Topics such as heart failure, stroke, diabetes and hypertension were popular among all global health professionals, likely correlated with the rising burden and prioritisation of non-communicable diseases in RLS. It could also be argued that many national programmes have created clear and well-distributed guidelines for the treatment of priority infectious diseases such as HIV, malaria and tuberculosis, and hence, doctors are less likely to turn to UpToDate for information on the management of those diseases. One limitation of our study is our inability to separately analyse behaviour of those exclusively using UpToDate and those using UpToDate and other EBCRs. Our data sets were aggregated and anonymised to adhere to ethical standards.

How else can this usage data inform clinical delivery and public health? Investigators have recently shown interest in using online search log data to make epidemiological predictions. Researchers at Google showed that general public searches for flu symptoms and medicines can be used to accurately predict the spread of the influenza epidemic in the USA.19 Some have suggested that UpToDate search data may provide an even better prediction of the flu epidemic, as physicians are better positioned than the general public to identify appropriate searches for the presenting symptoms.20 21 These findings provide for the possibility of UpToDate search logs to serve as inexpensive epidemiological surveillance in RLS should the density of UpToDate-using providers in resource-limited geographies increase.

In 2013, the Institute of Medicine reported that only 54% of care provided in 12 large US cities followed existing Clinical Practice Guidelines—the best available evidence for the treatment of a particular disease at the time.22 Gawande23 suggested that changing clinical practice is a slow and cumbersome process that requires removal of barriers as well as person-to-person mentoring, talking and nudging. It is clear that cost is one barrier to EBCRs that will need to be removed to realise the vision for evidence-based medicine for all. We are hopeful that the next steps towards this vision will be bigger and broader and close the knowledge gap quickly.