Electronic referral systems in health care: a scoping review.

Introduction: Electronic referral (eReferral) systems have been designed with the dual purpose of decreasing wait times and improving workflow efficiency. Evidence about the clinical and economic value enabled through the use of eReferral systems is limited. Our objective was to review the evidence base for the effectiveness and cost-effectiveness of eReferral systems. This review is part of a bigger project to inform the economic benefits of a regional eReferral implementation program. Material and methods: A systematic search was conducted to capture the available literature on the effectiveness and cost-effectiveness of eReferral system interventions. Evaluation of eReferral system for cost or outcome(s) were included. Strictly e-consultation systems were excluded. We only included publications in English.
Results: We found 274 citations. After removing duplicates and conducting levels one and two screenings, nine publications qualified. Results were divided into four categories: cost or cost-effectiveness analysis, changes in workflow efficiency, the quantity of referrals, and the quality of referrals. A full economic evaluation, conducted in Denmark, found that an eReferral system was cost-effective compared with a paper-based referral system. Of the other eight studies, three demonstrated positive changes in referral processing; two evaluated changes in the quality of the referrals, and three evaluated if the eReferral system increased the quantity of referrals. Discussion: The evidence base on the effectiveness of eReferral systems to improve communication between primary care and specialists and to decrease wait times is positive but limited. Economic evaluations are needed to examine the clinical and economic value of eReferral systems in health care.

Introduction

Timely access to specialist care is critical to achieve better health outcomes and reduce health care costs for patients and the health system.1 Access to specialists depends on several factors, including a patient’s clinical condition, a patient’s age, the type and location of the referring clinic, a specialist’s availability, as well as the level of communication and type of information shared between primary care physicians and specialists in the referral process.2,3 Paper-based referrals by fax, the standard process in many practices, have been associated with referral and booking delays due to incomplete or missing information such as patient’s data, clinical laboratories, X-ray images.4

Electronic referral (eReferral) systems have been designed to improve wait times and efficiency by electronically standardizing information and communication within the referral process.5 According to Liddy (2015), electronic referrals allude to the automation of the referral process in which appointment and other information regarding the consultation are transferred between two or more health care providers.6 The results of a recent systematic review by Naserias et al (2015) of related e-consultation systems indicated considerable variability in success factors such as improving access to specialist care, reducing wait times, and improving quality of the referral process.7 The review identified 27 publications reporting on a mix of e-consultation program solutions and related eReferral systems (Norway, England, Denmark, The Netherlands, New Zealand). Most of these programs were implemented in single site settings (eg, hospitals) and have not been implemented at regional or national levels. Given the additional functions of e-consultation solutions, their evidence cannot be translated to the more common-place eReferral systems that do not support integrated telemedicine consultation. Compared to the literature on e-consultations (see, for example, review by Heimly et al, 2009), no review has focussed on the evaluation of eReferral systems from a cost and benefits point of view and it is currently unknown if eReferral systems are cost-effective strategies.8

Our objective was to describe and appraise the evidence-base on the effectiveness and cost-effectiveness of eReferral systems. This review is part of a bigger project to inform the economic benefits of a regional eReferral implementation program. A secondary goal of this review was to inform future evaluations of eReferral systems. Based on expert consensus, we hypothesized that the evidence-base for eReferral system interventions was limited and inconclusive.

Material and methods

Search strategy and inclusion/exclusion criteria

A search was conducted to identify literature reporting on the incremental cost and cost-effectiveness as well as changes in quality, quantity, and efficiency of referrals associated with eReferral systems from PCP to specialists compared to paper-based referrals in the general population. The search was conducted using PubMed and OVID databases (Embase since 1974 and OVID Medline Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) since 1946). Different combinations of MeSH terms and keywords were used to find the most relevant studies without date of publication restriction (see Table 1). Only publications in English were included. The search was conducted from October to December 2016 and updated in June 2018. In addition, a gray literature search was conducted in parallel to identify reports, books, and websites regarding evaluation of electronic referral systems compared with paper-based process. The gray literature review was performed using the Google search engine based on similar keywords and terms used in the PubMed and OVID database searches (Table 1).

Table 1

MESH terms and keywords

Search	Key term		Term 2		Complementary term
#1	Electronic referral	AND	-	OR	Referral
#2	Electronic referral		Cost		Referral/cost analysis
#3	Cost-benefit analysis		Electronic referral		Cost-effectiveness/referral
#4	e-referral	-	-

Selection process and data analysis

Studies retrieved by the search strategy were first screened for inclusion and exclusion criteria based on a review of the title and abstract by one reviewer (AA). To be included, studies had to evaluate an eReferral system for cost or outcome(s). Studies evaluating strictly e-consultation systems, studies not published in English, letters to the editor, and commentaries and opinions were excluded. Each document identified in the gray literature was checked against any publication reporting the same results. In that case, the journal article was included instead the gray literature document (for example, there was a report of e-referral program implementation in the San Francisco General Hospital that was published as an article,13). In a second step, two readers performed a full evaluation of the articles (AA and JET) for those studies selected at the first stage and were screened for final inclusion/exclusion. In case of differences in the selection for any article, both readers exposed their points and reached a consensus. References of the selected studies were also searched for inclusion/exclusion.

Data from these publications were extracted in a purpose-built data collection form. Key information abstracted included study characteristics (eg, country, year of study), study objectives, study design, and outcomes (cost or cost-effectiveness analysis; changes in quality of referrals, quantity, and efficiency of the eReferral process). Due to the nature of the studies, it was not possible to combine the studies using statistical methods (eg, meta-analyses). Rather, we described the studies on a narrative basis.

Results

Literature search results

The literature search, including the hand search of the reference lists of included papers, yielded 493 citations (377 from PubMed, 115 from OVID) of these, 205 articles were duplicates. After evaluation of titles and abstracts, we excluded 265 articles because they were not referring to e-Referral evaluations (see Figure 1). After a second screening of full texts (including the review update), eight publications were included in the review. In addition, we found one governmental report from the gray literature search that met the inclusion and exclusion criteria that also was included in the analysis. Four studies were conducted in the United States, two in Canada, one in the UK, one in Denmark, and one in New Zealand. The studies were published from 2004 to 2018. The majority of the studies (7/9) evaluated eReferral from primary care physicians (PCP) to hospital-based specialists, while the rest focused on PCP to non-hospital-based specialists.

Figure 1

Selection flow for published literature*.

Study design

Most studies (6/9) evaluated changes in referral patterns with the implementation of an eReferral system using surveys, questionnaires, and time-motion approaches. Only one study was a full economic evaluation that evaluated both costs (eg, handling or process costs, operational costs, and equipment costs) and benefits associated with the eReferral system. The remaining studies evaluated either the number of referrals, time to book an appointment, referral processing times, and provider satisfaction associated with eReferrals. While almost all studies reported the results using basic descriptive statistics (7/9), two studies conducted a regression analysis to estimate the impact of the eReferral implementation on the number of referrals. The study samples ranged from only a few referrals to those with more than 700,000 referrals. Tables 2 and 3 present more details on the characteristics of each study.

Table 2

Study characteristics

First author	Country	Objective	Study design	Sample characteristics	Referrals from–to	Type of outcomes
Cost or economic evaluation	Changes in quantity of referrals	Changes in quality of referrals	Changes in workflow efficiency or productivity
Cannaby9 (2004)	Denmark	To evaluate the cost-benefit of electronic referral system	Time-motion	41,235 referrals	General Practitioner to Hospital	YES	-	-	-
Yeuen Kim10 (2009)	US	To assess the impact of electronic referrals on workflow and clinical care.	Web-based questionnaire	298 General Practitioners	General Practitioner to Hospital	-	-	-	YES
Scheibe11 (2015)	US	To evaluate the use and impact of an electronic referral system in rheumatology		2,105 referrals	General Practitioner to Hospital	-	-	-	YES
Scott12 (2009)	UK	To evaluate time to process a referral	Interviews	99 referrals	General Practitioner to Hospital	-	-	-	YES
Kim-Hwang13 (2010)	US	To determine the impact of electronic referral on specialty referrals	Visit-based questionnaire	505 questionnaires	General Practitioner to Hospital	-	-	YES	-
Corwin14 (2014)	New Zealand	To assess whether feedback improves the quality of referral letters from general practice to secondary care and how electronic referrals affect the quality of referral letters.	Referral letters’ Feedback from peers	340 referral letters	General Practitioner to Hospital	-	-	YES	-
Chambers15 (2015)	US	To facilitate and sustain patient referral using an Electronic Health Record and a provider education program	Pre-post intervention analysis	75 referrals	General Practitioner to Diabetes program	-	YES	-	-
MacGregor16 (2009	Canada	To evaluate the completeness and adequate number of referrals using an electronic referral system	Pre-post intervention analysis	10 clinics (referrals number was not reported)	General Practitioner to Pediatric Hospital	-	YES	-	-
Doumouras17 (2018)	Canada	To determine the impact on referral rates to bariatric surgery after converting to an online referral system from a fax-based system.	Pre-post intervention analysis	5,317 referrals	All type of physicians or specialists to the Ontario Bariatric Network	-	YES	-	-

Table 3

Summary of main study findings and conclusions

Author	Main findings	Authors’ conclusion
Cannaby9 (2004)	Total health system annual savings of €3,512,146 associated with the implementation of the e-referral system. Annual potential savings in direct costs of €1,900,000. Compared with the paper-based referral process, the e-referral system saves up to € 1.6 in the average cost per minute spent during the referral process. Savings of €0.65 per capita if all the referrals are sent electronically.	The study suggests that significant cost savings are possible from the widespread adoption of electronic messaging in healthcare. Further research is needed to understand what the quantifiable benefits of electronic communication in healthcare are.
Yeuen Kim10 (2009)	72% of Primary Care Providers felt that electronic referrals have improved guidance of the pre-subspecialty visit workup 89% of participants respond the e-referral had improved the process overall. 54% of participants felt an improvement in wait times for new appointments. 79% of participants reported a positive change in the workflow and clinical care 7% were dissatisfied using the e-referral system	The majority of PCPs felt electronic referrals improved quality, access and clinical care. The impact of e-referral system ingenerate timely access to care should be considered for further analysis.
Scheibe11 (2015)	Pre-consultation exchange increased from 55% in 2008 to 74% in 2011 due to the use of e-referral system From the total pre-consultations, 63% were subsequently scheduled for a face-to-face appointment.	The e-referrals system allowed specialists to conduct a pre-consultation exchange to determine the need of a consultation. The e-referral system allowed for triage a high number of referrals.
Scott12 (2009)	81% of e-referrals were processed within one hour after received. The e-referral system improved satisfaction. From a 10-point Likert scale, the minimum score was 8.	Findings reported improvements in the speed with which referrals were sent to the central office as well as the speed with which they were processed once received
Kim-Hwang13 (2010)	The reason for referral was difficult to identify in 19.8% of medical visits using a paper-based referral compared to 11% in the eReferrals (p=0.03). 2.6% of the eReferrals were deemed not completely appropriate compared with 6.4% of the paper-based referral (p=0.21). Medical specialty clinicians indicated that they requested follow-up in 84% of paper-based referrals vs. 90% of e-referrals (p=0.06). Surgical specialists indicated that they requested follow-up in 76.2% (paper-based) vs. 58.1% (e-referral).	The e-referral system helped specialists to identify the clinical question for referral. Inappropriate referrals were lower using the e-referral system compared with the paper-based process The adoption of the e-referral system was associated with the reduction of unnecessary follow-up for patients.
Corwin14 (2014)	The implementation of an e-referral system did not lead to improvements in the quality of referrals compared with the paper-based process (p=0.20) Feedback from peers (GP’s) improved the quality of referrals from 81.4% to 86.9% of adequate referrals (p-value 0.05)	Feedback from peers to GP seems to have a better effect in the quality of referrals than the adoption of an electronic referral system.
Chambers15 (2015)	On average per month, patients referred increased from 1 to 3 after the e-referral system implementation Referrals increased from 3 to 6 on average per month after the e-referral system implementation combined with the provider education program.	The segmented regression analysis showed that the increase in the number of referrals was significant when the e-referral system and the provider education program were combined and not significant when the e-referral system was evaluated alone.
MacGregor16 (2009	Before the implementation of the electronic referral system, 30% of referrals were either incomplete or inappropriate (before the implementation of the e-referral system). After the implementation of the electronic referral system, incomplete referrals increased in 2.7% and online referrals increased 6%.	The e-referral system seems to have benefits regarding the incompleteness of referrals. Authors mentioned that other benefits can be: increase workflow efficiency, ensure patient´s information confidentiality among others.
Doumouras17 (2018)	Referrals increased (overall) by 68% after the e-referral system implementation. GP that are female, live within 50k of the Health Center and with less than 5 years of experience, has a greater odds to refer using the electronic referral system	Compared with the paper-based referral process, the implementation of an e-referral system increases the number of referrals (significant).

Economic evaluation of the eReferral systems

Only one study (1/9) was a full economic evaluation comparing the costs and outcomes associated with eReferrals in a cost-benefit analysis. In this study, Cannaby et al (2004) compared an eReferral system with a paper-based referral process in Denmark to estimate the time and resource utilization during the referral process between primary care providers to hospital-based specialists.9

Each stage of the referral process was mapped and then a time-motion approach was implemented to estimate the costs of processing a referral (eg, minutes to process a referral, number of times that the referral is returned with missing information, etc.). Staff costs were estimated for the eReferral system and the paper-based referral process by applying average wage rates to the time-motion data. Compared to the paper-based referral process, the eReferral system was estimated to save up to €1.6 in the average cost per minute spent during the referral process. The authors extrapolated these results to more than 700 thousand referrals in Denmark and found possible savings of €3,512,146 associated with the implementation of this eReferral system in Denmark.

Effects on efficiency of the referral process

Most of the studies analyzed changes in the referral process (7/9) (eg, workflow, wait times, and clinical care). Using an 18-item questionnaire, Kim et al (2009) conducted a study in the United States to assess the impact of eReferral on the workflow and clinical care on 298 primary care providers. The results indicated that 72% of the primary care providers felt that eReferral had improved the ability to track referrals, guide pre-visit work-up, and decrease wait times for new appointments. Overall 21% of the respondents reported no change in the workflow and clinical care while 7% were dissatisfied with the eReferral system.10

In another US study, Scheibe et al (2015) analyzed 2,105 referrals to rheumatologists from primary care providers from 2008 to 2012. They found that the main benefit of the eReferral system was the ability of specialists to participate in back and forth pre-consultation communication, including requesting more information. Their analyses suggested that pre-consultation exchange increased from 55% in 2008 to 74% in 2011. At the same time, only 63% of these pre-consultations were subsequently scheduled for a face-to-face appointment. The remaining 37% of the pre-consultations did not lead to a face to face appointment because the referrals were deemed inappropriate for a rheumatologist, the clinical reason leading to the referral was resolved by using the eReferral system or more information was requested before proceeding with an appointment.11

Scott (2009) conducted a study in Wales evaluating urgent cancer referrals from General Practitioners (GPs) to specialists through an eReferral system. In this study, 99 referrals from 5 GP practices were analyzed from January to July 2007 as part of a pilot program. The results indicated that the eReferral system improved the speed with which referrals were sent to the central office as well as the speed in which they were processed once received. Overall 81% of the 99 referrals were processed within 1 hr. In addition, GPs were highly satisfied with the eReferral system as measured by a 10-point likert scale (1= completely dissatisfied, 10= completely satisfied). The lowest satisfaction level recorded was an 8.12

Studies focus on the efficiency showed a reduction on wait times as well as inappropriate referrals. Additionally, they found a good perception from the care providers given a better track and communication with the patients.

Effects of electronic referral process on quality of referrals

Kim-Hwang et al (2010) conducted an analysis of the quality of referrals in the US associated with eReferrals compared to a paper-based system based on 505 questionnaires.13 Results indicated that the reason for referral was difficult to identify in 19.8% of the medical visits using a paper-based referral compared with only 11% of the eReferrals (p=0.03). Only 2.6% of the eReferrals were deemed not appropriate compared with 6.4% of the paper-based system, but this difference was not statistically significant (p=0.21). Regarding the need to follow-up, results varied depending on the type of provider. For example, medical specialty clinicians indicated that they requested follow-up in 84% of the paper-based referrals vs 90% of the eReferrals (p=0.06) while surgical specialists indicated that they requested follow-up in 76.2% of their paper-based referrals vs 58.1% of their eReferrals.

In a pre-post analysis performed in New Zealand, Corwin et al (2014) assessed two interventions to improve the quality of referrals and the percentage of adequate referrals to specialists.14 The first intervention evaluated whether feedback to GP (N=15) and nurses (N=2) from peers improved the quality of the referral letters to specialists. The second intervention evaluated the implementation of an eReferral system. Findings showed that the first intervention (ie, feedback to GP) improved the proportion of adequate referrals from 81.4% to 86.9% (p-value of 0.05). In the case of the eReferral system, the intervention did not improve the proportion of adequate referrals or quality of referrals. (p-value of 0.20).

Effects of electronic referrals process on quantity of referrals

The study performed by Chambers et al (2015) evaluated the changes in the number of diabetes referrals following the implementation of an eReferral system with and without training on the use of the system.15 The analyses were based on 75 referrals from 6 different health centers that occurred from April 2012 to November 2014. After the implementation of the eReferral system without training, the monthly number of referrals increased from 1 to 3. Following the implementation, training was offered and the average monthly number of referrals went from 3 to 6. Regression results confirmed that the trend in referrals before and after the implementation of the eReferral system was not significant while the trend in referrals increased significantly following the training of providers to use the electronic system.

A second study conducted by MacGregor et al (2009) evaluated the implementation of an eReferral system in a pediatric hospital in Canada. Before the eReferral implementation, 30% of the referrals were either incomplete or inappropriate.16 The implementation of the eReferral system resulted in a decrease of 2.7% of the incomplete or inappropriate referrals, as well as the increase of 6% in the total number of referrals processed.

Another Canadian study conducted by Doumouras et al (2017) examined the change in the number of referrals following the implementation of an eReferral system to replace the fax-based referral process for bariatric surgery in Ontario.17 Using regression analyses, the authors reported that the average number of referrals per year went from 244 in 2011 to 671 in 2015 due to the implementation of the eReferral system. Regression analyses adjusting for health regions, rural or urban, distance from the bariatric center, type and gender of physician and years of experience showed that overall 68% of the physicians increased their number of referrals after system implementation. The authors also examined factors associated with increase in eReferral. At the individual level, being a female practitioner, a primary care specialist and with 0–5 years of experience were associated with an increase of referrals after the eReferral system implementation. At the region level, living closer to Bariatric centers (<50 km) was also a statistically significant factor explaining an increase in electronic referrals.

The three studies found an increase in referrals through the implementation of eReferrals.

Discussion

Our review indicates that there is a great deal of variation in study design, setting, and measurement which limits the generalizability of the existing findings. Most of the studies conducted evaluations through the use of surveys or questionnaires. Only one study was a full economic evaluation (cost-benefit) making difficult to generalize or evaluate if e-referral process is cost-effective across different settings. Only two studies (2/9) performed an econometric analysis and the other seven used descriptive statistic analysis.

The lack of a standardized design of studies (for example, use of non-standardized surveys) may suggest the need for methodological standards and frameworks to evaluate eReferral systems across settings.

Nonetheless, the evidence presented in these studies collectively indicates that eReferral systems have the potential to improve the quantity and quality of referrals and are associated with a high degree of satisfaction by physicians in many settings. Studies showed the since the implementation of eReferrals, the number of referrals increased, but there is also a possibility of cost savings for the system, a reduction of unnecessary referral or follow-up, given a better idea of consultation as well as a good perception from PCP in quality and access. Findings also showed that e-referrals can reduce missing or incomplete information in the referral process avoiding back and forth between PCP leading to an improvement in wait times in consultation follow-up and less time to process referrals from providers.13

The main strength of our study was the systematic capture and review of nine studies evaluating eReferral systems in a variety of settings. We also defined the studies across logical categories that identify the main purpose and limitations to their evaluation. These categories allow decision-makers and researchers to distinguish the evidence-base and develop more complete study designs. However, further analysis should focus on the creation of feasible methodological standards for the design of economic evaluations in this area and measurement of a complete set of outcomes.

There are some limitations associated with our review. First, due to the small quantity of evaluations of eReferral systems, we were only able to describe the results and key findings of each study rather than conduct a more conclusive meta-analysis, regression analysis or related quantitative method to evaluate the overall effect of eReferral systems in quality, quantity, efficiency, and cost-effectiveness. Although we took a very systematic approach in searching the published literature, we did not keep track of the number of websites and documents that we reviewed for inclusion/exclusion. Similarity to the published literature, while we identified many potential documents, those were not related to e-referrals. Nonetheless like any literature search, it is possible that some studies are not captured in this review. Another limitation was that e-consultation and eReferral systems are described using similar terms, which made it more difficult to identify and delineate the evidence for eReferral systems alone.

Despite these limitations, the results of this scoping review indicate that further research is warranted to understand the effectiveness of eReferral systems within health care. Further economic analyses are needed to determine if initial capital and training costs are justified by efficiencies and health system benefits after implementation, and how referral volumes influence cost recovery. In addition, cost-analyses of e-referrals systems can also inform future cost-effectiveness or cost-benefit analyses. Nonetheless to better inform decision-making, comparative analyses based on strong study designs (eg, randomized controlled trial) are also needed to determine the effectiveness or cost-effectiveness of eReferral systems compared to self-referral systems and complete e-consultation systems in settings where more than one solution is viable.

We found that the evidence base for electronic referral systems is positive but limited. The evidence base is too limited to determine if eReferral systems are cost-effective interventions as most of the studies have evaluated eReferral systems on the basis of referral quantity and quality without considering costs. This scoping review is a first step to understanding the evidence-base for eReferral system implementation and methodological frameworks for their evaluation.