Telephone triage in general practices: A written case scenario study in the Netherlands.

OBJECTIVE: General practices increasingly use telephone triage to manage patient flows. During triage, the urgency of the call and required type of care are determined. This study examined the organization and adequacy of telephone triage in general practices in the Netherlands.
DESIGN: Cross-sectional observational study using a web-based survey among practice assistants including questions on background characteristics and triage organization. Furthermore, practice assistants were asked to assess the required type of care of written case scenarios with varying health problems and levels of urgency. To determine the adequacy of the assessments, a comparison with a reference standard was made. In addition, the association between background characteristics and triage organization and the adequacy of triage was examined.
SETTING: Daytime general practices.
SUBJECTS: Practice assistants. MAIN OUTCOME MEASURES: Over- and under-estimation, sensitivity, specificity.
RESULTS: The response rate was 41.1% (n = 973). The required care was assessed adequately in 63.6% of cases, was over-estimated in 19.3%, and under-estimated in 17.1%. The sensitivity of identifying patients with a highly urgent problem was 76.7% and the specificity was 94.0%. The adequacy of the assessments of the required care was higher for more experienced assistants and assistants with fixed daily work meetings with the GP. Triage training, use of a triage tool, and authorization of advice were not associated with adequacy of triage.
CONCLUSION: Triage by practice assistants in general practices is efficient (high specificity), but potentially unsafe in highly urgent cases (suboptimal sensitivity). It is important to train practice assistants in identifying highly urgent cases. KEY POINTS: General practices increasingly use telephone triage to manage patient flows, but little is known about the organization and adequacy of triage in daytime practices. Telephone triage by general practice assistants is efficient, but potentially unsafe in highly urgent cases. The adequacy of triage is higher for more experienced assistants and assistants with fixed daily work meetings with the general practitioner.

Introduction

Triage is the process of determining the level of urgency and type of healthcare required in requests for help: telephone advice, consultation or home visit with a general practitioner (GP), or referral to the emergency department or ambulance care. Telephone triage is a vulnerable part of the care process: the assessment is made without visual input and a balance has to be found between efficiency (giving patients the lowest effective level of care) and safety (identifying patients in need of immediate care).[1,2] Telephone triage is increasingly used to manage workload in primary care.[3]

In the Netherlands, the quality of telephone triage in out-of-hours primary care services, GP cooperatives, has received relatively much attention.[4-8] In these large-scale organizations, triagists are qualified after professional training and use decision-support systems.[2] Previous research on the adequacy of urgency assessments by triagists at GP cooperatives and emergency departments showed that the urgency was over-estimated in 1–19% of telephone contacts (inefficient triage) and under-estimated in 7–41% (potentially unsafe triage).[2,5-10] Relevant characteristics of the Dutch healthcare system are listed in Figure 1.

Figure 1.

Features of the healthcare system of the Netherlands.[4] 1National Institute for Public Health and the Environment: Nationaal Kompas Volksgezondheid: http://www.nationaalkompas.nl. 2InEen: http://ineen.nl/wp-content/uploads/2015/11/Benchmarkbulletin_HAP_2014_def.pdf

In daytime general practices, telephone triage is performed by practice assistants, who generally have followed an intermediate vocational medical education of three years. A minority of the assistants are educated as a nurse. In contrast to out-of-hours GP cooperatives, most practice assistants in daytime practices have had no additional training as a triagist. To our knowledge little is known about the adequacy of triage assessments in daytime general practices.

To guide policy decisions aiming to optimize patient safety, it is relevant to know what background and organizational factors are associated with the adequacy of triage. Long working experience and extensive triage training of practice assistants are likely to be associated with more adequate triage, as experience and education are known to influence performance in general. The use of a triage decision-support system is also expected to have a positive effect [11]. The NHG Triage Index is a Dutch triage decision support system that is generally used in GP cooperatives and increasingly in general practices [12,13]. Moreover, getting feedback on performance might also be associated with the adequacy of triage. Feedback on performance can be obtained during daily work meetings with the GP and after authorization of contacts in which an assistant gave patients self-care advice.

Our study aim was to investigate the organization of telephone triage in general practices, the adequacy of the assessments of the required type of care (i.e. over- and under-estimation, sensitivity, specificity, and predictive values), and factors (i.e. characteristics of practice assistants and triage organization) associated with adequate assessments of the required type of care.

Material and methods

Design

We performed a cross-sectional observational study using a web-based survey among practice assistants workings in daytime primary care.

Population and setting

The survey was conducted among 2369 practice assistants working in general practices in the Netherlands. Half of the assistants (n = 1184) were asked to assess the required type of care in case scenarios. The contact details of the assistants were obtained from the Dutch Association of Medical Assistants (NVDA). About 30% of all practice assistants in the Netherlands are members of the NVDA. The members are a mix of new and inexperienced assistants and experienced assistants. We excluded members known not to be working (any longer) as practice assistants in primary care.

Questionnaire and case scenarios

The web-based survey contained questions concerning background characteristics of the practice assistants, the organization of triage in the general practice in which they were employed, and case scenarios. The background questions were based on an existing questionnaire for general practices [14] and were checked for completeness and relevance by the management of the NVDA and two practice assistants. Based on an earlier study in emergency departments,[13] on the medical experience of the researchers (including one GP and two general physicians), and on descriptions of health problems in triage systems, 36 case scenarios were written. The case scenarios varied in the degree of urgency and required type of care. The required care categories were based on the NHG Triage Index:[12,13] (i) immediate warning of GP and dispatch of ambulance if necessary, (ii) appointment for urgent consultation with GP within one hour, (iii) appointment for consultation with GP within three hours the same day, (iv) appointment for consultation with GP without time pressure, and (v) telephone advice by assistant. To be able to evaluate the safety of triage (i.e. potential unsafe triage decisions), we deliberately included more highly urgent case scenarios than actually occur.

All case scenarios were presented to an expert panel and field panel to determine the “reference standard” regarding the required type of care. The expert panel consisted of three triagists from GP cooperatives and three GPs, who were asked by e-mail to give for each case scenario their assessment of the appropriate type of care to be chosen by a practice assistant. We used triagists from GP cooperatives instead of practice assistants to determine the “correct answer” because they are formally trained and registered as telephone triagists and can be seen as experts in telephone triage. The field panel consisted of nine professionals (GPs and triage assistants) from one GP cooperative. The members of the field panel were asked to assess the case scenarios during a workshop and the most common score per case scenario counted as one “vote” in determining the reference standard. The six experts on the expert panel all had an individual vote, resulting in a total of seven votes per case scenario. The case scenarios were usable if there was a consensus of over 70%: at least five out of seven votes were for the same type of required care. Eventually, 19 of the 36 case scenarios were included in the study, of which six (32%) were highly urgent (category 1 or 2) and 13 (68%) low urgent (category 3, 4 or 5) (see Appendix 1 for the 19 case scenarios used).

Procedure

The practice assistants received an e-mail with a personal link to a secure website to complete the questionnaire. All assistants received questions regarding their background and the organization of telephone triage in the general practice. Half of the assistants were asked to assess the required type of care of a random set of four or five out of the 19 eligible case scenarios, as if they were telephone calls from patients in their practice. The number of cases per assistant varied, in order to present each case scenario equally often. The other half of the assistants received other questions that are beyond the scope of this article. The data collection took place in April–May 2013 during a period of 21 days, with a reminder on the tenth day.

Data analysis

Descriptive statistics were used to examine the background characteristics of practice assistants and organizational characteristics of the general practices. To calculate the percentages of correct estimation, and under- and over-estimation of the required care for the case scenarios, the assessments of the assistants were compared with the reference standard, including a calculation of the number of categories by which the assistants varied from the reference. In addition, sensitivity, specificity, and predictive values were calculated. Sensitivity was used as a measure for potential unsafe assessments, whereas specificity was used as a measure for inefficient assessments. For this purpose, the required care categories were dichotomized into high urgent care (category 1: GP consultation within one hour and category 2: direct help) and low urgent care (category 3: GP consultation within three hours; category 4: GP consultation without time pressure and category 5: telephone advice by assistant). Healthcare problems that can wait more than one hour (category 3, 4, or 5) were not considered highly urgent, because there is no chance that the patient’s condition will soon deteriorate or that delaying treatment will cause serious and irreparable damage.[5,15]

To examine which factors related to the adequacy of triage, we calculated the percentage of errors per practice assistant (error rate) in cases where three or more case scenarios had been assessed. Each deviation from the reference standard, no matter how large, counted as an error. The association between characteristics of the practice assistant and triage organization and the error rate was examined using a multiple regression analysis with forced entry in two blocks. In step 1, working experience and triage training were entered in the model. In step 2, use of triage tool, authorization of telephone advice, and work meetings were added. The error rate was the outcome. The data were checked for influential cases, linearity, multicollinearity, homoscedasticity, and normality of errors. Two cases were outliers and had deviating test scores. These influential cases were deleted from the regression analysis.

For each of the predictors we present the unstandardized regression coefficient (B) with 95% confidence interval (95% CI) and standard error (SE) and the standardized coefficient (beta). The analyses were performed using the statistical software package IBM SPSS 20™ (IBM Corp, Armonk, NY, USA). Results were considered significant at p < 0.05.

Results

Background characteristics practice assistants and general practices

The response rate was 41.1% (n = 973). All responding practice assistants were female and their mean age was 42.4 years. The respondents worked an average of 25.1 hours per week in a general practice and had an average of 13.3 years of experience as a practice assistant. About a third of the respondents (32.5%) had not followed any triage training. The NHG Triage Index was used in most contacts by 4.1% of the respondents. The majority of respondents (80.7%) used it in less than half of the telephone contacts.

In the majority of the general practices where the respondents worked, the GP and the assistants (92.4%) had daily work meetings, either during the (coffee) break (39.7%), at fixed times (37.7%), and/or in between seeing patients (33.3%). Almost all respondents gave telephone advice (99.5%) and 37.1% indicated that the advice is mostly authorized by the GP (Table 1).

Table 1.

Characteristics of practice assistants and general practices (n = 973).

Background characteristics	%
Age in years, mean (range)	42.4 (20–64)
Sex
Female	100
Working hours per week, mean (range)	25.1 (4–42)
Working experience in years, mean (range)	13.3 (0–47)
Highest completed medical education
Practice assistant	88.3
Nurse	5.3
Other	6.4
Triage training
Qualified triagist	11.6
Other (e.g. internal course)	55.9
No	32.5
Is/was triagist at GP cooperative
Yes	5.4
In the past	7.6
No	87.0
Frequency of use triage tool
Mostly (> 75%)	4.1
Often (50–75%)	15.2
Sometimes (25–50%)	41.2
Seldomly (< 25%)	35.4
Never	4.1
Moment of use of triage tool
Usually	28.1
Only in doubt	62.3
In retrospect (to check)	14.0
As a reference work during training or study	13.8
Daily work meeting with GP
During (coffee) break	39.7
At a fixed time	37.7
In between seeing patients	33.2
Other	3.6
No	7.6
Assistant gives telephone advice
Yes	99.5
Authorization of advice
Mostly (> 75%)	37.1
Often (50–75%)	8.4
Sometimes (25–50%)	21.8
Seldom (< 25%)	18.1
Never	14.6
Practice size
Solo/duo practice (vs. more than two GPs)	46.9
Practice location
Urban area (vs. rural)	45.0

Adequacy of triage

The response rate was 40.0% (n = 474). In 63.6% (1424/2240) of cases the assessment of the required care was the same as the reference standard. In 30.6% of cases (685/2240) the difference from the reference standard was one step (e.g. GP no time pressure versus GP < 3 hours) and in 5.8% (131/2240) two or more steps (e.g. GP < 3 hours versus direct help). The required care was over-estimated in 19.3% (433/2240) of cases and under-estimated in 17.1% (383/2240) (Table 2).

Table 2.

Assessment of required care: Practice assistants versus reference standard (n = 2240 cases assessed by 474 practice assistants).

	Reference standard
Practice assistant	Direct help	GP <1 hour	GP <3 hours	GP No time pressure	Telephone advice	Total
Direct help	179	88	16	0	6	289
GP < 1 hour	55	228	61	3	6	353
GP < 3 hours	12	110	221	84	34	461
GP no time pressure	0	34	67	281	135	517
Telephone advice	1	10	9	85	515	620
Total	247	470	374	453	696	2240

Direct help: direct action and immediate warning of GP and send in ambulance if necessary; GP <1 hour: appointment for urgent consultation with GP within one hour; GP < 3 hours: appointment for consultation with GP within three hours the same day, GP no time pressure: appointment for consultation with GP without time pressure; telephone advice: telephone advice by assistant. Items in bold: agreement between practice assistant and reference standard; dark grey cells: over-estimation of required care by practice assistant; light grey cells: under-estimation of required care by practice assistant.

The sensitivity of the assessments of the required care was 76.7% (550/717) and the specificity was 94.0% (1431/1523). The positive predictive value was 85.7% (550/642); this is higher than the a priori probability of a case requiring highly urgent care in this study (32.0%; 717/2240). The negative predictive value was 89.5%, while the a priori probability of a case requiring low urgent care was 68.0% (1523/2240).

Association with characteristics of practice assistant and triage organization

Table 3 gives the results of the multiple regression analysis. The error rate in the assessment of required type of care was significantly lower for more experienced assistants (B = –0.003): for each year’s increase in experience, the error rate reduced by 0.3%. Visual inspection of the scatterplot indicated that the error rate decreased a little faster in the first years of experience than in later years. However, adding a quadratic term to the model showed this term was not significant.

Table 3.

Multiple regression analysis: Predictors of error rate in triage assessments (n = 418).

	Error rate triage assessments
	B (95% CI)	SE B	β
Step 1
Constant	0.412 (0.362–0.461)	0.025
Working experience (years)*	–0.003 (–0.006–0.000)	0.001	–0.101
Triage training:
No training (Ref)
Qualified triagist	0.047 (–0.031–0.125)	0.040	0.064
Other (e.g. internal course)	–0.022 (–0.074–0.031)	0.027	–0.045
Step 2
Constant	0.497 (0.374–0.621)	0.063
Working experience (years)*	–0.003 (–0.006–0.000)	0.001	–0.104
Triage training:
No training (Ref)
Qualified triagist	0.045 (–0.035–0.124)	0.041	0.061
Other (e.g. internal course)	–0.019 (–0.072–0.035)	0.027	–0.038
Frequency of use of triage tool	–0.012 (–0.041–0.018)	0.015	–0.039
Frequency of authorization of advice	0.000 (–0.013–0.013)	0.007	0.001
Daily work meeting assistant and GP:
No work meeting (Ref)
At a fixed time*	–0.059 (–0.118–0.000)	0.030	–0.119
In between patients	–0.040 (–0.096–0.015)	0.028	–0.076
During (coffee) break	–0.041 (–0.096–0.013)	0.028	–0.085

*p < 0.05. Influential cases (n = 2), respondents who assessed less than three case scenarios (n = 5), and respondents with missing values (n = 49) on the included variables were excluded from the analysis. R2 = 0.019 for step 1, ΔR2 = 0.013 for step 2.

The error rate was significantly lower for assistants with fixed daily work meetings with the GP (B = –0.059). Assistants with fixed daily work meetings had an average error rate of 34% (n = 174) compared with 38% for assistants without a fixed work meeting (n = 295) (not in table).

Triage training, use of a triage tool, and authorization of advice did not predict the error rate. The amount of variance explained by the models was low, respectively 1.9% and 3.2%.

Discussion

Principal findings and interpretation

Practice assistants made an adequate assessment of the required care in 64% of cases, while 19% were over-estimated and 17% under-estimated. The sensitivity was not optimal (77%), which means that the practice assistants missed a significant number of the highly urgent help requests. However, since we over-represented the number of highly urgent case scenarios in the sample, the potential risk for patient safety in triage of real contacts in general practices will be lower. The specificity was high (94%), so practice assistants worked efficiently, rarely over-estimating low urgent help requests. In other words, they do not often give a “false alarm”.

The adequacy of the assessments of the required type of care was significantly higher for more experienced assistants and assistants with fixed daily work meetings with the GP. However, the clinical relevance of these findings is low: the absolute decrease in the error rate per year of experience was not substantial and also the difference in error rates between the groups with and without fixed work meetings was small. Triage training, use of a triage tool, and authorization of advice were not associated with the adequacy of triage.

The organization of telephone triage and training of practice assistants in Dutch general practices is not uniform. Only a small proportion of the practice assistants had been trained as qualified triagists and most assistants do not regularly use a triage tool. Almost all practices have daily work meetings between GPs and assistants, but usually not at a fixed time. Nearly all assistants give patients telephone advice by themselves, but this is not regularly authorized by the GP.

Strengths and limitations

This study is one of the first in its field and was conducted among a large group of practice assistants across the Netherlands.

The response rate was mediocre. However, the actual response rate is probably higher, because we do not know if the e-mail addresses of the non-respondents were in use; a number of the invitations to the survey probably did not reach the intended receivers. We included incomplete questionnaires in the analyses to avoid selection bias, as assistants who did not fully complete the questionnaire might have found the cases more difficult to assess than assistants who assessed all cases. However, we set a minimum number of three completed cases to maintain enough (variation in) cases for the calculation of the respondent’s error rate.

The error rate was the percentage of errors per practice assistant. It depends on the healthcare problem and time limits for possible treatments as to whether a one-step error in the upper care categories is more important than a one-step error in the lower care categories. However, we decided not to give a different weight to each possible error of each case scenario. We believe that an unweighted error rate over all cases gives the most objective and interpretable results.

Performing a survey with case scenarios instead of using simulated calls enabled us to reach a large group of assistants. Moreover, by using case scenarios instead of studying real patient contacts we could over-represent cases requiring highly urgent care and thus study the safety of triage. Written case scenarios have previously been used in research into triage and have proved to be a useful method.[9,16] A limitation of written case scenarios is that there is no possibility for the practice assistant to ask the patient additional questions. Furthermore, all information is presented at once, assuming that the practice assistant would collect this information. There was also no time limit to assess the cases.

Finally, the 19 included case scenarios had a panel consensus of 70% or more concerning the type of care required. Applying a stricter criterion would have decreased the number of eligible case scenarios. A post hoc analysis for the set of case scenarios with (almost) perfect agreement between the panel members showed similar results, but had less statistical power and less variation in case scenarios.

Comparison with previous studies

The percentage of agreement with the reference standard (64%) in our study was within the range (49–78%) of other studies in out-of-hours primary care that used simulated patients or case scenarios.[5,6,9,17] The percentages of under- and over-estimation were similar to those found by Giesen et al.,[5] but different from the results of Derkx et al.,[6] who found under-estimation to be 41% (our study 17%) and over-estimation was 1% (our study 19%). This latter finding might be explained by differences in methodology and case description.

Giesen et al. reported the same sensitivity (76%) and specificity (95%) as in our study, when comparing highly urgent and low urgent cases;[5] the other studies did not report these outcomes.

We did not find clinically relevant associations between the background of the practice assistants or triage organization and the adequacy of triage. This is in line with other studies regarding factors affecting triage decisions: neither the clinical background of the triagist [5,9,17] nor the length of experience [9,18] affected triage decisions. However, contrary to our findings, triagists trained in the use of a triage tool were found to have a lower rate of under-estimation of the urgency.[5]

Recommendations for practice and research

Because of the potential lack of safety of triage in highly urgent cases, practice assistants should be trained in recognizing alarm symptoms in the health problems patients present.

Moreover, in the context of patient safety, standards for authorizing telephone advice by practice assistants in daytime general practice are recommended. Furthermore, the NHG Triage Index is only being used moderately in general practices. Possibly, the assistants are not familiar with it or the triage tool is less appropriate in general practice. Examination of the suitability of the triage decision-support tool in general practice is required.

Finally, we could only explain a very small part of the variation in the error rate between practice assistants. Further studies in this area are recommended, for instance into psychological features and personality characteristics of practice assistants. Assistants might have individual approaches to risk that influence their triage decisions.[18] For example, their ability to cope with stressful situations might influence triage decisions.[19]