Quality indicators development and prioritisation for emergency medical call centres: a stakeholder consensus.

INTRODUCTION: Emergency medical regulation is a risky activity. In France, emergency medical societies have proposed activity and performance indicators, but their lists are non-exhaustive, unstructured and used heterogeneously among emergency medical call centres (Centres de Réception et de Régulation des Appels, CRRA). Our objective was to build by means of regional stakeholder consensus an operational quality dashboard for CRRAs.
METHODS: We conducted an observational step in a French CRRA from June to September 2018 and at the same time listed existing activity and quality indicators through a rapid international literature review. We adapted and classified all indicators identified in a structured table. We prioritised them from April to September 2019 by seeking consensus with one regulator physician and one medical regulation assistant from the 13 CRRAs of the largest French region. We used an adapted Delphi method with a prioritisation scale from 1 to 9.
RESULTS: The rapid review of literature included 33 studies among the 414 identified and, with the first observational step, resulted in a list of 360 quality indicators covering the following areas: material resources, human resources, quality approach, call handling and postcall support. 15 of the 26 members participated in the entire process. Seventy indicators were considered as priorities with strong agreement among participants. We built an operational dashboard of quality indicators deemed high priority and provided 70 descriptive indicator sheets.
CONCLUSION: Our study allowed to build an operational quality dashboard for CRRAs as a ready-to-use support for an internal audit, for prioritisation of quality approach actions and for national and international benchmarking. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Introduction

Emergency medicine is, by its nature and uncertain environment, a risky activity.1 2 More specifically, medical regulation has the daily challenge of responding appropriately and effectively to emergency medical calls; when necessary, it must send the appropriate resource as fast as possible. This particular exercise is recognised as telemedicine and is based on validated procedures and recommendations.1 3–5

Urgent call management and emergency medical aid organisation are specific to each country. Some have adopted a single emergency number, such as the UK with 999,6 the USA with 9117 and Europe with 112.8 Depending on the type of emergency, an appropriate unit is sent and transmissions are made with the nearest emergency service when necessary. However, many countries still have three separate numbers for emergency, depending on whether the emergency comes under urgent medical aid, police or fire department; each of them has its own call centre and mobile units.9

This is the case in France, where emergency medical calls are centralised into emergency medical call centres, known as CRRAs (Centres de Réception et de Régulation des Appels).10 Operational 24/7 via the call number ‘15’, a medical regulation assistant (ARM) receives calls before the expertise of a regulator doctor (MR), who can be either an emergency physician or a general practitioner, depending on the type of emergency to manage.3 The CRRA is an integral part of the emergency medical assistance service called SAMU (Service d’Aide Médicale Urgente) in France, whose mission is to make decisions on how to best handle various medical situations. These can range from giving a simple telephone advice to sending mobile emergency and resuscitation unit, to quickly bringing sick persons or victims to appropriate hospitals while providing first aid.10

Medical regulation practices are associated with serious adverse events. These incidents are specific and particularly complex due to the telephone treatment of medical calls and the intervention of non-medical services (firefighters, medical transport, police).1

The recent French context, marked by the ‘tragedy of Strasbourg’ in December 2017,11 as well as several complaints for non-assistance to people in danger relayed in the French press, has highlighted this complexity as well as the lack of security in the work of agents handling calls. In addition, terrorist attacks underline the role of medical regulation in health response coordination.

The number of calls to SAMU keeps increasing (about 31 million calls for 67 million inhabitants in France in 2017; +20% between 2013 and 201712), which shows high expectations from the public1 13 and increases the pressure felt by the agents to offer effective and quality service and ensure the ‘right care’.13

Nevertheless, in France, no formalised and harmonised quality approach exists in CRRAs. Medical regulation does not appear in certification processes.1 Emergency medical societies have proposed activity and performance indicators, but the resulting lists are non-exhaustive, unstructured3–5 and their use heterogeneous between CRRAs.1

International literature on the effectiveness and precision of the responses provided is quite rich, especially concerning cardiorespiratory arrests,14–17 but very poor in terms of quality approach and evaluation. Studies exist on quality and indicators identification concerning prehospital urgent care18 19 or emergency services,20 but not concerning the previous step, which is the reception and regulation of urgent calls. Performance seems to take precedence over global quality.

As a result, it seems urgent to propose a structured list of quality indicators for medical regulation, concerning human and material resources, activity, performance and quality of calls, in the form of a dashboard. This tool is designed to follow the progress of the quality approach, assess it, identify areas for improvement and pilot actions.21 With this in mind, considering stakeholder opinion and experience in seeking consensus seems necessary in developing and prioritising a set of quality indicators. The objective of this work is to build an operational quality dashboard for CRRAs by searching for consensus on a regional scale.

Materials and methods

Indicator development phase

An observational step was conducted from June 2018 to September 2018 in a French emergency department at the Poitiers University Hospital by the public health team. Two public health residents (LA, JG) came to the CRRA as observers for 2 days to better understand the functions of the SAMU and to identify the risks related to the practice of regulation. Observations consisted of a tour of the premises, presentation of the operation of the SAMU and of the CRRA team, and monitoring of regulation activity through the practice of double listening calls with ARMs and MRs.

In parallel, a rapid international review of literature22 was carried out to identify all existing quality and activity indicators. A rapid review is a form of knowledge synthesis in which components of the systematic review process known as ‘evidence summaries’ are streamlined to produce information in a short period of time. Although numerous rapid reviews have been produced, their varied methodologies can diversely impact research results.23 A rapid review protocol was compiled based on the ‘framework of rapid review methods’.24

The databases queried were PubMed, ScienceDirect, Cochrane Library and Cairn. A first exploratory phase of these databases helped refine the keywords. The search equations are presented in table 1.

Table 1

Databases and search equations for the rapid review of literature

Database	Search equation
PubMed	(call centers [MeSH Terms]) AND (quality OR indicators OR quality assurance, health care [MeSH Terms] OR quality improvement [MeSH Terms] OR quality of health care [MeSH Terms])
	(emergency medical dispatch [MeSH Terms]) AND (quality OR indicators OR quality assurance, health care [MeSH Terms] OR quality improvement [MeSH Terms] OR quality of health care [MeSH Terms] OR quality indicators, health care [MeSH Terms])
	(medical regulation) AND (risk OR quality OR indicators OR quality assurance, health care OR quality improvement OR quality of health care)
	(emergency medical call OR Emergency Medical Communication Centre) AND indicators
ScienceDirect	(emergency call centers) AND (quality OR indicators OR quality assurance, health care OR quality improvement OR quality of health care)
	(emergency medical dispatch) AND (quality OR indicators OR quality assurance, health care OR quality improvement OR quality of health care OR quality indicators, health care)
Cochrane Library	(call centers) AND (quality OR indicators OR quality assurance, health care OR quality improvement OR quality of health care)
	(emergency medical dispatch) AND (quality OR indicators OR quality assurance, health care OR quality improvement OR quality of health care OR quality indicators, health care)
Cairn	régulation médicale OU régulation SAMU

The following were the inclusion criteria:

Studies published in English or French.

Studies published since 1 January 1999 (in France, the development of quality procedures follows the ordinance of 24 April 1996 establishing an accreditation procedure implemented since 199925).

All types of studies.

The following were the exclusion criteria:

Title and abstract published in language other than English and French.

Title and abstract off subject.

Irrelevant population or irrelevant outcomes in full text, that is, not meeting the objective (identify quality indicators).

Articles were selected first after reading the title and the abstract by two reviewers (LA, JG), then the full text by one reviewer (LA). In case of disagreement on a selection, an agreement was achieved after discussion between the two reviewers.

Other sources consisted of recommendations and repositories published by various French emergency medicine societies and associations, as well as by the French National Authority for Health (Haute Autorité de Santé, HAS), and the General Inspection of the Administration and General Inspection of Social Affairs report.26

At the end of this preliminary step, all the recommendations and indicators concerning the SAMU regulation were listed in a table using the framework with structure, process and outcome indicators.27 Some indicators were developed by the public health team following the observational phase and after discussion with CRRA stakeholders and the French Society of Emergency Medicine (Société Française de Médecine d’Urgence, SFMU). Some indicators from the recommendations were adapted (reformulated or clarified) for better understanding and greater consistency.

Indicator prioritisation phase

An interventional step was conducted from April 2019 to September 2019 in the largest French region (the Nouvelle-Aquitaine region with 6 million inhabitants, or 10% of the French population) by the public health team of Poitiers University Hospital.

The listed quality indicators were prioritised using the HAS validated consensus method ‘Recommendations by formalized consensus’ (December 2010, updated March 2015).28 This approach is an adapted method from the Delphi one.29 Briefly, the HAS consensus method consists of an individual rating in two rounds, separated by an intermediate meeting. The method involves three groups: one steering group, one rating group and one reading group. A questionnaire is distributed with a discrete numerical scale graduated from 1 to 9 next to each item: a value of 1 means that the respondent considers the proposal totally inappropriate (or not indicated or not acceptable); a value of 9 means that the respondent considers the proposal totally appropriate (or indicated or acceptable); values of 2–8 reflect possible intermediate situations; and a value of 5 corresponds to respondent indecision.

Patient and public involvement

In this study, the steering group was made up of the medical manager of SAMU 86 (HD-F), a doctor from SAMU 86 involved in the quality process (JB), the head of the public health department of the University Hospital of Poitiers and coordinator of care risk management (VM), and a public health resident from Poitiers University Hospital (LA). The rating group consisted of volunteer ARMs and MRs from the 13 SAMUs of the Nouvelle-Aquitaine region. The sample consisted of one MR and one referral ARM, with at least 15 years of experience, for each CRRA. The reading group was extended to all CRRA agents in the region.

First, the approach was presented to the 13 SAMUs of the region in order to obtain their consent and their commitment to participate in the entire prioritisation process.

The questionnaire was drawn up from the exhaustive table of indicators, which was built from the development phase (observational step and rapid review of literature). The design was neat in order to facilitate participant acceptance and increase response rate. To preserve anonymity, the questionnaire was sent to the medical manager of each SAMU, who then forwarded it to the rating MRs and ARMs.

The rating was done individually via an online survey site, SurveyMonkey, and participants were given 2 weeks to answer the survey. To be sent, the questionnaire had to be completed in full, but the filling could be done in several times. Responses were anonymous.

The rating criterion used was the priority given to the indicator. Each item had to be rated on a scale of 1 to 9: 1 if the item was considered non-priority and 9 if it was considered absolute priority. A final item allowed participants to comment or suggest a new indicator. The results obtained were analysed according to the HAS method. The proposal was deemed priority when the median value was ≥7 and all ratings ≥5; non-priority when the median value was ≤3.5 and all ratings ≤5; and uncertain when the median value was between 4 and 6.5 (indecision) or when there was no consensus between the rating group members (all other situations). Proposals with strong (ratings in the range 7–9) or relative (ratings in the range 5–9) agreement for prioritisation were accepted as such, without discussion in the meeting or submission to the second round of rating.

At the end of the first round, a results restitution was planned, as well as a discussion with the rating group about items that did not reach consensus. These were submitted for a second round of rating, but only to members who had attended the discussion, according to the HAS validated method.28 The response and analysis methods remained unchanged.

At the end, all results as well as a process summary were returned to all SAMUs in the region via a newsletter sent by email. A structured descriptive sheet30 was drawn up for each prioritised indicator, which contains the title, objective, complete definition of its numerator and denominator if applicable, data collection modalities, details on the interpretation of the results and characteristics. Finally an operational dashboard was established based on the indicators deemed to be priorities to be able to give visibility of quality within the CRRAs through the results of each indicator: "yes", "no", "partially". This dashboard is in the form of an Excel document containing one tab per theme to be covered, then one line per indicator and its result opposite, with the possibility to directly access the descriptive sheet for each item. Every year it will be completed to follow the evolution of the quality approach within the CRRAs. This entire step was carried out by the steering group.

Results

The rapid review of literature included 33 studies in our work, from the initial 414 that resulted from various searches (figure 1).

Figure 1

Flow chart of the rapid review of literature for the development and prioritisation of quality indicators in emergency medical call centres.

The preliminary phase of observation, discussion with stakeholders and literature review identified 360 quality indicators concerning SAMU regulation.

All were classified in a table structured into five themes according to the field concerned: material resources, human resources, quality approach, call handling and postcall support (table 2). These five subjects were chosen according to the themes found in all the indicators listed, as well as by relying on the repositories of the association ‘SAMU-emergencies of France’3 5; we can thus find what allows a CRRA to function (human and material resources), the indicators which optimise this operation and the safety of the CRRA (quality approach), as well as the indicators directly linked to the activity of the CRRA (performance about calls and support for victims).

Table 2

Distribution of quality indicators for emergency medical call centres, listed in the initial table built from observation and rapid review of literature

Theme		Indicators (n)
1	Material resources	158
	Documents and procedures	51
	Exceptional health situation	22
	Amenities	85
2	Human resources	60
	Management	6
	ARM	11
	MR	10
	Other professionals	16
	Work time	10
	Age structure	1
	Teaching and coaching	6
3	Quality approach	105
	Generalities	6
	Training	31
	Assessment of professional practices	41
	Management of adverse events	12
	Research	15
4	Call handling	32
	Quantitative indicators	7
	Good practices	25
5	Postcall support	5
	Total	360
	From the observation	158
	From the rapid review	202

ARM, medical regulation assistant; MR, regulator doctor.

After presenting the approach and the prioritisation process by sending an electronic document to the Nouvelle-Aquitaine SAMUs, 13 MRs and 13 ARMs agreed to participate (table 3).

Table 3

Characteristics of respondents for prioritisation process in the 13 CRRAs of the Nouvelle-Aquitaine region

	Targeted participants	Round 1		Round 2
	n	n	%	n	%
ARM	13	8	62	6	75
MR	13	12	92	9	75
Total	26	20	77	15	75
SAMU represented	13	12	92	9	75

ARM, medical regulation assistant; CRRA, Centres de Réception et de Régulation des Appels; MR, regulator doctor; SAMU, Service d’Aide Médicale Urgente.

The first round of rating took place between 30 April and 23 May 2019, lasting approximately 3 weeks. The response rate was 77%; only one SAMU was not represented among the respondents (table 3).

At the end of the first round, 18.9% of the items were deemed priority with strong agreement, 33.6% priority with relative agreement, 15.6% uncertain due to indecision and 31.9% uncertain due to lack of consensus (table 4; detailed data not shown).

Table 4

Results of analysis according to the HAS method used for prioritisation of quality indicators in emergency medical call centres

Item judged	Round 1		Round 2		Total
	n	%	n	%	n	%
Priority with strong agreement	68	18.9	2	1.1	70	19.3
Priority with relative agreement	121	33.6	35	20.1	156	42.9
Uncertain due to indecision	56	15.6	55	31.6	55	15.2
Uncertain due to lack of consensus	115	31.9	82	47.1	82	22.6
Total	360		174*		363

*Round 2: the 174 items submitted for rating correspond to the 56+115 items judged uncertain in the first round plus 3 new items included.

HAS, Haute Autorité de Santé.

Before the second round, some items were reformulated by the steering group due to ambiguous or unclear formulation, according to the HAS method.28 Moreover three new items were integrated following the proposals of the participants.

The 171 items which did not obtain consensus in the first round as well as the 3 new items were submitted for a new rating to the 20 respondents from the first round.

The second round of rating took place between 25 June and 23 September 2019, for around 13 weeks. The response rate was 75%; nine SAMUs were represented (table 3). At the end of the second round, 1.1% of the items were deemed priority with strong agreement, 20.1% priority with relative agreement, 31.6% uncertain due to indecision and 47.1% uncertain due to lack of consensus (table 4; detailed data not shown).

Finally, out of the 26 participants included in the consensus approach, 15 participated in the whole process, representing a response rate of 58%. All in all, 70 items (19.3%) were considered priority with strong agreement, 156 (42.9%) priority with relative agreement, 55 (15.2%) uncertain due to indecision and 82 (22.6%) uncertain due to absent consensus (table 4).

At the end of the prioritisation process, a newsletter was returned to the 13 SAMUs presenting the results and prospects. A dashboard of indicators deemed high priority has been established (online supplemental table 1). Seventy descriptive sheets were written. The reading group contributed by simplifying some of them (structural indicators), reformulating some objectives and redrawing some items to improve the feasibility of data collection.

Discussion

Main results

To our knowledge, this is the first study on the development and prioritisation of quality indicators for emergency medical call centres. There were 360 indicators initially listed, thereby constituting the first structured and exhaustive list in France for prehospital emergency regulation.

The large number of indicators shows the complexity of medical regulation activity, its many risks at all levels (material and human resources, call management and quality approach) and the need to prevent them in order to improve quality of care. Such a large quantity of indicators cannot be realistically accepted, implemented and evaluated, hence the need to prioritise them for progressive implementation in CRRAs. Finally, 70 indicators were deemed to be high priority with strong agreement. This significant reduction in the number of indicators allowed creation of an operational dashboard and the writing of an indicator descriptive sheet for each.

The 70 prioritised indicators concern conventions, procedures and protocols, methods for effector triggering, exceptional health situations, equipment, call recording, training, the roles and number of concerned professionals, securing their work, relations with firefighters, regulation file keeping and call transfers. These emerging themes are coherent with the current context and stakeholder concerns (ARMs and MRs) and illustrate the need to secure regulation activity. It is interesting to note that no quantified performance indicator was found among the 70 highest priority indicators.

Place in the literature

In contrast, many works have been done about performance, in particular with regard to cardiopulmonary arrests,14–17 with the aim of showing how to improve the efficiency and accuracy of the response provided by emergency call centres.31 Agent occupation rate seemed to be the most important factor contributing to service quality as well as all quantitative indicators.

However, quality does not stop with performance. On the contrary, quality and safety of the structure, resources and processes can improve performance, as explained by Giroud in his publication.32 However, few indicators exist in the literature to assess these dimensions. In our study, the diversity of listed indicators and the choice of prioritised indicators confirm this need to consider regulation activity in its entirety, as well as human factors, in view of achieving more secure activity. Moreover, the notion of human factors has appeared in recent literature: for example, Rawshani et al33 concluded that there was a higher risk of not prioritising life-threatening patients during lunchtime. Work by Hamelin and Arzalier34 goes in the same direction; analysis of complaint files showed that considering human factors and fatigue is essential and underlines the importance of reporting on internal dysfunctions. The impact of feedback on performance was explored by Clawson et al,35 which showed that regular and objective information to emergency medical dispatchers about their performance significantly improved their rigour. Professional practice evaluation, feedback and adverse event analysis must be encouraged in order to achieve a positive culture of error.36 37 The role of human factors was also studied in telephone exchanges by Riou et al,38 who analysed the words used to initiate cardiopulmonary resuscitation (CPR) assisted by the dispatcher and their association with caller agreement. The quality of telephone communication should be formalised and evaluated.39 40

Securing the work of agents seems to be a major concern at present, as confirmed in our study by the strong consensus obtained for this proposal. In this complete item, well-being at work is taken into consideration. Indeed, a link has been demonstrated between job satisfaction and performance41; here again, human factors are important.

Thus our study seems to fill a gap in the exhaustiveness of the risk factors associated with CRRA activity, taking into consideration the human factor.

Strengths and weaknesses

This work was carried out in connection with the French Society of Emergency Medicine and was based on a validated method considering peer opinion, thereby strengthening its validity. The search for consensus has confirmed the need to formalise a quality approach as well as the relevance of the indicators. We initially obtained strong adhesion following presentation of the approach, and second the results gave no non-priority indicators and an average of >5 out of 9 (priority) for 96.7% of them. All in all 62.3% of the indicators obtained a strong or relative prioritisation consensus.

The first round of rating required two general reminders by email and a 1-week extension to obtain the maximum number of responses. The participation rate was suitable (77%). For a better questionnaire acceptability and an optimal response rate, the choice was made to adapt the results analysis of the first round, by accepting as such the proposals that have obtained a strong and relative agreement. By resubmitting to the rating only uncertain proposals (for which there was indecision or absence of consensus), the second round would be less heavy.

Furthermore, the possibility for the participants to express themselves freely at the end of the questionnaire was important for proposal integration or modification before the second round.

The second phase took place during the summer period, explaining the two general reminders and the two individual reminders, as well as the time limit for closing the 13-week survey. It nevertheless obtained a participation rate of 75%. Out of the 174 items resubmitted for rating, 37 finally obtained agreement. We can think that when each participant knows the median obtained by the group for each item, it becomes possible for some participants to review their judgement and reposition themselves accordingly.

Finally, only 58% of the participants went through the prioritisation process. However, Nouvelle-Aquitaine is the largest region in France, with 12 departments and 13 SAMUs, which allowed for 15 representative peers to participate in the whole process: doctors and ARMs, who are the main participants in regulation. The length of questionnaire is probably largely responsible, but it was necessary to be as exhaustive as possible from the start in order to be relevant in reduction by prioritisation. Because of this length constraint, it was possible to respond to the questionnaire in several phases and an effort was made on its presentation. Otherwise, the duration of the process (more than 4 months in total) as well as the vacation period during which it took place certainly favoured the number of ‘lost to follow-up’.

It was not possible to hold a meeting with the rating group between the two rounds, which can constitute a potential bias. The aim of the meeting was to return the results of the first round and to discuss the proposals deemed uncertain. Given constraints in each participant’s schedule, it seemed preferable to return the results by email so that everyone could benefit from the same feedback. To do this, each participant of the first round received an overall summary of the results of the analysis as well as a file with their own answers. The median obtained by the group was shown below each item on the second questionnaire so that each participant could situate himself in relation to the rest of the group during the second round, according to the HAS method. In the absence of discussion, comments and proposals brought up during the first round were considered and partly integrated into the questionnaire, after consultation with the steering group.

Implications

This work will facilitate the gradual integration of a quality approach within CRRAs. From now on, each SAMU in the region will be able to assess the quality and safety of its regulation activity. Using the dashboard, or a computerised mapping tool, will enable organisation of a periodic internal audit. The detailed descriptive sheets constitute an operational tool to also benchmark with other CRRAs in order to help each other out and improve everyone’s practice. Ultimately, once the installation is successful, the indicators that were prioritised with relative agreement could gradually be integrated until exhaustive implementation of the 363 indicators is achieved in a few years.

Moreover, this dashboard could be integrated into the quality account of regional establishments as well as in certification procedures. These indicators could also be integrated into the indicators for improving quality and safety of care (IPAQSS).42

Finally, this work could facilitate CRRA inclusion in the IFAQ procedure (Financial Incentive to Improve Quality).43

Areas for improvement

The search for consensus was carried out on a regional scale, but this approach could be extrapolated to national and international scale to increase the power of the study, as well as to standardise practices in terms of quality and safety and to do benchmarking. Moreover, we could integrate other actors of medical emergency, such as paramedics, nurses and general practitioners.

Initiatives are already emerging to strengthen medical regulation quality within CRRAs. Training with diploma was set up in 2019 and is henceforth compulsory to exercise the profession of medical regulation assistant.44 Telemedicine is also growing, with CPR assisted by phone or smartphone, remote video consultations to assess severity of patient illness or remove doubts, etc; these are valuable aids for regulation.

Furthermore, the Information System-SAMU (SI-SAMU) programme is being deployed in French territories and aims to make the information system and telecommunications means used by emergency medical services more reliable and secure. This centralised and national portal is intended to be used by all SAMUs.45

Conclusion

In conclusion, our study allowed identification of all quality indicators for SAMU regulation and prioritising them by means of regional stakeholder consensus in order to propose an operational dashboard for progressive implementation in CRRAs. In the future, we could extend the prioritisation process on a larger scale and include other CRRA agents to strengthen the power of the study and standardise practices.