Measuring and benchmarking the quality of two different organizational ways in delivering infant vaccination.

The aim of this study was the quality of service evaluation of two different organizational ways in delivering infant vaccination according to a Regional Vaccination Plan. Eleven vaccination centres were selected in two Local Health Units (ASLs) belonging to the Regional Health Service of the Lazio Region, Italy. The services offering paediatric vaccinations for children under three years of age, delivered without an appointment (VACP) or with the need for an appointment (VACL), were investigated. The quality aspects under evaluation were communicational efficiency, organisational efficiency and comfort. Subjective data were collected from different stakeholders and involve the elicitation of best and worst feasible performance conditions for the ASLs when delivering VACP/VACL services. Objective data consists in the observation of current performances of the selected vaccination centres. Quality scorecards were obtained from the combination of all data. Benchmarking between VACP and VACL, i.e., two different organisational ways in delivering infant vaccination, can be performed as a result of the probabilistic meaning of the evaluated scores. An expert of vaccination services, i.e., a virtual combination of patients, doctors and nurses, claims the quality of service delivery of the ASLs under investigation with probability 78.03% and 69.67% for VACP and VACL, respectively. In other words, for short, the quality scores of the ASLs were 78.03% for VACP and 69.67% for VACL. Furthermore our results show how to practically improve the current service delivery. The QuaVaTAR approach can result in improvements of the quality of the ASLs for the two different ways of delivering paediatric vaccinations in a simple and intuitive way.

Introduction

The concept of quality, which was embraced during the second half of the last century, has spread exponentially in many areas. In the healthcare field, providers and policy makers have engaged in an extended search for useful approaches to measure and improve service performance in terms of quality, volume and user's perspective [1-5].

In Donabedian's classic framework, service quality is represented by considering structure, process and outcome using classes of key performance indicators (KPIs). The first class considers the organisational resources that ensure the functioning of health services and may be obtained from data generated "routinely" at healthcare facilities. The second class measures relief provided to the patient using data obtained from properly completed records. The third aims to assess the outcomes of healthcare interventions [6].

The Declaration of Alma-Ata in 1978 established the right and duty to participate individually and collectively in the planning and implementation of healthcare [7].

A main problem in measuring the quality of healthcare service could result from the fact that often, due to the lack of standards, different KPIs can be chosen to evaluate different providers; and this could lead to some benchmark issues. Furthermore, the evaluation process should include directly and actively both users and other stakeholders who participate in the organisation and delivery of services. To manage the KPIs heterogeneity and overcome the partiality of the decision maker's "absolutistic" point of view, an innovative and quantitative approach was defined and applied in different hospital settings [8, 9]. Later, the focus of the project has shifted from the hospital to local services, with the "Quality in Vaccination: Theory And Research" (Qua- VaTAR) project, to assess the quality of vaccination services [10, 11].

The aim of this work is to illustrate a new application of the QuaVaTAR approach for the quality evaluation of selected paediatric vaccination centres of the Lazio Region of Italy according to the Regional Vaccination Plan 2012-2014 [12].

Materials and methods

In April-July 2010, two of the twelve Local Health Units (ASLs) within the Regional Health Service of the Lazio Region of Italy were involved in this case study according to their willingness to participate: ASL RMH, located in the edge of Rome; and ASL RMF, located outside the city. Eleven vaccination centres were selected: two centres in one of the five districts of the ASL RMH and nine centres in the four districts of the ASL RMF. The two different ways of offering paediatric vaccinations to children under three years of age, with the need for an appointment (VACP) and without an appointment (VACL), were evaluated. The service quality aspects under evaluation were communicational efficiency, organisational efficiency and comfort. For each of these aspects, two quality KPIs were selected. Communicational efficiency was represented as the means and time of communicating information related to the vaccine with oral or written support before or during vaccination. Organisational efficiency was represented by the time a user spent in the waiting room and the time a user spent for vaccination. Comfort was represented by the opening times, and the presence of toys and/or a nursing room. KPIs and their possible values were the same for VACP and VACL, except for waiting time ranges.

SUBJECTIVE DATA

An opinion survey was conducted at the two ASLs to collect "subjective data" related to the selected quality aspects of VACP and VACL. A well-defined questionnaire was administered to different groups of stakeholders: parents and escorts of immunised children (P/E), medical doctors (D) and nurses (N) working in the ASL. There were three sections in the questionnaire: an anonymous demographic section with questions on age, education, marital status and job; an informative section on the functioning of a generic vaccination service as presented by institutional guidelines; and a judgment section to assess specific performance conditions of delivery (defined in terms of the selected KPIs) containing 8 questions on communicational efficiency, 16 questions on organisational efficiency and 8 questions on comfort. Judgments expressed by stakeholders were used to assess probabilities (i.e., values ranging from 0-100%) of specific events related to communicational efficiency, organisational efficiency and comfort of the vaccination delivery. Different weights were chosen to combine judgments of each stakeholder: 0.3, 0.4, and 0.3 for P/E, D and N, respectively. This allowed to obtain the Expert, i.e. a "super" virtual stakeholder, point of view. The maximum and minimum probabilities which were assessed with this opinion survey define the best and worst achievable service quality scores for the ASLs.

OBJECTIVE DATA

In the same period of the survey, April-July 2010, the selected KPIs were measured at the vaccination centres of the ASLs during the service delivery. These observations were recorded in an "objective data" set. They included waiting times (minutes), duration of vaccinations (minutes), indicators of means of communication and indicators of comfort features (true/false values).

QUALITY SCORECARD

Subjective and objective data were entered into a spread sheet that easily implements a quality scorecard according to a well-defined algorithm [9, 11]. Specific weights were chosen to combine the quality aspects under investigation and evaluate the overall quality score.

The overall quality for VACP/VACL is given by the weighted sum of the corresponding quality aspects (i.e., organisational efficiency, communicational efficiency and comfort).

The relevance of communicational efficiency, organisational efficiency and comfort was equal to 0.5, 0.3 and 0.2, respectively for both VACP and VACL.

Results

Questionnaires were administered to 416 stakeholders. The main socio-demographic characteristics of the interviewees are summarised in Table I. Statistical difference was present only for educational level as for VACP 88% of parents or escort had a high educational level in contrast with 77% in VACL (p < 0.003).

Tab. I.

Socio-demographic characteristics of the interviewed stakeholders.

Stakeholder demographic characteristic
	VACP					VACL
Stakeholder	n°	Median age (SD)	High School	Married	Job	n°	Median age (SD)	High School	Married	Job
P/E	226	34 (6.7)	200	213	173	162	34.9 (5.7)	125	147	98
N	8	39.2 (8.4)	8	5	8	8	39.7 (13.2)	8	5	8
D	9	46.1 (9.6)	9	7	9	3	39.0 (9.8)	3	2	3
Total	243	35.6 (7.2)	217	225	190	173	35.2 (6.2)	136	154	109

VACP: paediatric vaccinations to children under three years of age with the need for an appointment

VACL: paediatric vaccinations to children under three years of age without an appointment

P/E: Parents and escorts of immunised children

N: Nurses

D: Medical doctors

Table II shows the results of the survey related to the communicational efficiency.

Tab. II.

Communicational efficiency of the vaccination delivery assuming different performance conditions.

Subjective data related to the communicational efficiency
Performance	VACP				VACL
	P/E (%)	N (%)	D (%)	Expert (%)	P/E (%)	N (%)	D (%)	Expert (%)
Information is provided through a brochure with the aid of a person in the waiting room	86.65	60.00	63.33	69.33	84.73	96.00	90.00	90.22
Information is provided through a brochure with the aid of a person during the vaccination phase	79.83	90.00	100	90.95	75.93	93.00	87.78	85.79
Information is provided through a brochure without the aid of a person at the time of the vaccination service direct call	54.89	60.00	51.67	55.13	53.40	57.80	57.78	56.47
Information is provided through a brochure without the aid of a person in the waiting room	53.22	40.00	43.33	45.30	51.79	51.00	60.56	55.06
Information is provided through a brochure without the aid of a person during the vaccination phase	41.39.	25.00	26.67	30.58	37.74	39.50	48.89	42.73
Information is provided orally by a person in the waiting room	71.63	75.00	50.00	63.99	69.02	75.50	70.00	71.35
Information is provided orally by a person during the vaccination phase	72.38	95.00	76.67	80.88	65.20	77.50	62.22	67.70
information is not provided	6.14	0	0	1.84	10.01	11.00	0	6.30

VACP: paediatric vaccinations to children under three years of age with the need for an appointment

VACL: paediatric vaccinations to children under three years of age without an appointment

P/E: Parents and escorts of immunised children

N: Nurses

D: Medical doctors

From the Expert's point of view (i.e., the combination of all stakeholders' points of view), the minimum value of communicational efficiency (1.84% for VACP and 6.30% for VACL) is obtained if the information is not provided. In contrast, the maximum value for VACP (90.95%) and VACL (90.22%) were assessed if the information is provided through a brochure with the aid of a person. The difference was related to the moment considered more efficient for communication: during vaccination for VACP, in the waiting room for VACL.

The maximum values of organisational efficiency (94.91% for VACP and 81.22% for VACL) were assessed if the service is provided with waiting time less than 10-15 minutes (for VACP/VACL) and duration time less than 10 minutes. In both cases, however, a vaccination time of 10-20 minutes was considered efficient (84.21% for VACP and 74.21% for VACL).

The maximum values of comfort (95.61% for VACP and 90.17% for VACL) were assessed if the vaccination ambulatory is open alternatively in the morning and in the afternoon with the presence of children's toys and availability of a nursing room.

During the study 198 vaccination deliveries were observed.

Table IIIa shows different performance conditions related to the communicational efficiency which were observed at the ASLs during the vaccination delivery. The majority of performances were observed whit information provided orally by a person during the vaccination phase.

Tab. IIIa.

Performance conditions related to the communicational efficiency of the ASLs vaccination services.

Objective data related to the communicational efficiency
Performance	VACP	VACL
	Performance frequency (%)	Performance frequency (%)
Information is provided through a brochure with the aid of a person in the waiting room	17.10	0
Information is provided through a brochure with the aid of a person during the vaccination phase	17.10	32.40
Information is provided through a brochure without the aid of a person at the time of the vaccination service direct call	0	0.60
Information is provided through a brochure without the aid of a person in the waiting room	0	0.60
Information is provided through a brochure without the aid of a person during the vaccination phase	0	0.60
Information is provided orally by a person in the waiting room	0	9.00
Information is provided orally by a person during the vaccination phase	65.80	56.80
Information is not provided	0	0

VACP: paediatric vaccinations to children under three years of age with the need for an appointment

VACL: paediatric vaccinations to children under three years of age without an appointment

For organizational efficiency the majority of performances were observed with a waiting time less than 10- 15 minutes (for VACP/VACL) and a vaccination time less than 10 minutes (Tab. IIIb).

Tab. IIIb.

Performance conditions related to the organisational efficiency of the ASLs vaccination services.

Objective data related to the organisational efficiency
VACP			VACL
Performance		Performance frequency (%)	Performance		Performance frequency (%)
Wt (minutes)	Dt (minutes)		Wt (minutes)	Dt (minutes)
< 10	< 10	41.00	< 15	< 10	51.70
10 e 20	< 10	22.60	[15-30]	< 10	23.60
20 e 30	< 10	12.10	[30-45]	< 10	7.90
> 30	< 10	13.60	> 45	< 10	1.10
< 10	[10-20]	4.50	< 15	[10-20]	7.90
10 e 20	[10-20]	3.10	[15-30]	[10-20]	5.60
20 e 30	[10-20]	3.10	[30-45]	[10-20]	1.10
> 30	[10-20]	0	> 45	[10-20]	0
< 10	[20-30]	0	< 15	[20-30]	0
10 e 20	[20-30]	0	[15-30]	[20-30]	0
20 e 30	[20-30]	0	[30-45]	[20-30]	0
> 30	[20-30]	0	> 45	[20-30]	1.10

VACP: paediatric vaccinations to children under three years of age with the need for an appointment

VACL: paediatric vaccinations to children under three years of age without an appointment

Wt = Waiting time (minutes)

Dt = Vaccination Time (minutes)

Vaccination time greater than 30 minutes were never observed.

Considering comfort, for VACP, 58.00% of the services offered vaccination in the morning and in the afternoon; 48.00% have a nursing room and 75.00% toys (48.00% have both); while for VACL vaccination are administered only in the morning; in most cases the service provided toys (84.00%), and in 27.50% a nursing room (18.80% both).

Table IV shows the quality scorecard of the ASLs under evaluation for the two ways of offering vaccination and for different stakeholders. From the cumulative result the probability that an expert claims the quality of VACP was 78.03%, while the probability that an expert claims the quality of VACL was 69.67%.

Tab. IV.

Quality scorecard of the ASLs vaccination services.

Quality scorecards
	VACP Overall quality = 78.03%			VACL Overall quality = 69.67%
Stakeholder	Organizational efficiency (weight 0.3)	Communicational efficiency (weight 0.5)	Comfort (weight 0.2)	Organizational efficiency (weight 0.3)	Communicational efficiency (weight 0.5)	Comfort (weight 0.2)
P/E	71.07	76.11	65.95	74.09	68.66	62.12
N	81.82	88.14	69.23	76.63	81.78	38.59
D	83.48	78.38	72.91	68.38	71.04	63.57
Expert = 0.3P/E + 0.3N + 0.4D	79.26	80.63	69.72	72.57	73.55	55.64

VACP: paediatric vaccinations to children under three years of age with the need for an appointment

VACL: paediatric vaccinations to children under three years of age without an appointment

P/E: Parents and escorts of immunised children

N: Nurses

D: Medical doctors

Discussion

By adopting the QuaVaTAR approach it is possible to evaluate quantitatively and qualitatively the organizational characteristics of the vaccination services. In a first study it was applied to evaluate the different quality in the provision of HPV vaccination in three ASLs of the Lazio Region of Italy [11]. In the present work the same method was applied to evaluate different ways of delivering the same children vaccination, i.e. VACP and VACL. Identical KPIs chosen for the first study were used here. This is not a limitation. In fact, due to the flexibility of the method, other KPIs could be chosen, depending on users' interest.

In this study VACP obtained a better result than VACL, and the only significant difference was seen in the instruction level of parents/escorts, where those with an higher instruction level preferred VACP, showing a greater interest in that type of service organization that allowed a better management of time.

Subjective data measure how stakeholders perceived the VACP and VACL service quality of providers. Focusing on communicational efficiency, the results suggest that the best way to perform communication is to provide information through a brochure with the aid of a person during the vaccination phase for VACP and in the waiting room for VACL. For organisational efficiency, as expected, the maximum value for VACP and VACL was for a waiting time of 10- 15 minutes, respectively, and a vaccination time of less than 10 minutes. A short waiting time with a vaccination time of 10-20 minutes was however considered efficient for both VACP and VACL. For comfort, as expected, the maximum value corresponded to the opening time both in the morning and in the afternoon with the presence of toys and nursing room. These were the so called targets for VACP and VACL. These values do not correspond to the theoretical maximum score values (i.e., 100%). This is not a "limitation" of the model; better performance measured by different KPIs could exist. In this study, however, only performances that could be effectively implemented by the ASLs were modelled.

It is worth noting that different stakeholders may have a different perception of the service quality also related to the modality of providing vaccination. For example, for communicational efficiency, while in VACL all category considered better to give information through a brochure in the waiting room, and this was the best achievable target, in VACP parents considered better this way while doctors and nurses considered more efficient to give information during the vaccination phase, and this was the best achievable value considered by the Expert. This seems logical; in fact when access is planned the vaccination phase could be the best moment for communication, as a long waiting time is not foreseen. When access is free, a longer waiting time can be expected by the P/E and this time could be used for communication.

Despite the best efficient way of communication considered also the use of a brochure, this was used during vaccination or in the waiting room only in 34% and 32% of VACP and VACL providing. The use of a brochure could therefore improve the quality of both VACP and VACL. For organisational performance does not seem that there is a difference between VACP and VACL, although the considered waiting times were different. A difference was seen for comfort related to the opening times of the service. In VACP in the majority of cases it is possible to have an appointment both in the morning and in the afternoon, while VACL is offered only in the morning. Although the double possibility proved to be the most efficient, the values reached 95.61% for VACP and 90.17% for VACL.

Conclusions

The top management of the ASLs needs to take in serious consideration that, in terms of business risks, 21.97% and 31.33% are the probabilities that an expert "do not" claim the quality of the ASLs for the two ways of delivering the pediatric vaccination, VACP and VACL, respectively. The good news is that there exist margins of improvements. And the general criterion to obtain this is simple and intuitive by using the QuaVaTAR approach. It is necessary to transform the performance conditions currently observed during the vaccination delivery in those which are better for the stakeholders, as suggested by the opinion survey which involved parents/escorts, nurses and doctors.