Quality measurement and benchmarking of HPV vaccination services: a new approach.

BACKGROUND: A new measurement process based upon a well-defined mathematical model was applied to evaluate the quality of human papillomavirus (HPV) vaccination centers in 3 of 12 Local Health Units (ASLs) within the Lazio Region of Italy. The quality aspects considered for evaluation were communicational efficiency, organizational efficiency and comfort.
RESULTS: The overall maximum achievable value was 86.10%, while the HPV vaccination quality scores for ASL1, ASL2 and ASL3 were 73.07%, 71.08%, and 67.21%, respectively.
CONCLUSIONS: With this new approach it is possible to represent the probabilistic reasoning of a stakeholder who evaluates the quality of a healthcare provider. All ASLs had margins for improvements and optimal quality results can be assessed in terms of better performance conditions, confirming the relationship between the resulting quality scores and HPV vaccination coverage.
METHODS: The measurement process was structured into three steps and involved four stakeholder categories: doctors, nurses, parents and vaccinated women. In Step 1, questionnaires were administered to collect different stakeholders' points of view (i.e., subjective data) that were elaborated to obtain the best and worst performance conditions when delivering a healthcare service. Step 2 of the process involved the gathering of performance data during the service delivery (i.e., objective data collection). Step 3 of the process involved the elaboration of all data: subjective data from step 1 are used to define a "standard" to test objective data from step 2. This entire process led to the creation of a set of scorecards. Benchmarking is presented as a result of the probabilistic meaning of the evaluated scores.

Introduction

Cervical cancer is the third most common cancer in women worldwide and the seventh overall, with an estimated 500 000 new cases and 275 000 deaths in 2008. In Italy, approximately 2800 new cases are diagnosed, and over 1000 women are estimated to die every year.,

The World Health Organization recognizes cervical cancer as the first type of carcinoma completely due to an infection: virtually all cervical cancer cases (99%) are linked to a genital infection with Human Papillomavirus (HPV). The persistent infection with a carcinogenic type of HPV represents the necessary condition for the evolution to cervical cancer.

In Italy, beginning in March 2008, a campaign of active and free offer of both bivalent and quadrivalent vaccines against carcinogenic HPV types was launched. The campaign was aimed toward girls 11 to 12 y of age, starting with those born in 1997., The goal of the HPV immunization program was to achieve coverage greater than 95% with three doses of the vaccine within the year 2011. Despite numerous campaigns and recommendations to the population by institutional agencies, as of December 31, 2011, the HPV vaccination coverage with three doses of the vaccine in the cohort of girls born in 1997 was only 65%.

To achieve the health goals for cervical cancer reduction, the quality of services requires improvement. It is necessary to reinforce all actions known to improve vaccination coverage, such as better communication to the target population and efficient offer of the vaccine, as required by the Italian National Vaccine Prevention Plan 2012–2014.

Quality of vaccination services is part of the broader context of the quality of health services and is currently considered an essential feature by patients-users, staff working in this area and institutions.

Many methods for analyzing and improving quality have been used in the health care arena, including models that utilize users perspectives, such as the “SERVQUAL” method and “Discrete Choice Modelling.”, However, the assessment of the quality of care should not exclude the direct and active involvement of both those who participate in the delivery of health services and those who directly use these services.

Within a health care organization, the desire to overcome the partiality of the doctor’s “absolutistic” point of view or the patient’s “individualistic” one stimulated the search for a new methodology to assess service quality and to deliver consistent results. This is useful for the comparison of different providers.

Since 2005, starting with the quality measurement needs of a hospital, the “Measures of Quality in Health Care Services” research project (i.e., a multidisciplinary research team composed of medical doctors, engineers, and mathematicians) created a model of quality events related to service delivery that is based on the theory of probability according to Bruno de Finetti. From this theory, a consistent and rigorous quality measurement process was further developed and applied to different settings.,- Beginning in 2010, the focus of the research shifted from the hospital to local health services, particularly vaccination centers. Generally, when evaluating vaccination activities a key performance indicator (KPI) approach is used, which is based on the assessment of facilities, personnel and all available resources. The administration of vaccines, the patient invitation and registration, the achievement of vaccination coverage within the established time limits, the notification for vaccine-related diseases and the surveillance of adverse reactions have been used as main quality indicators.

The “Quality in Vaccination: Theory and Research” (QuaVaTAR) group, composed of medical doctors, engineers, epidemiologists and mathematicians, was designed to implement a new approach to the quality measurement and benchmarking of HPV vaccination services. This approach is based on a well-defined mathematical model and was tested for its applicability for the evaluation of the vaccination services in the Lazio Region of Italy.

In this pilot study, three of 12 Local Health Units (ASLs) within the Lazio Region of Italy were involved based on their willingness to participate: ASL1, located in the center of Rome; ASL2, located on the edge of the city; ASL3, located outside of the city. HPV vaccination services were evaluated on the basis of their communicational efficiency (CE), organizational efficiency (OE) and comfort (Co).

Results

Step 1 of the measurement process (subjective measurement)

During the first phase of the study, i.e., step 1 of the measurement process, from April to July 2010, 264 questionnaires investigating the probability (i.e., opinion) that a stakeholder (a vaccinated woman, a parent, a doctor or a nurse) assert communicatively efficient (event E1), organizationally efficient (event E2) and comfortable (event E3) the HPV vaccination services, assuming different performance conditions (in the following indicated by H), were obtained. The socio-demographic characteristics of the different stakeholder groups showed that 120 of 170 parents had a high school or university education level, and the mean age was 42.7 y old (±5). Among vaccinated women, 23 out of 63 had a high education level, and the mean age was 16.5 y old (+/− 4.2). The weighted sum of the opinions of all the stakeholder categories allows the obtainment of the “virtual” Expert point of view.

Tables 1A, B, and C show the output of step 1 of the measurement process. For example, for communicational efficiency (CE), which is represented as the means and time of communicating information related to vaccination, the maximum probability of event E1, from an expert’s point of view, was 87.05%, assuming true the occurrence of performance condition H1 i.e., information is provided through a brochure with the assistance of a person in the waiting room. Additionally, the minimum probability of event E1 was 7.16%, assuming true the occurrence of performance condition H8 i.e., information is not provided (Table 1A). For organizational efficiency (OE), which is represented by the time spent by a user in a waiting room and the time spent by a user for vaccination, from an expert’s point of view the maximum value was 82.91% and the minimum value was 22.15% (Table 1B). For Comfort (Co), represented by the opening times and the availability of magazines in the waiting room, we obtained 88.52% and 52.99% (Table 1C).

.

Stakeholders opinions related to the communicational efficiency of the HPV vaccination service delivery

HPV vaccination (HPVV) communicative efficiency (subjective data)
HPVV performance condition (in terms of events)		HPVV survey questions (in terms of conditional events)	Survey result (in terms of conditional probabilities)
			Parent	Nurse	Doctor	Woman	Expert P(E1|Hx)
H1	Information is provided through a brochure with the aid of a person in the waiting room during the HPVV providing	You* assert communicatively efficient the HPVV (i.e., E1), assuming H1 true	88.43%	88.18%	91.19%	77.70%	87.05%>
H2	Information is provided through a brochure with the aid of a person during the vaccination phase of the HPVV providing	You* assert communicatively efficient the HPVV, assuming H2 true	80.30%	78.18%	84.38%	74.97%	79.93%
H3	Information is provided through a brochure without the aid of a person at the time of the HPVV service direct	You* assert communicatively efficient the HPVV, assuming H3 true	59.79%	66.36%	53.75%	57.46%	59.16%
H4	Information is provided through a brochure without the aid of a person in the waiting room during the HPVV providing	You* assert communicatively efficient the HPVV, assuming H4 true	54.13%	65.45%	49.38%	50.16%	54.74%
H5	Information is provided through a brochure without the aid of a person during the vaccination phase of the HPVV providing	You* assert communicatively efficient the HPVV, assuming H5 true	46.13%	48.18%	34.06%	43.13%	42.42%
H6	Information is provided orally by a person in the waiting room during the HPVV providing	You* assert communicatively efficient the HPVV, assuming H6 true	75.80%	76.36%	77.5%	68.85%	75.06%
H7	Information is provided orally by a person during the vaccination phase of the HPVV providing	You* assert communicatively efficient the HPVV, assuming H7 true	71.49%	70.00%	76.25%	66.64%	71.57%
H8	Information is not provided	You* assert communicatively efficient the HPVV, assuming H8 true	7.44%	5.45%	3.75%	14.05%	7.16%

(*) “You” means a parent (P), a nurse (N), a doctor (D), a vaccinated woman (V), or an expert (i.e., a virtual stakeholder whose opinion is obtained as a weighted sum of those of P, N, D, and V, exactly as 0.25*P+0.25*N+0.30*D+0.20*V).

Table 1B. Stakeholders opinions related to the organizational efficiency of the HPV vaccination service delivery

HPV vaccination (HPVV) organizational efficiency (subjective data)
HPVV performance condition (in terms of events)		HPVV survey questions (in terms of conditional events)	Survey result (in terms of conditional probabilities)
			Parent	Nurse	Doctor	Woman	Expert P(E2|Hx)
H1	The waiting time is less than 10 min and vaccination time less than 10 min	You* assert organizationally efficient the HPVV (i.e., E2), assuming H1 true	86.49%	76.36%	85.33%	83.00%	82.91%
H2	The waiting time is between 10 and 20 min and the vaccination time is less than 10 min	You* assert organizationally efficient the HPVV, assuming H2 true	74.15%	72.27%	76.67%	68.92%	73.39%
H3	The waiting time is between 20 and 30 min and the vaccination time is less than 10 min	You* assert organizationally efficient the HPVV, assuming H3 true	57.80%	55.45%	61.67%	52.75%	57.36%
H4	The waiting time is greater than 30 min and the vaccination time is less than 10 min	You* assert organizationally efficient the HPVV, assuming H4 true	42.70%	38.18%	37.67%	40.51%	39.62%
H5	The waiting time is less than 10 min and the vaccination time is between 10 and 20 min	You* assert organizationally efficient the HPVV, assuming H5 true	69.33%	87.27%	79.00%	67.23%	76.30%
H6	The waiting time is between 10 and 20 min and the vaccination time is between 10 and 20 min	You* assert organizationally efficient the HPVV, assuming H6 true	61.46%	81.36%	72.67%	62.69%	70.04%
H7	The waiting time is between 20 and 30 min and the vaccination time is between 10 and 20 min	You* assert organizationally efficient the HPVV, assuming H7 true	48.82%	57.73%	51.87%	48.77%	51.95%
H8	The waiting time is greater than 30 min and the vaccination time is between 10 and 20 min	You* assert organizationally efficient the HPVV, assuming H8 true	37.25%	42.27%	34.33%	37.75%	37.73%
H9	The waiting time is less than 10 min and the vaccination time is between 20 and 30 min	You* assert organizationally efficient the HPVV, assuming H9 true	64.91%	63.82%	53.44%	55.94%	59.40%
H10	The waiting time is between 10 and 20 min and the vaccination time is between 20 and 30 min	You* assert organizationally efficient the HPVV, assuming H10 true	55.52%	48.64%	41.88%	49.20%	48.44%
H11	The waiting time is between 20 and 30 min and the vaccination time is between 20 and 30 min	You* assert organizationally efficient the HPVV, assuming H11 true	48.16%	41.18%	31.56%	41.25%	40.05%
H12	The waiting time is greater than 30 min and the vaccination time is between 20 and 30 min	You* assert organizationally efficient the HPVV, assuming H12 true	37.08%	33.00%	22.00%	36.33%	31.38%
H13	The waiting time is less than 10 min and the vaccination time is greater than 30 min	You* assert organizationally efficient the HPVV, assuming H13 true	53.98%	48.33%	39.67%	48.75%	47.23%
H14	The waiting time is between 10 and 20 min and the vaccination time is greater than 30 min	You* assert organizationally efficient the HPVV, assuming H14 true	47.64%	41.11%	32.67%	43.59%	40.71%
H15	The waiting time is between 20 and 30 min and the vaccination time is greater than 30 min	You* assert organizationally efficient the HPVV, assuming H15 true	40.02%	20.89%	24.67%	36.42%	29.91%
H16	The waiting time is greater than 30 min and the vaccination time is greater than 30 min	You* assert organizationally efficient the HPVV, assuming H16 true	32.53%	14.44%	14.67%	30.04%	22.15%

(*) “You” means a parent (P), a nurse (N), a doctor (D), a vaccinated woman (V), or an expert (i.e., a virtual stakeholder whose opinion is obtained as a weighted sum of those of P, N, D, and V, exactly as 0.25*P+0.25*N+0.30*D+0.20*V).

Table 1C. Stakeholders opinions related to the comfort of the HPV vaccination service delivery

HPV vaccination (HPVV) comfort (subjective data)
HPVV performance condition (in terms of events)		HPVV survey questions (in terms of conditional events)	Survey result (in terms of conditional probabilities)
			Parent	Nurse	Doctor	Woman	Expert P(E3|Hx)
H1	The service is open only in the morning and magazines are available	You* assert comfortable efficient the HPVV (i.e., E3), assuming H1 true	47.99%	55.00%	56.50%	51.48%	52.99%
H2	The service is open only in the morning and no magazine is available	You* assert comfortable efficient the HPVV, assuming H2 true	54.09%	61.82%	60.81%	54.25%	58.07%
H3	The service is open both morning and afternoon and no magazine is available	You* assert comfortable efficient the HPVV, assuming H3 true	78.31%	78.18%	88.38%	70.23%	79.68%
H4	The service is open both morning and afternoon and magazines are available	You* assert comfortable efficient the HPVV, assuming H4 true	88.32%	91.82%	91.56%	80.08%	88.52%

(*) “You” means a parent (P), a nurse (N), a doctor (D), a vaccinated woman (V), or an expert (i.e., a virtual stakeholder whose opinion is obtained as a weighted sum of those of P, N, D, and V, exactly as 0.25*P+0.25*N+0.30*D+0.20*V).

Step 2 of the measurement process (objective measurement)

In step 2 of the measurement process, 115 observations related to HPV vaccine delivery in ASL1, ASL2 and ASL3 were collected. CE, OE and Co performance conditions considered in step 1 were investigated. Using the CE aspect as an example, Table 2A shows that the communicative performances (from H1 to H8) differed among the ASLs. ASL1 and ASL2 always (100%) provided information orally by a person during the vaccination phase (H7), while ASL3 provided the information in various ways: 38.50% as defined in event proposition H3 (information is provided through a brochure), 23.00% as defined in H6 (information is provided orally by a person in the waiting room) and 38.50% as defined in H7. Tables 2B and 2C show the results for OE and Co in the same way as CE in Table 2A.

Table 2A. Performance related to the communicative efficiency of the HPV vaccination service delivery

HPV vaccination (HPVV) communicative efficiency (objective data)
HPVV performance conditions (in terms of events)	Results (in terms of probabilities)
	ASL 1	ASL 2	ASL 3
H1 of Table 1A	0%	0%	0%
H2 of Table 1A	0%	0%	0%
H3 of Table 1A	0%	0%	38.50%
H4 of Table 1A	0%	0%	0%
H5 of Table 1A	0%	0%	0%
H6 of Table 1A	0%	0%	23.00%
H7 of Table 1A	100%	100%	38.50%
H8 of Table 1A	0%	0%	0%

Table 2B. Performance related to the organizational efficiency of the HPV vaccination service delivery

HPV vaccination (HPVV) organizational efficiency (objective data)
HPVV performance conditions (in terms of events)	Results (in terms of probabilities)
	ASL 1	ASL 2	ASL 3
H1 of Table 1B	54.10%	29.20%	25.00%
H2 of Table 1B	8.30%	37.50%	33.00%
H3 of Table 1B	0%	20.80%	17.00%
H4 of Table 1B	16.70%	8.30%	25.00%
H5 of Table 1B	8.30%	0%	0%
H6 of Table 1B	0%	0%	0%
H7 of Table 1B	0%	4.20%	0%
H8 of Table 1B	4.20%	0%	0%
H9 of Table 1B	0%	0%	0%
H10 of Table 1B	0%	0%	0%
H11 of Table 1B	0%	0%	0%
H12 of Table 1B	4.20%	0%	0%
H13 of Table 1B	0%	0%	0%
H14 of Table 1B	0%	0%	0%
H15 of Table 1B	0%	0%	0%
H16 of Table 1B	4.20%	0%	0%

Table 2C. Performance related to the comfort of the HPV vaccine service delivery

HPV vaccination (HPVV) comfort (objective data)
HPVV performance conditions (in terms of events)	Results (in terms of probabilities)
	ASL 1	ASL 2	ASL 3
H1 of Table 1C	0%	17.00%	33.00%
H2 of Table 1C	0%	0%	0%
H3 of Table 1C	0%	83.00%	42.00%
H4 of Table 1C	100%	0%	25.00%

Step 3 of the measurement process (final results)

In step 3 of the measurement process, the probabilities of the main quality events under evaluation were obtained using the algorithm described in the mathematical appendix in Supplemental Material. Table 3 shows the probabilities of E1, E2, and E3, that is, the quality scores regarding the HPV vaccination services, in comparison with the resulting minimum and maximum values, for each aspect, i.e. CE, OE, and Co, in each ASL under evaluation. The overall HPV vaccination quality scores were 73.07%, 71.08%, and 67.21% for ASL1, ASL2, and ASL3, respectively, and the overall minimum and maximum reference values were 20.82% and 86.10%.

Table 3. Main events under evaluation and their probabilities of being true from an expert’s point of view

HPV vaccination (HPVV) Quality Scorecard
Quality facts (in terms of events)		Scores (in terms of probabilities)
		Results			Reference values
		ASL1	ASL2	ASL3	Min	Max
E1	An expert asserts communicatively efficient the HPVV service delivery	71.57%	71.57%	67.60%	7.16%	87.05%
E2	An expert asserts organizationally efficient the HPVV service delivery	67.79%	69.12%	64.66%	22.15%	82.91%
E3	An expert asserts comfortable the HPVV service delivery	88.52%	75.23%	73.00%	52.99%	88.52%
Overall quality scores		73.07%	71.08%	67.21%	20.82%	86.10%

The probabilities in this table were obtained combining probabilities shown in the previous Tables by means of Equation 2 in the mathematical appendix in Supplemental Material. The last row is obtained as the sum of the expected value of the overall quality of services scores by means of Equation 3.

Discussion

This pilot study was performed to test the applicability of the measurement process of quality for HPV vaccination centers by a multidisciplinary team: the QuaVaTAR group.

Step 1 of the process was performed to collect “subjective data,” i.e., opinions, by means of a survey (questionnaire). The results of the survey were used to measure how stakeholders perceived quality of the service of a “standard provider” (i.e., a provider compliant with the best practice/recommendations or guidelines). The adjective “subjective” should not be interpreted as meaning “without objectivity.” Rather, this term refers to the fact that a subject (i.e., a stakeholder) can express its opinion (i.e., a probability) about a fact (i.e., an event) using all available data and experience (i.e., using maximum objectiveness). Different performance conditions defined by relevant KPIs were investigated to elicit the best and worst service delivery conditions for each stakeholder’s point of view. To obtain an expert point of view, the opinions expressed by each stakeholder were combined by means of numerical weights that represent the relevance recognized in a function of the stakeholder’s role in the quality evaluation of the service delivery. For example focusing on HPV vaccination CE, from the expert point of view the best way to perform communication is to provide information through a brochure with the aid of a person in the waiting room (Table 1A). This was the so-called “HPV vaccination CE target” for all the ASLs. In contrast, and as expected, the worst way to perform communication is obtained when information is not provided at all.

In this pilot study, a remarkable consensus was obtained among the respondents for the KPIs identified for CE: in fact, the difference between the achievable maximum and minimum values ranged from 87.50% to 7.16%. The choice of indicators for OE were also appropriate. Indicators for Co, instead, most likely did not represent elements of the service delivery considered particularly important for respondents, and the difference between the achievable maximum and minimum values was small (88.52% and 52.99%).

The fact that the best and worst service delivery conditions do not correspond to the theoretical maximum and minimum probability values (100% and 0%, respectively) is not a limitation of the model. Other performance conditions may be better (e.g., a higher value for HPV vaccination CE could be obtained if information is implemented by means of emails in response to customers’ questions) or worse (e.g., the value 0% could be related to the communication of incorrect information) than those investigated. However, not all performance conditions that could be effectively observed by the working group or implemented by the ASLs (assuming budgeting or organizational limitations) were investigated.

Differences in the perception of the stakeholders of the service quality are accepted. For example, analyzing our results in detail if information was not provided, doctors perceived this CE condition as worse in respect to nurses, parents, and vaccinated women (Table 1A).

It is important to note that there are no prescriptions for the minimum number of questionnaires. In fact, according to the subjective probability theory, any available information can be used to assess (prior) probabilities of events of interest. Thus, even if only a small number of questionnaires are available, they are useful in providing an initial evaluation of the probabilities of events of interest. In the event that further information becomes available, prior evaluations can be updated to obtain new (posterior) probabilities.

Step 2 of the process was performed to collect “objective data,” i.e., observations on those KPIs used in step 1 that describe the performances of the ASLs under evaluation.

Some KPIs can be “constant,” such as those related to the structure of the organization, and only one observation is needed to assess their values until some organizational change occurs. Other KPIs can be “variable” and more observations per service delivery are needed, even if there are no prescriptions for the minimum number as described for questionnaires. For example, regarding HPV vaccination CE (Table 2A), all information is provided orally by a person during the vaccination phase (100% of the HPV vaccination delivery) in ASL1 and ASL2, as determined by the local organization. In ASL3, however, three conditions with differing occurrence were observed; this variability was due to the absence of a specific commitment in the means of communication.

Step 3 of the process was performed to compute both the HPV vaccination service quality scores (i.e., the probabilities of E1, E2, and E3) and the “overall” quality scores of the ASLs (Table 3). At this point, it was necessary for the working group to quantify the relevance of the quality aspects under investigation (i.e., CE, OE, and Co) by means of some numerical weights. In this study, the working group decided to assign a higher relevance to CE, taking into account that the HPV vaccination targets a young-adult population, for which it is established that accurate information is one of the most effective ways to increase vaccination coverage.,

Table 3 clearly shows that all ASLs perform differently: for example, ASL1 and ASL2 have the same HPV vaccination CE score (71.57%), while the HPV vaccination OE score is higher for ASL2 (69.12%) than for ASL1 (67.79%) and the HPV vaccination Co score is higher for ASL1 (88.52%, equal to the maximum value) than it is for ASL2 (75.23%). In relation to the maximum overall quality score (86.10%), all ASLs have margins for improvements. In regard to CE (Table 2A), ASL1 and ASL2 both performed 100% of the HPV vaccination delivery by providing oral information from a person during the vaccination phase. In the opinion of the Expert, the value obtained (71.57%) for this condition (Table 1A) is not the maximum achievable. Therefore, ASL1 and ASL2 could improve the HPV vaccination quality by providing information through a brochure with the aid of a person in the waiting room. More improvements can be achieved by taking into account similar considerations for OE and Co. It is important to emphasize that by analyzing the results of this pilot study, a correlation between quality scores and levels of vaccination coverage of the different ASLs has been detected. In fact, in ASL1 and 2, where the highest quality scores were obtained, the levels of immunization coverage for three doses of the HPV vaccine are higher than those observed in ASL3. Further work could identify a target value accepted by the Health Authorities. The strength of this approach lies in the possibility to define a common set of quality events (e.g., the fact that a patient or a doctor asserts efficient the HPV service delivered by a vaccination center) for a set of providers of the same type (e.g., vaccination centers). Thus, even if the performance of a service delivery can be modeled by different KPIs due to many different factors (e.g., clinical governance, budget limits, etc.), quality evaluation of a single provider and benchmarking among providers can be performed through the probability assessment of such aforementioned events. In summary, our approach explicitly represents the probabilistic reasoning of a stakeholder (e.g., patient, doctor, etc.) who evaluates the quality of a healthcare provider. This approach can be used to predict the quality of the delivery of a service, and it is possible to build different models, assuming different stakeholders, weights and indicators.

The weaknesses of such an approach is in the startup effort and complexity necessary to define the resulting model, as it requires a bottom up analysis in order to represent the service delivery process and the evaluation from different stakeholders’ points of view. The subsequent implementation phases, i.e., the observation of real providers during the delivery of services and the resulting quality assessments, are only expensive in terms of time and costs proportional to the duration of the evaluation period, but not complex to perform.

Materials and Methods

A three step process was defined and used. It is mainly based on the algorithm described in the mathematical appendix in Supplemental Material, and more extensively in a recent PhD Thesis.

In our algorithm, the problem of evaluating the quality of a health care service was handled in a Bayesian inferential setting using a general framework by which it is possible to handle different kinds of information (i.e., managing uncertainty and vagueness together) related to the events of interest. In this context, it is not necessary to perform any sensitivity analysis, statistical tests or confidence interval evaluation.

In this particular application we were interested in the probability evaluation of three main events: E1, E2, E3, respectively defined by the propositions “You assert communicatively efficient the HPV vaccination service,” “You assert organizationally efficient the HPV vaccination service” and “You assert comfortable the HPV vaccination service.” The term “You” in each proposition is interpreted as an HPV vaccination service stakeholder (real as a patient, a doctor, a nurse, or a parent, or virtual as an “expert,” i.e., a combination of the real stakeholders). Thus, in our model, what we refer to as “scores” are precise probabilities of well-defined events related to the quality of a service delivery. It is therefore not necessary to assess confidence intervals or apply any statistical tests because the model is deliberately sensitive.

Between April and July of 2010, three ASLs in the Lazio region of Italy were selected, based on their willingness to participate, to collect stakeholder’s opinions, i.e., “subjective data,” on quality aspects of HPV vaccine delivery. The working group decided to represent communicational efficiency (CE) as the means and time of communicating information related to vaccination (with oral or written support before or during vaccination, or not provided at all), organizational efficiency (OE) by the time spent by a user in a waiting room and the time spent by a user for vaccination. Comfort (Co) was represented by the opening times and the availability of magazines in the waiting room. An “ad hoc” questionnaire was administered to selected stakeholders: parents of HPV immunized subjects, adolescent, and young women (12–25 y old) vaccinated against HPV, medical doctors and nurses working in the ASLs. There were three sections in the questionnaire: (1) an anonymous demographic section with questions on age, sex, education, marital status and occupation; (2) an informative section on the functioning of a generic vaccination service as presented by institutional guidelines; (3) a judgment section containing 8 questions on CE, 16 questions on OE, and 8 questions on Co. Stakeholders were asked to express opinions in terms of probabilities, i.e., values ranging from 0% to 100%, of the events E1, E2, and E3 related to the HPV vaccine delivery assuming different performance conditions (H). The aforementioned questions were prepared based on the general model described by Coletti et al. and were related to three main conditional events defined as follows:

E1|H: i.e., you assert communicatively efficient the HPV vaccination service, assuming the performance condition H;

E2|H: i.e., you assert organizationally efficient the HPV vaccination service, assuming the performance condition H;

E3|H: i.e., you assert comfortable the HPV vaccination service, assuming the performance condition H.

The term “you” in each proposition can be interpreted as an HPV vaccination service stakeholder and H can be defined by ranges or values of selected KPIs which describe the service delivery.

For example, one question was “At what level between 0 and 100% do you assert communicative efficient the HPV vaccination service, assuming that an operator of the staff gives information on the vaccination by means of a leaflet in the waiting room?” Another question was “At what level between 0 and 100% do you assert organizational efficient the HPV vaccination service, assuming that the waiting time is 15 minutes and the vaccination time is 10 minutes?” A single answer from a stakeholder, i.e., an opinion, can be used to assess directly the conditional probability of the investigated conditional event. If, as in our case, there are more answers related to the same questions, these can be integrated for the assessment. In this application the working group proposed to assign a specific relevance (weight) to the opinions of each stakeholder category, with the chosen weights 0.25, 0.20, 0.30, and 0.25 for parents, vaccinated women, doctors and nurses, respectively. Summing the stakeholders’ opinions and taking into account the aforementioned weights, the group derived the opinion of the so-called Expert, i.e., a virtual HPV vaccine service stakeholder.

During the same period of the survey (April–July 2010), KPIs related to the performance conditions that were investigated in the judgment section of the questionnaire were observed in the ASLs during the service delivery and recorded in a paper form by the operators of the QuaVaTAR group. These data, i.e., “objective data,” included waiting times, length of vaccinations, means of communication and comfort features.

Step 3 of the measurement process (activities and criteria)

At the end of the evaluation period, objective and subjective data were entered into a Microsoft Excel spreadsheet that implemented the algorithm described in the mathematical appendix in Supplemental Material. The QuaVaTAR group assigned relevance to the quality aspects under investigation by means of some numerical weights: 0.5, 0.3, and 0.2 for CE, OE, and Co, respectively. With all of these available data, the algorithm was allowed to compute the HPV vaccination service quality scores and finally the overall quality scores of each of the ASLs.

Conclusions

A new approach to quality measurement and benchmarking of health care services was investigated. This approach was based on a well-defined mathematical algorithm composed of three main steps and based on a well-defined probabilistic inference framework. Step 1 involved the elicitation of the best and worst performance conditions related to service delivery by means of a survey that considered different service stakeholders’ points of view (i.e., subjective data collection). Step 2 involved the observation and collection of data regarding the performance during the service delivery by some providers under investigation (i.e., objective data collection). Step 3 involved the elaboration of all gathered data using subjective data from step 1 to define a standard to test the objective data from step 2.

To realize the QuaVaTAR pilot study, this new approach was applied to measure the quality of HPV vaccination services in the Lazio Region of Italy by evaluating three related quality aspects and four stakeholders’ perspectives. It was possible to represent the probabilistic reasoning of stakeholders who evaluate the quality of a HPV vaccination service. This approach could also be used to predict the quality of service delivery from an integrated (i.e., Expert) point of view. All ASLs have margins for improvements and optimal quality results can be assessed in terms of better performance conditions, confirming the relationship between the resulting quality scores and HPV vaccination coverage.

The QuaVaTAR pilot study was recognized as exemplar research by the Lazio Region Government with an explicit citation in the Regional Vaccination Plan for the period 2012–2014and future work is planned to extend the study to all 12 ASLs of the Lazio Region in collaboration with Regional Health Authorities.