Development and Validation of a Performance Assessment Checklist Scale for Vaccine Administration.

Objective Vaccination technique is a crucial skill for medical trainees to learn, especially in the current coronavirus disease 2019 pandemic. To this end, validated assessment tools are essential in teaching appropriate techniques. However, valid instruments for assessing vaccine administration skills have not yet been developed. We therefore explored the development and validation of an assessment tool for vaccination techniques based on expert consensus. Methods We implemented a modified Delphi process to develop a vaccination technique assessment tool. We then conducted a validation study to establish the reliability and validity of the tool. Results Two rounds of the modified Delphi process were performed to generate a 19-item, vaccination performance assessment checklist. In the validation study, the linear weighted kappa value for inter-rater reliability of the overall checklist score was 0.725. Spearman's correlation coefficient between the mean checklist score and the global rating was 0.98 (p<0.01). Conclusions This is a pioneering study examining the development and validation of an assessment tool for vaccine administration techniques. The tool will be widely used in vaccination-related education.

Introduction

Infectious diseases are caused by microorganisms. They continue to pose a serious threat to public health around the world (1,2). The 2019 global coronavirus disease (COVID-19) pandemic is still ongoing, causing severe social and economic upheaval as well as devastating health consequences (3).

Vaccines are essential for the prevention and control of infectious diseases (4). The smallpox vaccine, which was developed by the physician Edward Jenner in 1796, was the first successful vaccine developed against contagious diseases (5). In the centuries since, the number of vaccine types has increased one after another, significantly reducing the burden of infectious diseases. Recently, vaccines against severe acute respiratory syndrome coronavirus 2 have been administered in many countries around the world in order to control the COVID-19 pandemic.

Given this situation, vaccine administration techniques are an even more important skill than ever for medical students and physicians to master. Rigorous measurements and assessments are necessary to ensure that they acquire and maintain clinical skills. However, to our knowledge, there is no published, validated performance assessment tool for evaluating one's vaccination technique, although methods have been developed to assess other clinical procedures, such as central venous catheter insertion, chest tube insertion, and lumbar puncture (6-8).

In Japan, the lack of opportunities for medical students to experience medical procedures during their clinical training has long been an issue (9). Although the shift from traditional “observational” to “participatory” clinical practice was touted in the 2000s, the problem of inadequate opportunities has not improved commensurately since. In this regard, in recent years, several universities in Japan, including the University of Tokyo, have started offering courses wherein students, under the supervision of physicians, administer vaccinations to hospital staff. However, while such courses are believed to be effective in training medical students, the lack of tools for assessing the approach has prevented us from confirming its effectiveness. Such tools are expected to prove very useful for assessing vaccine delivery technique and training students around the world.

We therefore explored the development of an assessment tool for vaccination administration techniques and provided evidence supporting the reliability and validity of this tool.

Materials and Methods

Setting

There are multiple routes of vaccine administration and several vaccination sites for each route. In the present study, we developed an assessment list based on intramuscular or subcutaneous vaccination of the upper extremities, which are most common in the daily clinical setting.

Modified Delphi methodology

A modified Delphi method was implemented to develop a content-fidelity tool in order to evaluate the vaccine delivery techniques. The Delphi method is a tool employed to reach a reliable consensus on a specific problem or point among experts in a given field (10). It consists of repeated questionnaires and anonymous feedback summarized according to group responses (10,11). It has been shown to provide adequate evidence for the content validity of research products (12).

In the traditional Delphi method, the initial set is created by the participant in the first round of the Delphi process. In the present study, after reviewing the literature, the research team employed a modified Delphi method to create an initial list. We reviewed the literature extensively but were unable to identify a validated tool for assessing vaccine administration. Therefore, we developed the initial 19-item list based on the Vaccination Guidelines [World Health Organization (WHO) and the National Immunization Technical Advisory Groups (NITAGs) worldwide] and a systematic review literature concerning tools for assessing clinical procedure skills (13-22).

Panelists

Inclusion of multiple stakeholders in the Delphi process facilitates the acceptance of feedback and effective implementation of the tool (23). Therefore, we aimed to include panelists to ensure representation of five groups of stakeholders: infectious disease specialists, attendings, educational experts, nurses, and medical students.

An expert group composed of these five stakeholder groups was constructed using purposive sampling combined with the snowball sampling method. The research team compiled a list of potential panelists (purposive sampling) and asked them to provide the names of other potential panelists (snowball sampling). A total of 47 panelists (22 infectious disease specialists, 10 attendings, 5 education experts, 5 nurses, and 5 medical students) were thus invited by e-mail. Infectious diseases specialists included both pediatric and adult infectious diseases specialists. The attending physicians worked in a primary-care setting, where they performed vaccination services themselves and had extensive experience in teaching these skills to medical students and residents. All nurses had experience administering immunizations and providing assistance in such procedures. All of the medical students were sixth-year students (clinical students) at the University of Tokyo and had experience participating in the vaccination course at the university, so they were able to vaccinate themselves easily and smoothly. Study participation was voluntary and anonymous. The panelists received a small cash reward for completing the survey. All panel members consented to participate in the study.

Data collection and analyses

We used SurveyMonkey to distribute the survey electronically. At the start of each round, panelists received feedback on the previous round. We asked panelists to rate each item on a five-point Likert scale as follows: 1 (unimportant), 2 (of little importance), 3 (neutral), 4 (relevant), and 5 (very relevant). We gave the panelists the opportunity to suggest word changes, identify duplicates, and suggest additional questions in each round. Means and standard deviations were calculated, and the list was edited on the basis of panelists' comments.

Although there are no clear rules for determining whether or not a consensus has been reached in the Delphi procedure, by citing existing studies, the following three criteria were used to determine whether or not a consensus had been reached: 1) mean value of ≥4, 2) a standard deviation (SD) of <1.25, and 3) ≥75% of panelists scoring 4 or 5 (11,24,25). A thematic analysis of the free text responses was performed. Data from each round were analyzed by three researchers (HF, TM, and DS). When questions remained for discussion, other coauthors were invited to comment on the question.

Validation phase

After completing the scale, a screening survey was conducted on the residents of Suwa Central Hospital. Candidates were all trainees (postgraduate years 1-5) who were in the hospital during the validation study. All 27 eligible trainees agreed to participate in the study. The vaccine administration procedure was assessed by two independent experts using a checklist scale to evaluate the effectiveness of vaccine administration. The assessment was scored on a five-point Likert scale (1=unable to perform step, 2=performs step with much help, 3=performs step with some help, 4=performs step with minimal help, and 5=performs step easily and with fluidity). Trainees found that participation in the survey was optional and not related to their evaluation in the hospital. Informed consent was obtained from all participants.

Inter-rater reliability was tested with the aid of the calculation of a linear weighted kappa value. The agreement level can be interpreted as follows: ≤0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.60, moderate agreement; 0.61-0.80, substantial agreement; and >0.80, almost perfect agreement (26).

To observe concurrent validity, the Spearman rank correlation coefficient was computed among the overall performance assessment checklist scale scores and the global rating of the procedures. The global rating is a rating throughout the procedure and was determined using the following question: “On a scale of 0 to 10, where 0 is the worst vaccinator possible and 10 is the best vaccinator possible, how would you rate this vaccinator?” A Spearman correlation coefficient value of >0.30 is considered meaningful (27). All data were analyzed using the SPSS Statistics software program, ver. 27.0 (IBM Japan, Tokyo, Japan).

Ethical considerations

All participants provided their written informed consent. The study was approved by the Institutional Review Board of the University of Tokyo (2020364NI).

Results

Instrument development through modified Delphi process

A total of 45 experts participated in this Delphi survey. Seven participants (16%) were included in the sampling using the snowball method. Participants came from all over Japan (Table 1).

Table 1.

Demographics of the Delphi Panelists.

Characteristic		No. (%)
Sex
Female		17 (38)
Male		28 (62)
Years in practice (for doctors and nurses only)
<10 years		1 (2)
10-19 years		24 (53)
20-29 years		11 (24)
≥30 years		4 (9)
Profession
Doctor		35 (78)
Nurse		5 (11)
Medical student		5 (11)
Practice settings
Community hospital		20 (44)
University/University hospital		17 (38)
Clinic		7 (16)
Other		1 (2)
Regions
Tohoku		4 (9)
Kanto		18 (40)
Chubu		10 (22)
Kinki		6 (13)
Chugoku		1 (2)
Shikoku		3 (7)
Kyushu		3 (7)

In the first round of the Delphi process, all 45 panelists returned completed questionnaires. In this first round, 5 of the 19 items were supported. Two items with similar meanings were combined. Two items were removed because they did not meet the acceptance criteria and received several negative comments from committee members. Three items were added to the list at the suggestion of the panelists. The wording of 10 points was changed according to the opinions of the panelists. Therefore, based on the results of the first round, a new list consisting of 19 items was developed.

In the second round of the Delphi process, all 45 panelists reconstructed a fully completed list. Because all questions met the three accepted criteria, it was concluded that no further rounds were required and that a consensus had been reached. Therefore, we obtained a 19-item checklist for evaluating the effectiveness of the vaccine administration technique (Table 2).

Table 2.

Results of the Final Delphi Round.

Items	Mean score (SD)	Number of panelists who scored 4 (relevant) or 5 (very relevant) (%)
1. Introduces self to patient	4.07 (1.14)	36 (80)
2. Checks patient’s identity	4.93 (0.25)	45 (100)
3. Checks that indication exists for the vaccine administration	4.96 (0.21)	45 (100)
4. Obtains consent for procedure from patient or representative	4.42 (0.84)	39 (87)
5. Prepares and checks for necessary equipment	4.73 (0.54)	43 (96)
6. Performs hand hygiene	4.84 (0.52)	44 (98)
7. Positions the patient so that his/her arms are in the proper position	4.73 (0.54)	43 (96)
8. Identifies proper site of needle insertion*	4.82 (0.44)	44 (98)
9. Checks for alcohol sensitivity**	4.44 (0.81)	38 (84)
10. Disinfects the puncture site with an antiseptic (alcohol or alcohol free wipes)**	4.22 (0.85)	37 (82)
11. Informs the patient that the skin will be punctured	4.60 (0.54)	44 (98)
12. Punctures*	4.82 (0.39)	45 (100)
13. Ensures that there is no numbness after the puncture	4.27 (1.01)	37 (82)
14. Injects the vaccine	4.67 (0.71)	44 (98)
15. Removes needle	4.42 (0.97)	37 (82)
16. Disposes of used items separately in the trash according to whether they are sharp or infectious.	4.91 (0.29)	45 (100)
17. Documents	4.58 (0.87)	39 (87)
18. Communicates with medical and administrative staff as appropriate throughout the entire process	4.76 (0.71)	43 (96)
19. Communicates with the patient as appropriate throughout the entire process	4.89 (0.32)	45 (100)

*For details, please refer to the guidelines of each country and facility. The procedures commonly performed in Japan are as follows:

•Subcutaneous injection: Punctures in the lower third of the midline of the posterior side of the upper arm at a 45° angle to the skin

•Intramuscular injection: Punctures in the intersection of the anteroposterior axillary line (the line connecting the upper end of the anterior axillary line with that of the posterior axillary line) and the vertical line from the mid-acromion or 2-3 finger breadths below the mid-acromion at a 90° angle to the skin

**In a strict sense, if the skin is visibly clean, disinfection is not required medically. However, it is customary in some countries and facilities. Please follow the guidelines of each country and facility.

Instrument validation

Table 3 shows the characteristics of the verification survey participants. There were no missing data.

Table 3.

Demographics of the Participants of the Validation Survey.

Characteristic	No. (%)
Sex
Female	11 (41)
Male	16 (59)
Years of residency
1	5 (19)
2	4 (15)
3	6 (22)
4	9 (33)
5	3 (11)

The linear weighted kappa value for the total checklist score was 0.725 (95% confidence interval 0.551-0.899), indicating significant agreement. The Spearman correlation coefficient between the total checklist score and the overall procedure score was 0.98 (p<0.01), exceeding the cut-off value of 0.30.

Discussion

In this study, we developed a checklist scale for evaluating the effectiveness of vaccine administration procedures and provided evidence of its reliability and effectiveness. We used a modified Delphi method to develop the scale while encouraging all stakeholders to participate as panelists. During the validation study, the scale demonstrated solid reliability and validity, suggesting its utility for improving immunization skills based on an assessment of trainees' immunization practices.

Although the WHO and NITAG do not recommend skin cleansing in all cases, we decided to include it in the list. This is because in some countries, the procedure is considered normal. To our knowledge, especially in Japan, most medical institutions perform this procedure, and if the vaccinator does not perform it, the patient may become anxious. However, we have mentioned in the footnotes that the actual implementation of this assessment tool should be done in line with the guidelines of each country and institution, as there may be instances where certain procedures are not normally carried out.

Specific methods of determining where to vaccinate vary by country and institution. For example, intramuscular injection into the deltoid muscle, three fingers under the acromion, is generally recommended. However, in Japan, a previous study (28) stated that this site was adjacent to the axillary nerve and the posterior brachial artery and thus was unacceptable. Instead, the intersection of the anteroposterior axillary line (the line connecting the top of the anterior axillary line with the tip of the posterior axillary line) and the vertical line at the mid-acromion has been proposed as a suitable injection site (29). In view of the above, the assessment items are simply listed as “Identifies proper site of needle insertion” and “Punctures,” with mention made in the footnotes that the actual implementation of this assessment tool should be done in line with the guidelines of each country and institution. In addition, for reference, we have described the procedure commonly performed in Japan.

One of the strengths of our study is the response rate during Delphi rounds. Response frequency may affect the validity of the consensus obtained (30). The response rate in this study was very high, at 100%, in all Delphi rounds, which seems to enhance the validity of the consensus. In addition, the validity of the study was secured through the participation of various stakeholders, such as doctors, nurses, and medical students.

However, several limitations associated with the present study also warrant mention. First, the panelists did not meet in-person at the Delphi round for discussion. In face-to-face discussions, anyone can express their views, but the downside is that one member can influence other members and their ratings. In addition, in-person discussions are being hampered by the COVID-19 pandemic. Therefore, we opted for a modified Delphi process with a non-face-to-face format. Second, the number of panel members was relatively small. There is no single rule concerning the ideal number of panelists to include in the Delphi method (30). A minimum of 20 panelists is recommended (31), but including too many panelists can result in drop-outs. The response rate in this study was 100%. Third, there was the issue of translation. Some nuances may have been lost in translation from Japanese to English. Fourth, the verification survey was conducted at one center, and the sample size was relatively small. It may be helpful to continue multicenter testing with large sample sizes. Fifth, in the verification survey, the raters were not blinded to the trainees, which could lead to prejudice. However, a recent study reported a high inter-rater reliability between direct observation by an unblinded observer and an assessment using a video recorded by a blinded rater when performing an objective structured assessment for a technical skills-based evaluation (32). Therefore, this limitation may be considered acceptable.

Conclusion

In this study, a vaccine performance assessment checklist was developed by gathering the opinions of various stakeholders. This tool has now been verified for reliability and validity and can be used for educational activities related to vaccine administration.

The authors state that they have no Conflict of Interest (COI).