Vaccine storage practices and the effects of education in some private medical institutions.

OBJECTIVES: Although vaccination rates have increased, problems still remain in the storage and handling of vaccines. This study focused on inspecting actual vaccine storage status and awareness, and comparing them before and after education was provided.
METHODS: In the primary inspection, a status survey checklist was completed by visual inspection. A questionnaire on the awareness of proper vaccine storage and handling was also administered to vaccine administrators in private medical institutions in 4 regions in Gyeongsangbuk-province. One-on-one education was then carried out, and our self-produced manual on safe vaccine storage and management methods was provided. In the secondary inspection, the investigators visited the same medical institutions and used the same questionnaire and checklist used during the primary inspection. The results before and after education were compared, by treating each appropriate answer as 1 point.
RESULTS: The average checklists score was 9.74 (out of 15 points), which increased significantly after education was provided (by 0.84, p<0.001). The participants demonstrated improved practices in recording storage temperatures (p=0.016), storing vaccines in the center of the refrigerator (p=0.004), storing vaccines with other medication and non-medical items (p=0.031) after education. The average score calculated from the questionnaires was 10.48 (out of 14 points), which increased after education (by 1.03, p<0.001).
CONCLUSIONS: This study suggests that vaccine storage practices and awareness are inadequate, but can be partially improved by providing relevant education. Repetitive education and policy-making are required to store vaccines safely because one-off education and unenforced guidelines offer limited efficacy.

INTRODUCTION

Vaccinations have contributed to decreasing serious outbreaks by preventing incidence and propagation of contagious diseases in advance. The vaccination program is a cost-effective enterprise that provides vaccines for preventing the outbreak and propagation of vaccine-preventable diseases. Implementing a high vaccination coverage rate and the effect of the vaccine on the disease will determine whether or not a vaccination program can successfully prevent an outbreak or the propagation of the disease [1]. Recently, many developed and developing countries have pushed forward with extending the coverage of required vaccinations for every resident their jurisdictions, and many countries are making efforts to eliminate the geographic and economic limits to vaccination among low and vulnerable classes. In Korea, phased extension policy was chosen by government as the coverage plan for the required national vaccinations. Thus, the private medical institution participation method of the National Expanded Program on Immunization had been implemented since 2005 [2]. Daegu Metropolitan City and Gunpo City participated in the pilot project that enforced vaccinations for 13 infectious diseases. The objects were children 0 to12 years old and the project was performed at private medical institutions; the objects were reimbursed for the entire cost of their individual inoculations [2]. This developed into the enforcement of the National Immunization Program (NIP), since 2009, in which a child can receive the part of their inoculation cost at most private medical institutions in Korea [3]. If a child 12 years old and younger is vaccinated at a designated private medical institution, the cost of the vaccination is covered by the government on the condition that the patient receives vaccinations for 8 of the required vaccinations including 22 administrations. Implementing such programs for increasing vaccination rates has been one of the elements that enabled us to prevent contagious diseases; the Korean government has been making such policy-related efforts for many years.

However, in order to ensure success for the vaccination program, the efforts not only toward increasing vaccination rates, but also toward maintaining the quality of vaccines are needed. Due to the fact that most vaccines maintain their potency for a short amount of time at room temperature, the cold chain is the core of the system used for managing the quality and safety of vaccines. In addition, the principles related to management of vaccines such as installation of thermometers, maintenance of appropriate temperatures, recording temperatures, location of the vaccines inside a refrigerator, defrosting a frozen vaccine, as well as using the refrigerator exclusively for vaccine storage must be followed in order to finally administer the vaccine at full titer. For example, in one localized outbreak of measles in the U.S. during the 1970s, illness was associated with inoculation at a medical institution where the vaccines were stored on a refrigerator door shelf, not in the refrigerator's center [1]. In Australia, it was claimed that incorrect storage of vaccines could have contributed to the resurgence of pertussis [1]. Likewise, the vaccination program can be successful in preventing the occurrence and spread of illness only if the effect of a vaccine itself can be maintained through biological preparation; the storage and management of the vaccine are important, but the regulations are incomplete, and the distribution enterprise and healthcare institution management in the field is even more vulnerable [4]. Moreover, there is almost no research that has intensively discussed the actual state of vaccine storage and management, and standardized guidelines along with systematic education programs to address these issues are rare. This study focused on inspecting actual vaccine storage status and awareness at private medical institutions, and compared them before and after education was provided.

METHODS

I. Study Subjects

Among the private medical institutions located in 4 regions (Nam-county in Pohang-city, Yeongju-city, Sangju-city, and Mungyeong-city) in the Gyeonsang buk-province, we randomly selected 10 medical institutions (2 hospitals and 8 clinics that had been entrusted with the NIP and that had carried out infant/children vaccination in August 2010) from each region. Out of a total of 40 private medical institutions, we investigated 39 that had agreed in advance over the phone to our visit. We requested a visit to those medical institutions, but we did not speak about the details such as whether or not the medical institutions possess a refrigerator exclusively for vaccine storage, or asking for records of the refrigerator temperature.

The investigation was performed over two time periods (primary and secondary inspection). The person in charge of the vaccinations and a private medical institution supervisor of each public health center in the jurisdiction of the private medical institutions were visited. Using the checklist, we observed the actual status of vaccine storage and handling, and investigated the awareness of the person in charge of vaccine management using the questionnaire. The questionnaire was performed under the principle that stated that the person in charge of vaccine management in each private medical institution would be the subject of the research, but in case there was not a designated person in charge, the questionnaire was performed based on the person who conducted the most work, on average, relating to vaccine management.

The checklist was prepared for grasping the actual state of vaccine storage and handling, and was self-produced by referring to existing studies and foreign guidelines [1,5,6]. The checklist content includes the following: whether or not there was a designated person in charge of vaccine management, whether or not the person had undergone vaccine management training, whether or not the temperature for vaccine storage was monitored and recorded, whether or not the appropriate temperature (2℃ to 8℃) was maintained, whether or not the expiration date was heeded, and whether or not equipment of refrigerator reserved for vaccine storage and emergency provision. A person in charge of vaccination and a private medical institution supervisor of a public health center inspected the checklist items.

The questionnaire for determining the awareness of proper vaccine storage and management procedures was self-produced by compiling from previous studies and foreign guidelines [1,5,6]. After inspecting the checklist, the questionnaire was administered by person in charge of the public health center vaccinations. Using Likert scale for the questionnaire answers, out of five answer choices ('strongly agree', 'agree', 'neither agree nor disagree', 'disagree', 'strongly disagree'), we let the subjects provide answers to the questions according to their knowledge regardless of the present storage practices.

After the questionnaire was completed, the people in charge of vaccinations at each public health center who were also well-informed of knowledge related to vaccine storage supervised the one-on-one education of the questionnaire subjects. The education manuals used were leaflets that were self-produced in advance.

The content of the education manuals were as follows: 1)When receiving delivery of vaccine, to maintain the cold chain, check whether or not the packaging has been destroyed, and check the expiration date. 2) When storing a vaccine, the appropriate temperature is 2℃ to 8℃. 3) Record the temperature of the refrigerator every hour. 4) Preserve a daily record of the temperature over the previous two years. 5) Install a refrigerator thermometer in the center of the refrigerator. 6) Do not store vaccines in areas where a change in temperature occurs easily, such as a refrigerator door shelf or the front of a refrigerator. 7) Do not keep other kinds of drugs or food items in the vaccine refrigerator. 8) Store vaccines according to their type for the purpose of easy distinction. 9) Expired vaccines need to be discarded immediately. 10) The emergency power supply, in case of blackout, needs to be checked once a week. 11) The opening time of multi-dose vials needs to be recorded. 12) In case a vaccine is opened, do not store it again and do not re-use it the next day.

The secondary inspection was performed using an identical questionnaire and identical checklist by revisiting the same medical institution 4 weeks after the day of the primary inspection. The subjects of the secondary inspection were the same persons as those in the primary inspection.

II. Statistical Analysis

In order to compare the actual status of vaccine storage and handling, the appropriate procedure related to each item was treated as 1 point. In the questionnaire, where the positive answers were the correct answers, 'strongly agree', and 'agree' were treated as 1 point, and 'neither agree nor disagree', 'disagree' and 'strongly disagree' as 0 points. Where the negative answers were correct, 'disagree' and 'strongly disagree' were treated as 1 point, and 'neither agree nor disagree', 'agree', and 'strongly agree' as 0 points. In order to determine if each item of the questionnaire and the checklist had any distinction according to the type of medical institution, we used the chi-squared test. We added all the scores for each item, and we compared the average score of the actual status and awareness by type of medical institution through one-way ANOVA.

We compared whether or not there was a difference in the actual procedures followed before and after education and inspection, according to the items, using the Mcnemar's chi-squared test. We compared the total score of the primary and secondary inspections using the paired t-test. We did a comparison of the awareness questionnaire results, according to the items, using the Wilcoxon's signed-rank test. And the total scores of the primary and secondary inspection were compared using the paired t-test. All statistical analyses were performed using SPSS version 17.1 (SPSS Inc., Chicago, IL, USA).

RESULTS

I. General Characteristics

We report the general characteristics of the questionnaire subjects and the relevant private medical institution in Table 1. According to the analysis of the research subjects, 18 (46.2%) were male, 21 (53.8%) were female, and the majority were in their 30s and 40s (16 people in each 10-year age range, 41.0%). The order according to duty was as follows: 20 physicians (51.3%), 12 nurses (30.8%), 3 nurse aides (7.7%), 2 pharmacists (5.1%), and 2 members of office staff (5.1%). For work experience at a medical institution, 37 people had worked for more than 5 years, and the majority had worked for more than 11 years (25 institutions, 64.1%). Among the 39 private medical institutions investigated, there were 8 hospitals (20.5%), 12 pediatric clinics (30.8%), and 19 other clinics (miscellaneous clinics, 48.7%). As for the average number of vaccinated people, in one week, the majority of medical institutions (25 institutions, 64.1%) were vaccinating less than 20 people, and 14 institutions vaccinated less than 10 people (35.9%). Fifteen institutions had more than 13 kinds of vaccines (38.5%).

Table 1

General characteristics of questionnaire subjects and the private medical institutions where they work (n=39)

II. Vaccine storage and Management Practice

Table 2 shows the checklist items for the vaccine storage and management practice inspection, as well as the first set of inspection results. Out of the 39 private medical institutions, only 3 (7.7%) had refrigerators exclusively for storing vaccines, and they were all hospital-level medical institutions. Most of the medical institutions were maintaining the appropriate temperature of the refrigerator for storing vaccines. All 12 pediatric clinics had maintained an acceptable temperature but 2 miscellaneous clinics and 1 hospital maintained the refrigerator temperature deviated from the acceptable range. There was no significant difference among the three types of institutions. In 2 medical institutions (5.1%), vaccines that needed to be in the freezer were stored inside the refrigerator, and both of them were pediatric clinics. A total of 10 (25.6%) medical institutions stored multi-dose vials without recording the opening dates and as many as 6 miscellaneous clinics (31.6%) were shown but there were no significant differences among the three types of institutions. Other than the aforementioned items, some of items that showed significant differences according to the type of medical institution were whether or not the medical institutions had a refrigerator exclusively for vaccine storage, whether or not the institution had emergency power supplies in case of a blackout, and whether or not there were extra refrigerators in case of an emergency. There were no pediatric clinics that with an emergency power supply in case of a blackout, but extra refrigerators were equipped the most in pediatric clinics (91.7%). Overall, item number 11, whether or not the medical institution stored expired vaccines, was adhered to the most as none of the 39 institutions were storing expired vaccine.

Table 2

Vaccine storage and handling status according to itemized checklists by institution types and change after education (n=39)

Values are presented as no. of 'yes' answers (%).

1Changes from primary inspection (before education) in 'yes' answers of all 39 institutions were calculated using the McNemar χ2 - test.

2Differences according to institution type were evaluated within each inspection by Fisher's exact test of the χ2 - test.

3p-value could not be calculated because there were no 'yes' answers.

The score for the checklist of the primary inspection was 9.74 ± 0.39 (SE) out of 15 possible total points. The comparison results, according to the kind of medical institution, were in the order of pediatric clinics 10.58 ± 0.39, hospitals 9.62 ± 0.62, and miscellaneous clinics 9.26 ± 0.71. There were no significant differences in checklist scores according to the type of medical institution (Figure 1).

Figure 1

Average score of vaccine storage and handling status and change after education by institution type.

Bars represent the mean±SE, and a perfect score is 15 points; 4 weeks after education, pediatric clinics, other miscellaneous clinics, and the three types in aggregate had increased average scores by 0.75, 0.95, and 0.84 points, respectively, compared with before education (*p<0.05, **p<0.01, ***p<0.001). There were no significant differences among institution types in scores either before or after education.

III. Awareness Investigation

The questionnaire questions and results for the awareness investigation before education are shown in Table 3. The question that was answered incorrectly the most was that using even a general refrigerator is appropriate for storing vaccines as long as the temperature is maintained properly (item 4), and only 6 people (15.4%) out of all subjects answered 'strongly disagree' or 'disagree'. There was a significant difference according to the type of medical institution, and in particular, all the subjects of the pediatric clinics had the wrong answer for that question. Items answered correctly included 'a refrigerator for vaccine storage only is needed' (item 2), 'vaccines should be stored in a refrigerator designated only for vaccine storage' (item 3), which had 26 people (66.7%) and 25 people (64.1%), respectively. Though there was no significant difference according to the types of medical institutions there was a low awareness demonstrated in pediatric clinics. There were 3 people (7.7%), all from pediatric clinics, who knew that freeze and thaw cycles are allowed with frozen vaccines (item 9), and the difference according to types of medical institutions was significant. The question with the highest number of subjects that showed proper awareness was that multi-dose vials of vaccines should be discarded within a certain period after opening (item 8).

Table 3

Awareness of vaccine storage and handling according to question by institution type and change after education (n=39)

Values are presented as no. of correct answers (%).

1Changes from primary administration (before education) in the correct answer for all 39 participants taken together were calculated using the exact test of Wilcoxon's signed rank test.

2Differences according to institution type were evaluated within each inspection by Fisher's exact test of the χ2 - test.

3p-value could not be calculated because all the answers were correct.

The average awareness score of the primary inspection was 10.48 ± 0.34 (SE). Based on the results according to the type of medical institution, the order was as follows: hospitals 11.75 ± 0.86, miscellaneous clinics 10.36 ± 0.49, and pediatric clinics 9.83 ± 0.50. The resulting value was highest overall for the hospitals but there was no significant difference (Figure 2).

Figure 2

Average score of awareness of vaccine storage and change after education by institution type.

Bars represent the mean±SE, and a perfect score is 14 points; 4 weeks after education, pediatric clinics, other miscellaneous clinics, and the three types in aggregate had increased average scores by 1.25, 1.00, and 1.03 points, respectively, compared with before education (**p<0.01, ***p<0.001). There were no significant differences among institution types in scores either before or after education.

IV. Comparison of Before and After Education

We compared the actual status of vaccine storage between the primary and secondary inspection. Medical institutions that record the refrigerator temperature (item 4) increased significantly from 27 institutions (69.2%) to 34 institutions (87.2%). The number of medical institutions that stored vaccines in the center of the refrigerator (item 8) increased significantly from 17 institutions (43.6%) to 26 institutions (66.7%), and institutions that stored vaccines with other kinds of medications or food significantly decreased from 14 institutions (35.9%) to 8 institutions (20.5%) (Table 2). The average secondary inspection score of the checklist for all of the private medical institutions together was 10.58 ± 0.34 (SE), and it significantly increased by 0.84 points compared to the primary inspection (p<0.001) (Figure 1). For each type of medical institution considered separately, the score after education for hospitals increased by 0.75 (p=0.320), pediatric clinics by 0.75 (p=0.032), and miscellaneous clinics by 0.95 (p=0.002) (Figure 1).

Awareness after education, compared to before the education, significantly improved in items 5-7, which were about the completion of temperature records, exclusive storage of vaccines, and selective storage of vaccines according to the type and expiration date, as well as items 10-14 which were light-proof storage, installation of a thermometer in the center of the refrigerator, storing the vaccines in the center of a refrigerator, installation of an independent generator, and educating people in charge of vaccines. The awareness score of all the medical institutions after education was 11.51 ± 0.28 (SE), and it increased by 1.03 points compared to before the education (p<0.001) (Figure 2). According to the types of medical institutions, hospitals increased by 0.75 points (p=0.197), pediatric clinics by 1.25 points (p=0.006), and miscellaneous clinics by 1.00 point (p=0.005) (Figure 2).

DISCUSSION

If a group of people has a high immunity level, even with some susceptible people in the group, there is a small possibility that an outbreak can occurs. In particular, if a human is the only host organism, increasing the level of immunity of a group can eradicate an infectious disease [7]. Using this method, smallpox has been the only disease humankind has completely eradicated up to the present time, but the number of eradicated diseases can be increased through vaccination programs. Diseases such as polio and diphtheria are at the level of being almost entirely eradicated [2]. However, until the present time, the focus has been on coverage rates of vaccines, and the problems related to the quality of handling after the production of a vaccine have received little attention. Then, Lerman and Gold [8] in 1971 and Krugman et al. [9] in 1974 reported that one plausible factor that resulted in outbreak of measles in children with a vaccination history was improper storage of vaccines and thus the effect of vaccines could have been reduced. Therefore, problems related to storage and management of vaccines captured people's attention. However, reviewing the following studies, recognition related to the importance of vaccine storage has not been well established yet, and the actual post-production handling conditions have been diverse [5,10-15]. Steinmetz et al. [10] investigated the actual conditions for storing live measles, mumps, and rubella virus vaccines. In 1981, 60% of the investigated medical institutions had the refrigerator temperature at higher than 8℃, and they reported that only 40% were storing vaccines at the recommended location in the refrigerator. In the research performed by Hunter [11], only 3 medical institutions out of 36 inspected had installed a thermometer inside the refrigerator. In Bishai et al. [12] 22% of refrigerators had an inside temperature that was higher than 8℃, and 46% had stored the vaccines on the refrigerator door shelf. Thakker and Woods [13] stated that out of 40 subjects who answered their questionnaire, only 16 knew the correct vaccine storage method. Yuan et al. [5] investigated both knowledge and actual conditions related to the storage and handling of vaccines from a primary care physician's office, and only 6% of participants provided correct answers to all 4 of the questions, which showed the necessity of awareness improvement. At the same time, they stated that only 10% of the investigated medical institutions had installed thermometers inside their refrigerators, and about one-third of them had temperatures outside the recommended range. Just as mentioned earlier, despite the fact that many studies have reported on the importance and seriousness of vaccine storage and management conditions, the efforts of health personnel including policy-decision-making institutions to remedy the situation have been insufficient.

In Korea, the distribution system for vaccines in private medical institutions is one in which the hospitals and clinics have direct deals with either the manufacturing company or sales company, or are being provided by the wholesale markets; it is very important to properly maintain the cold chain throughout all these processes. The cold chain, also known as the low-temperature distribution system, is a delivery system that maintains a low temperature from the time of production until the time of supply, and is the core principle of management of vaccine quality and safety. Although this study was not able to widen the research to include the whole distribution process including the manufacturing company, sales company, and the wholesale market, the authors apprehended the actual conditions of vaccine storage and handling in private medical institutions located in 4 regions in Gyeongsangbuk-province that performed infant and child vaccinations.

There are two ways in which the titer of the final administrated vaccine can be directly decreased due to an incorrect storage method: one way is to use vaccines that were not stored at the appropriate temperature, and the other is to use vaccines whose expiration date has passed [6,9]. The latter can include using opened multi-dose vials of vaccine that have been exceeded a certain acceptable time period for use, after they have been opened. The questionnaire items that most directly reflect the two abovementioned ways are 6, 7, 11, and 12. The checklist for actual status of item 6 is the measurement of direct storage temperature, and item 7 showed that vaccines were stored at a wrong temperature. Items 11 and 12 reflect whether it is possible to use vaccines anytime from slightly to extremely past their expiration date. As a result of the primary investigation, 3 medical institutions were not maintaining the refrigerator temperature at 2℃ to 8℃, and there were even as many as 2 institutions that had stored vaccines in the refrigerator that were supposed to be stored in the freezer. The former institutions were all hospitals or miscellaneous clinics, whereas, the latter were all pediatric clinics. However, we cannot determine whether a certain type of medical institution had an actual serious problem related to vaccine storage based only on the results mentioned above, because the temperature of the refrigerators were measured only once, and vaccines that have to be kept inside the freezer like varicella-containing vaccines are administrated frequently at pediatric clinics. In addition, this is because there was a practical limit on the number of investigation subjects. None of the medical institutions that participated in the investigation that stored expired vaccines, but 25.6% of the institutions did not record the opening date of multi-dose vials of vaccine. Out of those 25.6% institutions, there were relatively more miscellaneous clinics, but the difference among types of institutions was not significant. The important thing to keep in mind is that administrating vaccines without abiding by items 6, 7, 11, and 12 can possibly lead to failure of formation of the antibody.

Among the private medical institutions investigated, 3 of them were furnished with refrigerators exclusive for vaccine storage, but most of them were storing vaccine in typical refrigerators. For storing vaccines, stand-alone freezers and refrigerators without freezers are preferred because maintaining the temperature is easier than with a combination unit [6]. Even if the equipment has a refrigerator and freezer combined, each compartment must have an exterior door and temperature-maintaining equipment. The type of refrigerator that should not be used for storing vaccine is a dormitory-style refrigerator. This type of refrigerator has access to the refrigerator and freezer at the same time without a separate door, which forms one small unit. The cooling coil is normally located in the freezer section of this type of refrigerator, and the temperature sensor is not sensitive enough to fulfill its intended function. In particular, once it decreases the temperature, many times the temperature of the refrigerator falls below 0℃. Even if the vaccine is stored for a short period of time, it is necessary to use a refrigerator that is appropriate for storing vaccine to prevent the vaccine from being exposed to a temperature that is either too high or too low, thus, preventing the spoiling of the vaccine [6].

After the primary inspection, it was found that only 7.7% of the medical institutions were storing only vaccines in the refrigerator. It is unacceptable to store other medical products or food in a place where vaccines are stored, because other drugs of foods can pollute vaccines. In addition, there would be more frequent opening and closing of the door, which could make it difficult to maintain the appropriate temperature for the vaccines [6]. All the better, maintaining the appropriate temperature is very important, and thus regular checking of the temperature is very important. However, the principle of keeping the vaccine in the center of a refrigerator, the location where the temperature is rather consistent, was adhered to by less than half of all the institutions inspected (43.6%). Refrigerators without a thermometer were found in as many as 12.8% of the institutions, and 69.2% of institutions recorded the storage temperature regularly. In addition, only 15.4% had an available emergency power supply in case of a blackout. From these facts, it was found that many medical institutions were exposing vaccine to unstable temperatures and that caution about this was lacking [12]. Not only can this decrease the titer of vaccine, but it can also lead to discarding vaccine and re-vaccination due to failure of antibody formation, which would lead to decrease in the credibility of the vaccine efficacy to the patients.

The average score from the checklist was 64.9%, which was a rather low in percentage, so we discovered that the overall actual condition needs to be ameliorated. Although there was not a significant difference according to the type of institution, pediatric clinics had a slightly higher score than the other institutions. Because pediatric clinics provide many patients with vaccinations, there was a high possibility that the storage conditions would have been good [1].

The average score from the awareness questionnaire was 74.9%, and the figure was relatively higher than the storage condition average score. However, based on the point of view that one needs to have a correct understanding for every item in order to handle vaccines properly, this awareness score is not satisfactory. Yuan et al. [5], also considered answering all the questions correctly as very important. Pediatric clinics had the lowest awareness score, which was contradictory to the actual conditions of investigation, but there was no significant difference among types of institutions.

When we compared the checklist score and questionnaire score before and after education, we found that the actual conditions of the overall vaccine storage and awareness had improved. All of the 3 types of institutions improved in both actual conditions and awareness; the pediatric clinics and other miscellaneous clinics showed significant differences.

Most of the checklist items were possible to improve mostly through awareness improvement and individual effort, except for whether or not the institution was equipped with refrigerators exclusive for vaccines, and whether or not an emergency power supply was present. After education was provided, we saw that the actual conditions significantly improved for items such as completion of recording vaccine storage temperature, storing the vaccine in the center of the refrigerator, and storing vaccines separately from other drugs or food. Although actual conditions related to vaccine storage such as a change in behavior of the vaccine storage manager could be further improved with a greater effort, the items that were not possible to improve could have been due to the limitations of one-off education. Repeated education for the improvement of behavior needs to be enforced. In addition, if a periodic reminder such as a written document or an electronic document was sent to an institution, along with enforcement of surveillance and feedback, problems related to vaccine storage or management could be largely solved. These items need to be supported with policies, because their fulfillment would significantly decrease without the support of policies.

In awareness questionnaire, compared to before education, after education several items significantly improved, including awareness of filling out the vaccine storage temperature records, storing vaccines separately from other drugs or food, storing vaccines according to type and expiration date, storage in light-proof conditions, installation of a thermometer in the center of the refrigerator, not storing vaccines on the refrigerator door shelf, the necessity for having an independent generator available in preparation for a blackout, and awareness of the necessity to educate the person in charge of vaccine management. Out of these items in the questionnaire, awareness about storing vaccines according to type and expiration date, and operation of an emergency power supply had significantly improved, but improvement in actual practice was meager. When the subjects were aware of those items, there were a few cases where conditions would have allowed those items to be implemented and practiced, but the reality was that in most of the institutions, resources were insufficient for purchasing a refrigerator exclusively for vaccines or possessing an independent generator. Refrigerators designated specifically for vaccine storage cost 9 million won (about $US 9000) per 780 liters, and realistically speaking, private medical institutions whose quantity of vaccinations is less than 10 cases per week cannot afford vaccine refrigerators. Therefore, policy should not stop at implementing regular inspections and surveillance, but it is imperative that it also include a governmental system that provides resources for private medical institutions to purchase refrigerators for vaccine storage. Most of the problems can be improved by individual effort, but for those problems that cannot be solved too easily, as mentioned earlier, we may assume that it is due to the limits of a one-off education of the person in charge of vaccine management because it may not actually modify behavior itself. Therefore, the results of this research enabled us to confirm that repetitive education, centralized management, and surveillance are important.

Self-produced educational leaflets provided at the same time as one-on-one education included only the basic knowledge required of people in charge of vaccine management. In Korea, the situation is dire enough that we need to produce a standardized manual related to vaccine storage and management. In the U.S., the Centers for Disease Control and Prevention publicly announced their newest standardized guidelines for vaccine storage and handling on their website for users to access easily. On the other hand, in Korea it is difficult to find a standardized guidelines written in Korean except for a couple of sections from the Handbook for safe use of vaccine [16].

There are some limitations to this study. First, the private medical institutions could have had a bias, because we called them on the phone prior to the visit asking for their permission, and although they might not have known the detailed items related to the inspection, they knew we would visit them on the matter of vaccination. Second, we only targeted private medical institutions that were controlled by 4 public health centers in a particular region, which means that the results may not be generalizable to the other regions of Korea. Third, although public health centers administer a large number of vaccinations, we performed the study in collaboration with people in charge of the public health center; thus, we did not include them as study targets, but it is necessary to broaden the range and the subjects in future studies. Fourth, measuring temperatures inside the refrigerators was not done repeatedly; it was only performed once.

Despite these limitations, this study demonstrated that proper awareness and actual practices related to vaccine storage and management are rare in Korea. From a narrow perspective, the goal of this study was to improve the awareness and practices of health care providers in some regions of Gyeongsangbuk-province. By extension, this study is meaningful in the sense that it proposes establishing policies to enforce repeated education, developing standardized manual, and performing periodic management inspection by communicating the results to the people in the public health policy decision-making process throughout the country. In conclusion, vaccination is the most effective and the strongest method ever developed for humans to prevent and eliminate contagious diseases, and a system needs to be established for proper vaccine storage and quality control to increase its efficacy.