Programmatic assessment of electronic Vaccine Intelligence Network (eVIN).

eVIN is a technology system that digitizes vaccine stocks through a smartphone application and builds the capacity of program managers and cold chain handlers to integrate technology in their regular work. To effectively manage the vaccine logistics, in 2015, this technology was rolled-out in 12 states of India. This study assessed the programmatic usefulness of eVIN implementation in the areas of vaccine utilization, vaccine stock and distribution management and documentation across selected cold chain points. A pre-post study design was used, where cold chain points (CCPs) were selected using two-stage sampling technique in eVIN states. Pre-post comparative analysis was carried out on the identified indicators using both primary and secondary data sources. The vaccine utilization data reflects that the utilization had reduced from 305.3 million doses in pre-eVIN period to 215.0 million doses in post-eVIN period across 12 eVIN states, resulting into savings of approximately 90 million doses of vaccines. Number of facilities having stock-out of any vaccine showed a significant reduction by 30.4% in post-eVIN period (p<0.001). There was a 4.0% drop in facilities reporting minimum stock of any vaccine after implementation of eVIN. Facilities with maximum stock of any vaccine had increased from 37.4% in pre-eVIN to 39.2% in post-eVIN. During the pre-eVIN period, only 38.6% facilities updated vaccine stock on a daily basis, while in post-eVIN period, 53.5% facilities updated vaccine stock on daily basis. The completeness of records in the vaccine stock registers, indent form and temperature logbook have been substantially improved in the post-eVIN period (p<0.001). eVIN had helped in streamlining the vaccine flow network and ensured equity through better vaccine management practices. It is a powerful contribution to strengthen the vaccine supply chain and management. Upscaling eVIN in the remaining states of India will be crucial in improving the efficacy of vaccines and cold chain management.

Introduction

The Immunization Supply Chain and Logistics (ISCL) system, the backbone of any immunization program, plays a key role in improving the immunization coverage with quality and equity through timely supply of safe and potent vaccines [1]. The Universal Immunization Programme (UIP) of India, one of the largest public health programmes in the world, caters to ~26 million new born and ~30 million pregnant women via 9 million sessions every year through 27,000 cold chain points [2]. The Immunization Supply Chain and Logistics (ISCL) system in the country has played a significant role in achieving the current coverage levels while dealing with the challenges of vaccine storage, distribution and management [3].

In India, several challenges related to the management of ISCL present themselves [4]. One among them is the maintance of records. Initially, for vaccine records and management, mainly four types of paper formats were used–comprehensive log book for every Cold Chain Equipment (CCE) to record temperature as well as details of repair and maintenance; vaccine stock register–issue and receipt; vaccine distribution register for immunization sessions, and vaccine and logistics indent form for recording temperature, maintenance and repair details of CCEs, for the vaccine stock, daily issue and logistics indent [5]. These formats require manual data entry, compilation and consolidation, which lead to a delay in real-time visibility of stock levels and temperatures. This leads to weak inventory and stock-flow record keeping practices and subsequently delays vaccine stock visibility [6]. Moreover, the temperature monitoring of CCE was largely dependent on the availability of a dedicated human resource at the cold chain point (CCP) [7]. This type of system maintenance poses serious challenges to the quality of recording, reporting and monitoring the temperature of CCEs.

In order to address the issues and strengthen the vaccine supply chain, in 2015, eVIN was introduced by the Ministry of Health and Family Welfare (MoHFW), Government of India to digitize the details of stock and storage temperature and enable real time visibility of vaccine inventories [8].

eVIN is a stock management technology that digitizes vaccine stocks through a smartphone application and builds the capacity of program managers and cold chain handlers to integrate the technology in their regular work. eVIN technology uses a smartphone, a web-based application, temperature loggers and cloud-based server to digitize vaccine stock inventory and storage temperature from every vaccine store and CCP located at peripheral government health facilities. It allows remote temperature monitoring for all CCE with automated alert mechanisms to alert designated staff in case if the temperature of equipment differs from the predefined range [9]. It’s an integrated package of people, process and product [10]. For successful implementation of the eVIN system, the capacity development of cold chain handlers (people) is also ensured [11].

To evaluate the usefulness of eVIN implementation and to visualize its benefits and challenges, a programmatic assessment was undertaken. Although several immunization supply chain studies including the “National Effective Vaccine Management (EVM) Assessment” by the National Cold Chain Vaccine Management Resource Centre has been conducted on vaccine and cold chain management, no assessment has been conducted on the implementation of eVIN and its programmatic outcomes. This piece of work is part of a larger study on the ‘Techno-economic Assessment of eVIN’, conducted to disseminate learnings for a national scale up of the eVIN [12].

Method

The assessment was conducted in 12 states (Assam, Chhattisgarh, Gujarat, Jharkhand, Manipur, Nagaland, Odisha, Bihar Himachal Pradesh, Madhya Pradesh, Rajasthan and Uttar Pradesh) where eVIN was launched initially. A pre-post comparison was used on key performance indicators for programmatic assessment. As obtaining data for one-year prior to implementation of eVIN was challenging, therefore, six months period was chosen as the reference period for pre-eVIN phase. The pre-eVIN reference period was of six months for all CCPs, however it varied for different states and districts due to different timeframe of eVIN rollout. The specific duration of pre-eVIN phase are as mentioned: most of the CCPs in a district were covered either in between the period of April 2015 to September 2015 (13 districts), April 2016 to September 2016 (12 districts), October 2015-March 2016 (11 districts) and October 2011- March 2012 (1 district). The post eVIN reference period was from October 2017 to March 2018 for the assessment.

Minimum number of CCPs required for the study was calculated to be 502 considering 43% of PHCs reported instances of stock-out [13], 10% non-response rate and 1.2 design effect [14]. It was further increased to 617 CCPs in order to draw valid conclusions at the state levels.

Selection of CCPs in the eVIN states were done using two-stage sampling design. In the first stage, districts were selected followed by selection of CCPs in the second stage. In each eVIN state, number of sampled districts was decided based on Probability Proportion to Size of CCPs. In total 37 districts were selected using systematic random sampling technique after arranging the districts in ascending order based on the proportionate share of cold chain point in the total cold chain point in state. Further, CCPs were randomly selected in each of the selected district. A detailed methodology is available in the larger study document [12].

Quantitative data was obtained using structured questionnaire from CCPs pertaining to stock management, temperature monitoring, cold chain equipment, and documentation aspects of vaccine supply chain. The primary data for pre-eVIN phase was done from stock registers, vaccine distribution registers, temperature log books and other important registers. Completeness and accuracy were analyzed in the assessment. Completeness was seen of Indent form [15], vaccine stock register [16], and temperature log book. Accuracy was assessed through stock register and eVIN record, eVIN record and physical count. Computer Assisted Personal Interviewing (CAPI) technique was employed using tablets/mobiles for real-time data collection and data entry. Further, data quality assessment was done using survey CTO [17] and MS Excel. Data analysis was conducted in MS Excel and STATA 13 software.

A set of selected indicators fulfilling the objective are used in this paper for which pre-post data was collected to represent the programmatic assessment of eVIN (Table 1).

Table 1

List of programmatic assessment indicators, data sources and definition used.

Indicators	Data sources	Definition
Vaccine utilization	Secondary	Vaccine utilization focuses on the utilization of doses and pipeline stock savings of vaccine doses in pre- and post-period of eVIN implementation at the national level. Vaccine utilization in doses was computed as: Vaccine Utilization in million doses = (Opening doses + Doses supplied from GMSD/ supplier)-Closing balance
Stock Management	Primary & Secondary	This includes details on the events of stock-out, minimum and maximum stock or extent of excess stocks and missed opportunity [18] post eVIN.
Vaccine distribution	Primary	The effectiveness was assessed through lesser replenishment time, complete order fill rate [19], minimal expiry of vaccines at stores [20] and CCP to CCP sharing of vaccines.
Vaccine management practices	Primary	It included record keeping practices and vaccine stock updating duration
Documentation	Primary	Assessed through checking completeness and accuracy of records.Completeness: Data completeness was categorized as a) more than 90%, b) between 80 to 90%, and c) below 80%. Above 90% completeness indicated less than or equal to 10 instances missed for critical indicators; 80 to 90% indicated less than or equal to 20 instances missed and below 80% indicated more than 20 instances missed.Accuracy: It was done by tallying vaccine-wise stock kept in Ice Lined Refrigerator (ILR) and information recorded in stock register and eVIN record. Two different comparisons were carried out: One between stock register and eVIN record and another between eVIN record and physical count.

Results

The study found that the utilization of vaccines reduced from 305.3 million doses in pre-eVIN period to 215.0 million doses in post-eVIN period, demonstrating 29.6% reduction in utilization of doses. The facilities witnessing stock-out of any antigen had also reduced from 37.8% in pre-eVIN period to 26.3% in post-eVIN period. Number of facilities having stock-out of any vaccine showed a significant reduction of 30.4% in post-eVIN period (p<0.001). In the areas of minimum and maximum stock management situations, it was observed that the instances of minimum stock of any antigen was occurred at 49.8% facilities in the pre-eVIN period, compared with 47.8% facilities in the post-eVIN period. The maximum stock of any antigen had increased from 37.4% in pre-eVIN to 39.2% in post-eVIN period, indicated 4.8% increase in facilities witnessing events of maximum stock. The analysis of missed opportunities due to reduction in stock-outs revealed that after implementation of eVIN, fewer beneficiaries were getting omitted due to stock-out as compared in the pre-eVIN period. The highest reduction was observed in DPT (reduced by 70.0%) (p<0.001), and lowest in BCG (reduced by 6.4%) and almost no change in Hep-B (reduced by 0.2%) after eVIN implementation. Furthermore, mean number of days (of vaccine expiry) had reduced from 428 in pre-eVIN to 384 in the post-eVIN period at CCP level signifying that the ‘First Expiry First Out’ (FEFO) was being practiced. Overall improvement in utilizing the Government of India registers and reduction in usage of loose papers across facilities after the introduction of eVIN was statistically significant (p<0.001). For stock management, the updating of vaccine stock register daily had improved from 38.6% in pre-eVIN to 53.5% in the post-eVIN period, reflecting an overall improvement in vaccine management practices. The completeness of data in the vaccine stock registers significantly improved from 29.0% in pre-eVIN to 75.0% in post-eVIN in all eVIN states (p<0.001) (Table 2).

Table 2

Summary of results of indicators evaluated for programmatic assessment of eVIN.

Indicators	Sub-indicators	Assessment Indicators	Pre-eVIN	Post-eVIN	% Reduction
Vaccine Utilization		Utilization of doses (in Million)	305.3	215.0	29.6
Vaccine Stock Management	Stock out	Facilities experienced stock out of vaccines (in %)	37.8	26.3	30.4
	Minimum Stock	Facilities observed minimum stock of any vaccines (in %)	49.8	47.8	4.0
	Maximum Stock	Facilities observed maximum stock of any vaccines (in %)	37.4	39.2	-4.8
	Missed Opportunity	Immunization sessions missed due to stock-out and resulting into missed opportunities (in numbers)
		BCG	2,096	1,961	6.4
		HEP-B	1,837	1,833	0.2
		OPV	13,954	8,159	41.5
		DPT	7,382	5,902	20.0
		Penta	7,093	2,109	70.3
		Measles	5,969	3,467	41.9
		TT	3,729	2,593	30.5
	Vaccine Stock Updating Duration	Facilities updated vaccine stock daily (in %)	38.6	53.5	27.9a
Vaccine Distribution	Expiry days of vaccines	Mean expiry days	428	384	10.3
Vaccine Management	Proper record keeping practices	GoI registers (in %)	56.2	97.4	42.3a
		Loose papers (in %)	11.2	0.7	93.8
Documentation	Vaccine stock register	CCPs with completeness of vaccine stock registers (in %)	29.0	75.0	61.3a

is % increase

Vaccine utilization

A significant reduction in the utilization of vaccines was observed after the introduction of eVIN. The maximum saving of doses was seen for Hep-B with a decrease by 77.5%. The reason for the reduction in utilization may be attributed to the limited usage of Hep-B at institutions following the introduction of Pentavalent vaccine. In addition, 34.7% of doses of OPV, 33.2% doses of TT, 22.9% doses of Pentavalent and 21.1% doses of measles vaccines were saved during the post-eVIN period (Table 3).

Table 3

Vaccine-wise saving of doses.

Antigen	Utilization of doses (in Million)		Saving (%)
	Pre-eVIN	Post-eVIN
BCG	45.4	37.5	17.4
DPT	29.8	27.3	8.4
Hep-B	28.9	6.5	77.5
Measles	44.0	34.7	21.1
OPV	47.6	31.1	34.7
Pentavalent	45.5	35.1	22.9
TT	64.1	42.8	33.2
Total	305.3	215.0	29.6

State-wise utilization

The state-wise utilization of vaccines leading to realistic demand reflection of vaccines at the national level. Assam, Chhattisgarh, Himachal Pradesh and Uttar Pradesh have shown more than 30.0% saving of utilized doses. Other states reported marginal saving of doses. However, a marginal decrease of 4.4% and 4.3% has been observed in Madhya Pradesh and Rajasthan respectively (Fig 1).

Fig 1

State-wise reduction in vaccine utilizationa.

a Data of Jharkhand, Manipur, Nagaland and Himachal Pradesh are not presented as the number of CCPs from these states are less than 30.

Vaccine stock management

Stock-out of vaccines

Vaccine-wise analysis for stock-outs showed that all vaccines observed a remarkable reduction in stock-out at facilities across eVIN states (Table 4). Secondary data from UNDP was analyzed for Bihar and Manipur due to under-reporting of data during the visits. Facilities reported stock-out had substantially reduced for all except DPT vaccine (59.8% for OPV, 56.9% for TT, 56.1% for Hep-B and 47.7% for Measles vaccine).

Table 4

Facilities reporting stock-out of antigens.

Antigens	% of facilities
	Pre	Post	% Reduction
BCG	10.0	7.3	27.0
DPT	14.4	13.3	7.6
HEP-B	10.7	4.7	56.1
Measles	14.9	7.8	47.7
OPV	16.9	6.8	59.8
PENTA	8.3	4.9	41.0
TT	6.5	2.8	56.9

State-wise stock-out

Number of facilities having stock-out of any vaccine showed a significant reduction by 30.4% in post-eVIN period (p<0.001). Remarkably not a single instance of stock-out occurred in Chhattisgarh. Out of eight states, five states including Bihar, Chhattisgarh, Gujarat, Rajasthan, and Uttar Pradesh have showed more reduction in stock-out of any vaccine at facility than the overall reduction of 30.4% (Fig 2).

Fig 2

State-wise reduction in vaccine stock-outa.

a Data of Jharkhand, Manipur, Nagaland and Himachal Pradesh are not presented as the number of CCPs from these states are less than 30.

Minimum stock of vaccines

The facilities reporting minimum stock of vaccine was more in the case of DPT and Pentavalent compared to any other vaccines. The facilities showing minimum stock of DPT had increased from 32.9% in pre-eVIN to 34.5% in post-eVIN phase (Table 5).

Table 5

Facilities reporting Minimum Stock in Pre- eVIN and Post- eVIN period by antigen.

Antigens	% of facilities
	Pre	Post	% Reduction
BCG	33.4	29.8	10.8
DPT	32.9	34.5	-4.9
HEP-B	19.9	15.9	20.1
Measles	31.9	27.2	14.7
OPV	36.3	31.4	13.5
PENTA	24.6	26.4	-7.3
TT	29.8	17.3	41.9

State-wise minimum stock

The facilities observing minimum stocks of any vaccine declined in Assam, Bihar, Chhattisgarh, Madhya Pradesh, Rajasthan, and Uttar Pradesh. However, more facilities observed minimum stock in Odisha. There was no change in minimum stock status in Gujarat. Though there was a 4.0% drop in facilities reporting minimum stock of any vaccine after implementation of eVIN (Fig 3).

Fig 3

State-wise reduction in minimum stock-out of vaccinesa.

a Data of Jharkhand, Manipur, Nagaland and Himachal Pradesh are not presented as the number of CCPs from these states are less than 30.

Maximum stock of vaccines

The facilities observing maximum stock had increased for TT, Pentavalent, OPV, Measles and BCG, while a decrease is noted for Hep-B and DPT despite the implementation of eVIN (Table 6).

Table 6

Facilities reporting excess stock in Pre- eVIN and Post- eVIN period by antigen.

Antigens	% of facilities
	Pre	Post	% Reduction
BCG	25.1	25.8	-2.8
DPT	28.2	21.6	23.4
HEP-B	23.7	17.2	27.4
Measles	21.4	23	-7.5
OPV	26.7	30.3	-13.5
PENTA	21.4	27.2	-27.1
TT	20.1	27.7	-37.8

State-wise maximum stock

Pre- and post-eVIN comparison showed an overall 4.8% increase in maximum stock at facilities. This may be attributed to the increased visibility across all levels of supply chain post eVIN implementation. No change is seen in the facilities observing maximum stock of any vaccine especially in Assam, and Chhattisgarh. Most of the facilities in Bihar, Gujarat, Odisha, Rajasthan and Uttar Pradesh showed an increased maximum stock situation in post-eVIN phase.

Missed opportunity of vaccines

The subsequent number of missed opportunities for all antigens had reduced after post-eVIN implementation, however, the reduction in missed opportunity due to BCG and Hep-B was found to be marginal. The highest reduction of 70.3% in events of missed opportunities was observed for Pentavalent (p<0.001) (Table 2).

Vaccine distribution management

Replenishment time

An attempt was made to assess the impact of eVIN on mean replenishment time and change in replenishment time between supply and indenting of vaccines across cold chain facilities. Mean replenishment time between supply and indent had reduced by 57% across the facilities in 12 states. Overall reduction in replenishment time across facilities was statistically significant (p<0.001). The time between indent and supply of vaccines had decreased for all the states except Jharkhand.

Order fill rate

It is observed that overall, there was a marginal increase of 2% in order fill rate from pre- to post-eVIN phase.

Mean expiry days

Number of days left in expiry of the vaccine at the level of cold chain points indicated an overall improvement of 10.3% in management of vaccines. Among the 12 states, Uttar Pradesh showed the highest improvement in managing expiry days in post-eVIN intervention (Fig 4). Odisha demonstrated a gross reduction in the mean expiry days left for the vaccines. Across all levels of supply chain, adherence to FEFO was observed.

Fig 4

State wise reduction in mean expiry days left for vaccinesa.

a Data of Jharkhand, Manipur, Nagaland and Himachal Pradesh are not presented as the number of CCPs from these states are less than 30.

Vaccine management

Record keeping practices

The utilization of the GoI register across the facilities was improved by 42.3% after eVIN implementation. In Chhattisgarh, this utilization was 100.0%. The utilization of Government of India register was encouraging across the facilities in the 12 eVIN states. The usage of loose papers across facilities after the introduction of eVIN was reduced by 93.8%. The practice of using loose papers had stopped in Chhattisgarh, Gujrat, Madhya Pradesh, and Odisha. Usage of loose paper in Bihar was not observed even in pre-eVIN phase. The overall improvement in utilizing the GoI registers and reduction in usage of loose papers across facilities after the introduction of eVIN was statistically significant (p<0.001).

Vaccine stock updating duration

During pre-eVIN period, only 38.6% facilities updated vaccine stock on a daily basis, while, during post-eVIN period, 53.5% facilities updated vaccine stock on daily basis. The facilities reported weekly (including daily) updating of stock registers had gone up from 72.5% to 81.1% in the post-eVIN phase.

Documentation

Completeness

The completeness in indent forms had increased from 26.0% in pre-eVIN to 69.0% in post-eVIN (p<0.001). The highest improvement was evident in Nagaland (from 0.0% to 100.0%) followed by Chhattisgarh (from 0.0% to 96.8%) and Uttar Pradesh (from 48.0% to 95.0%). In the case of vaccine stock registers, data completeness significantly improved from 29.0% in pre-eVIN to 75.0% in post-eVIN in all eVIN states (p<0.001). Completeness of temperature logbook was assessed based on whether temperature was plotted in the morning and evening as mandated, whether defrosting and preventive maintenance records are maintained as mentioned. The completeness of the temperature logbook significantly improved from 28.5% in pre-eVIN to 69.8% in post-eVIN across e-VIN states (p<0.001).

Accuracy

Overall, 93.6% match in all eVIN states being observed while comparing the vaccine-wise total stock recorded in stock register with vaccine-wise total stock recorded in eVIN record. Physical stock count of each vaccine kept in ILR and DF at CCPs was done by the field investigator and the findings corroborated with the eVIN record. Overall, 92.0% facilities have shown match of count with eVIN update across all eVIN states.

Discussion

The vaccine utilization data of Immunization Division, MoHFW reflected savings of approximately 90 million doses of vaccines after eVIN implementation. These savings could be attributed to the roll-out of eVIN across and other initiatives such as the introduction of open vial policy, effective vaccine management assessments, and continuous follow-up with an improvement plan in place. Overall, facilities observing stock-out of any vaccine have significantly reduced after eVIN introduction. But the instances of minimum and maximum stock remained unimproved. Even after reduction in stock-out, 26.3% facilities still observed stock-out in the post-eVIN phase. The variation in mechanism of vaccine distribution may also be affecting stock-outs. The vaccines are being pushed or pulled irregularly on weekly or monthly basis and often depending on the need of the facilities. The analysis of missed opportunities due to reduction in stock-outs revealed that after implementation of eVIN, fewer beneficiaries were getting omitted due to stock-out as compared in the pre-eVIN period. The highest reduction was observed in DPT (reduced by 70.3%), and lowest in BCG (reduced by 6.4%) and almost no change in Hep-B (reduced by 0.2%) after eVIN implementation. Data pertaining to adherence of FEFO (First expiry First out) practices at the level of cold chain facilities and at the level of district vaccine stores was captured in the assessment. The mean number of days (of vaccine expiry) has reduced from 428 in pre-eVIN to 384 in the post-eVIN, at CCP level signifying that the ‘First Expiry First Out’ (FEFO) is being practiced. The completeness of record keeping was checked against applicable fields of the indent form, register and temperature logbook. Significant improvement in the completeness of records was observed in post-eVIN across eVIN states.

Conclusion

eVIN has set up a strong example of how technology can be leveraged to enhance efficiency and effectiveness of the public health measures. eVIN system is playing a pivotal role in effective and efficient management of vaccine supply, supervision and monitoring. The findings of this assessment suggest positive changes in the areas of vaccine utilization, stock management and distribution and documentation. States are benefitting with the implementation of eVIN and have been able to improve planning, management of stocks and distribution of vaccines to the last mile.

Limitations of the study

Limiting the observation period of six months for assessment due to possibility of non-availability of registers in pre-eVIN period beyond this time frame was a challenge. Also, as the eVIN was rolled out in a phase-wise manner across the states so time period varied among the states for pre-eVIN assessment. Although no foreseen biases were observed but this may cause concern in exact comparison of pre-post period estimates for states.

29 Jun 2020

PONE-D-20-02529

Programmatic Assessment of electronic Vaccine Intelligence Network (eVIN)

PLOS ONE

Dear Dr. Dhalaria,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Aug 13 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Khin Thet Wai, MBBS, MPH, MA (Population & Family Planning Resear

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. If you developed and/or translated a questionnaire as part of this study and it is not under a copyright license more restrictive than Creative Commons Attribution (CC-BY), please include a copy, in both the original language and English, as Supporting Information.

3.Thank you for stating the following financial disclosure:

'The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.'

At this time, please address the following queries:

Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.

State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

If any authors received a salary from any of your funders, please state which authors and which funders.

If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere. Please clarify whether this conference proceeding or publication was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript.

5. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

Additional Editor Comments:

This is an essential study in support of vaccine supply and logistic management system.

(1) To clarify whether the.larger study has been published or not and if yes please add the citation.

(2) To add limitations of the study in the discussion section.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript is well written on a topic of National relevance. I have a few queries as under:

1. Methods: Please mention the pre-eVIN period. Is it the same or different for different CCPs?

2. There are 9 EVM criteria indicators. Why has only 4 criteria been selected for assessment in this study?

3. How was the design effect of 1.2 taken? How was the intra-class correlation calculated?

4. Please elaborate the sampling strategy

5. How is the data in pre-eVIN phase collected?

6. Data collection method is unclear, as in methods section, mention of Computer Assisted Personal Interviewing (CAPI) has been mentioned whereas in Table 1, under documentation, mention has been made of 'checking completeness of records'

7. The pre-eVIN period of 6 months has been compared with just 3 months of post-eVIN, hence the data of both periods cannot be compared. The reason for this is not understood.

8. Results: Is the data of pre-eVIN and post-eVIN period averaged for a month? As the post-eVIN period is smaller,

and hence 215.0 million doses were utilized as compared to 305.3 million doses utilised in a longer pre-eVIN period. How can the change be attributed to eVIN?

9. Instead of vaccine utilization, vaccine wastage could have been calculated.

10. If there is clustering of births in some months, vaccine utilization may be different, can it be compared?

11. There are other factors which affect vaccine utilization or vaccine stock outs, like difficulty in delivery of vaccine to the CCPs because of difficult road conditions, vaccine drop-outs particularly for Pentavalent vaccine, OPV, etc in the pre-eVIN and post-eVIN period, etc.

12. The labelling of graphs has to be done properly

You are requested to re-look at the above comments and amend your manuscript.

Reviewer #2: Please explain the following points to make the manuscript more informative

The manuscript shall be useful for scaling up the services to the whole of India.

1. Sample size calculation was based on stock out rates. What was the reduction expected (effect size) Design effect 1.2 was considered. Please provide reference for the same.

2. Comparison was made pre eVIN and post eVIN. Six moths reference was taken Pre eVIN and three months post eVIN was compared. Comparison should have been done for similar period/ same months.

3. Major gain was reduction of Utilization Doses. (29.6% reduction of utilization doses). How many beneficiaries received immunization n pre eVIN and Ppost eVIN period.

4. How was states were selected. Expect Himanchal and Gujarat all other states started from weaker health system had poor indicators at the begining.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr.Paramita Sengupta

Reviewer #2: Yes: Binod Kumar Patro

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Sep 2020

Additional Editor Comments:

This is an essential study in support of vaccine supply and logistics management system.

(1) To clarify whether the larger study has been published or not and if yes please add the citation.

Yes, the larger study has been published as a report, but not as a peer reviewed document. The citation of the published report has been added in the manuscript as reference number 12.

(2) To add limitations of the study in the discussion section.

Limitation of the study is included after conclusion section of the manuscript.

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

________________________________________

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

________________________________________

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript is well written on a topic of National relevance. I have a few queries as under:

1. Methods: Please mention the pre-eVIN period. Is it the same or different for different CCPs?

For Pre eVIN, assessment period was fixed as six months’ time frame before the month of roll-out of eVIN. However, due to roll-out of eVIN in a phase wise manner, the exact six months duration had been different for states and districts. The detail on time of roll-out have been included in the manuscript.

2. There are 9 EVM criteria indicators. Why has only 4 criteria been selected for assessment in this study?

In this study, 5 EVM criterion have been used which are on vaccine utilization, stock management, vaccine distribution, vaccine management practices and documentation. The other indicators on eVIN are human resource and training, vaccine wastage, temperature monitoring and cold chain equipment. The larger study has captured all these domains but due to a different study design or source of information, these indicators have not been mentioned in this manuscript.

3. How was the design effect of 1.2 taken? How was the intra-class correlation calculated?

Design effect of 1.2 has been recommended for facility surveys in developing countries. Reference document is added in the manuscript. The design effects for most of the facility survey estimates of interest is considered to be very low because (1) the list of samples is not clustered at all and (2) both the cluster sizes (that is, number of sample facilities) and the intra cluster correlations in the area sample is small. Therefore, for purposes of calculating sample sizes that the value is about 1.2 at the maximum.

4. Please elaborate the sampling strategy

All eVIN states were selected for pre- and post-design. For the selection of CCPs, two-stage sampling technique was deployed; the first stage for selecting districts, followed by selection of CCPs in the second stage.

Stage 1: Selection of districts: Within each eVIN state, districts were selected based on Probability Proportional to Size Sampling (PPS). The total number of selected districts in 12 eVIN states was 37. Based on relative proportion of each state, the number of districts to be sampled was: Assam (3), Chhattisgarh (3), Gujarat (3), Jharkhand (2), Manipur (1), Nagaland (1), Odisha (3), Bihar (4), Himachal Pradesh (1), Madhya Pradesh (5), Rajasthan (3), and Uttar Pradesh (8).

The districts were arranged in ascending order based on the proportionate share of cold chain points out of the total cold chain point in the state. An interval (N/n) factor was calculated by dividing the number of total districts (N) in the region by the number of districts (n) to be selected. After selecting the first district randomly, every (N/n)th district was selected until the required number for districts was obtained.

Stage 2: Selection of Cold Chain Points: Cold chain points were randomly selected in each of the selected district.

It has been added in the manuscript under “Methods”

5. How is the data in pre-eVIN phase collected?

Data collection for programmatic assessment was carried out between May and July 2018. Quantitative data was collected from the stock registers, vaccine distribution registers, temperature log books and other important registers at CCPs. Completeness and accuracy were assessed through stock registers and eVIN record, eVIN record and physical count. CAPI or “Computer Assisted Personal Interviewing” technique was employed using tablets/mobiles for online data entry. This ensured the quality of data collection and elimination of time involved in data entry.

This also has been added in the manuscript under “Method”.

6. Data collection method is unclear, as in methods section, mention of Computer Assisted Personal Interviewing (CAPI) has been mentioned whereas in Table 1, under documentation, mention has been made of 'checking completeness of records'

The questionnaire used in the study is now available as supplementary information with the manuscript. All the quantitative information was collected in CAPI.

To check whether the documentation practices, has improved with the roll-out of eVIN, records were observed to see their completeness and accuracy. The information on completeness and accuracy was quantified into three categories (available in the questionnaire) and the observation was marked in the CAPI in respective question’s reply.

7. The pre-eVIN period of 6 months has been compared with just 3 months of post-eVIN, hence the data of both periods cannot be compared. The reason for this is not understood.

We regret for the error in mentioning the post-eVIN period. The period for post-eVIN was from October 2017 to March 2018 (6 months) for the assessment. Correction has been done in the manuscript.

8. Results: Is the data of pre-eVIN and post-eVIN period averaged for a month? As the post-eVIN period is smaller, and hence 215.0 million doses were utilized as compared to 305.3 million doses utilised in a longer pre-eVIN period. How can the change be attributed to eVIN?

For both the Pre-eVIN and Post-eVIN period, data was collected for 6 months duration separately for each of the phase. There was a typing mistake while mentioning the post eVIN period, we apologies for the same. Since both the periods are of equal length (6 months) therefore the data is comparable.

There has been huge reduction in vaccine utilization from 305.3 million doses in pre-eVIN period to 215.0 million doses in post-eVIN period across 12 eVIN states, resulting into savings of approximately 90 million doses of vaccines therefore the change can certainly be attributed to eVIN in absence of any other intervention in vaccine supply chain arena.

9. Instead of vaccine utilization, vaccine wastage could have been calculated.

We agree that the direct presentation of vaccine wastage would have given a clearer picture but unfortunately the vaccine wastage record is highly under-recorded in pre eVIN phase. In the larger study report, the vaccine wastage has been computed using UNDP’s data but not presented in this manuscript due to compatibility issues with the study design.

Vaccine utilization is a broader variable which includes both usage and wastage so vaccine wastage is indirectly taken care of under vaccine utilization.

10. If there is clustering of births in some months, vaccine utilization may be different, can it be compared?

The Health Management Information System (HMIS) data of India do not suggest clustering of births in specific months at the state level in these 12 states. The vaccine utilization might get affected at cold chain point level, but it gets nullified at the higher levels like district, state or national level. Therefore the six months observation period each for pre and post eVIN phase can be compared.

11. There are other factors which affect vaccine utilization or vaccine stock outs, like difficulty in delivery of vaccine to the CCPs because of difficult road conditions, vaccine drop-outs particularly for Pentavalent vaccine, OPV, etc in the pre-eVIN and post-eVIN period, etc.

Agree, that the difficult road condition has the potential to affect the vaccine stock-out, but its contribution in total stock-out is very small. Plus the poor road condition is a factor which might be existing in pre as well as and post both time frames.

The vaccine coverage is directly related with vaccine utilization. While planning this study, a matching was done to understand the antigen wise number of beneficiaries in pre and post phase. The analysis suggested that the coverage has gone up in post eVIN phase compared with pre eVIN phase.

12. The labelling of graphs has to be done properly

There was overlapping in the values and labels which has been corrected and incorporated in the manuscript.

You are requested to re-look at the above comments and amend your manuscript.

Reviewer #2: Please explain the following points to make the manuscript more informative

The manuscript shall be useful for scaling up the services to the whole of India.

1. Sample size calculation was based on stock out rates. What was the reduction expected (effect size) Design effect 1.2 was considered. Please provide reference for the same.

This assessment gives an idea of supply chain and logistics management system with and without eVIN by analyzing the specific indicators in pre and post eVIN phase. This required a given number of cold chain facilities to be representative of pre and post eVIN phase separately. Keeping the study objectives in mind sample size was calculated using a prevalence estimate rather than effect size.

Reference for design effect is added in the manuscript. A design effect of 1.2 was considered after a thoughtful consultation of published evidence in developing countries.

2. Comparison was made pre eVIN and post eVIN. Six months reference was taken Pre eVIN and three months post eVIN was compared. Comparison should have been done for similar period/ same months.

Apologies for the typing mistake in mentioning the period of post eVIN period. Pre and post eVIN period both were of six months duration for each of the phase. This has been corrected in the manuscript.

3. Major gain was reduction of Utilization Doses. (29.6% reduction of utilization doses). How many beneficiaries received immunization n pre eVIN and Ppost eVIN period.

The number of beneficiaries who received vaccines in pre eVIN and post eVIN phase is out of the scope of this assessment. The main aim of eVIN roll out was to manage vaccine supply chain and logistics therefore this study focused on this specific aspect. The vaccination coverage is dependent upon several supply and demand side factors including efficient management of eVIN system. The HMIS data provides the number of beneficiaries but it has not been presented due to self-reporting mechanism inherent in it.

4. How was states were selected. Expect Himanchal and Gujarat all other states started from weaker health system had poor indicators at the beginning.

There was no state selection in this assessment. By March 2018 (when this study was planned), eVIN roll- out was completed in 12 states of the country. Therefore all these 12 states were considered for a comparison of indicators in pre vs post eVIN phase.

The state wise rolling out of eVIN was a consensual decision between UNDP and state governments. ITSU had no role in the same.

________________________________________

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Paramita Sengupta

Reviewer #2: Yes: Binod Kumar Patro

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

14 Oct 2020

Programmatic Assessment of electronic Vaccine Intelligence Network (eVIN)

PONE-D-20-02529R1

Dear Dr. Dhalaria,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Khin Thet Wai, MBBS, MPH, MA (Population & Family Planning Resear

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Paramita Sengupta

Prof and Head, Department of Community Medicine and Family Medicine,

AIIMS Kalyani, Nadia, West Bengal

26 Oct 2020

PONE-D-20-02529R1

Programmatic Assessment of electronic Vaccine Intelligence Network (eVIN)

Dear Dr. Dhalaria:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Khin Thet Wai

Academic Editor

PLOS ONE