Evaluation of the rapid, multi-country, parallel process, multi-tasking approach to startup of short-term technical assistance to improve service delivery in newborn and child health in the context of USAID's Zika response in four Eastern and Southern Caribbean countries.

Background: In 2018, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project started a new partnership with four Eastern and Southern Caribbean countries impacted by the Zika virus: Antigua and Barbuda, Dominica, St. Kitts and Nevis, and St. Vincent and the Grenadines.  The goal of the project was to provide short-term technical assistance (STTA) to strengthen the health systems' capacity to detect newborns and young children potentially affected by Zika and to address their health needs.  To meet these objectives, ASSIST developed an innovative approach based on its existing model for service delivery improvement. This novel approach is known as Rapid, Multi-country, Parallel Process, Multi-tasking Approach for a Project Startup (RMPP-MAPS).  An evaluation was conducted to document the STTA startup activities, to identify enabling and constraining factors, and to capture lessons learned.
Methods: An external consultant conducted remote in-depth interviews with individuals involved in the startup using semi-structured interview guides and retrieved data from the review of project documents.
Results: Using RMPP-MAPS, the ASSIST Project successfully implemented the startup for complex STTA in four countries within less than four months, spanning mid-May to early September 2018. Project milestones included achieving buy-in from stakeholders, co-developing the technical scope and materials, and rapidly executing critical operational functions.  Dedicated project teams, country leaderships, and local champions were essential to overcoming the main challenges, which included a condensed timeframe, lack of in-country offices, and country-level factors such as a shortage of health care workers and a weak health infrastructure.  Conclusions: The RMPP-MAPS is a feasible and resource-efficient mechanism of interest to implementers, donors, and low and middle-income countries facing temporal and financial limitations to rapidly addressing public health priorities. Copyright:

Introduction

Background

The rapid spread of the Zika virus (ZIKV) infection to countries in the Latin American and Caribbean (LAC) region in 2015 and 2016 and its association with serious health consequences such as microcephaly prompted the World Health Organization (WHO) to declare the ZIKV infection a public health emergency of international concern (PHEIC) on February 1, 2016 . Although the WHO declared an end to the PHEIC in November of the same year, the ongoing Zika transmission in the region as well as the potential for future outbreaks required continuous vigilance to improve health systems preparedness and capacity in prevention, surveillance, and management of emerging infectious diseases .

In 2016, the United States Agency for International Development (USAID), the international community, and country governments began responding to the ZIKV epidemic in the LAC region . In 2017, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project implemented by University Research Co., LLC (URC) in Chevy Chase, MD, started activities in the Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Jamaica, Nicaragua, and Paraguay to strengthen Zika-related health services to deliver evidence-based, person-centered quality care with a focus on pregnant women, newborns, and women of reproductive age . In May 2018, USAID requested that ASSIST expand its efforts to four more countries in the Eastern and Southern Caribbean (ESC): Antigua and Barbuda, Dominica, St. Kitts and Nevis, and St. Vincent and the Grenadines. The request came in the last year of a three-year USAID Zika response plan in the LAC region. During this time, there was also a substantial decline in the number of Zika cases observed in the region. Therefore, the decision was made to design the effort as a one-year short-term technical assistance (STTA) activity aimed at improving detection and addressing the unique health needs of children potentially affected by Zika in the four countries. The STTA planned to achieve these objectives by strengthening newborn and well-baby care systems and early childhood development (ECD) programs in all functional health facilities providing childbirth services and well-baby care. These efforts were in line with national and USAID goals to improve public health emergency response and to strengthen the resilience of health systems to address future emergencies.

Rapid Multi-Country, Parallel Process, Multi-Tasking Approach for Project Startup

Given the short timeline and overarching objectives, ASSIST developed and applied the Rapid, Multi-country, Parallel Process, Multi-tasking Approach to Project Startup (RMPP-MAPS) in the four countries. The RMPP-MAPS required simultaneous initiation and execution of multiple planning processes; including but not limited to, achieving buy-in from Ministries of Health and participating facilities, co-developing the workplan and sustainability strategy, developing technical content and operational approach, identifying essential needs of facility teams to implement the activity, and initiating the procurement processes to address the logistical needs of the activity.

Basis and rationale for the RMPP-MAPS

The fundamental concept underlying ASSIST’s health systems improvement approach is understanding that “every system is perfectly designed to achieve exactly the results it achieves .” ASSIST views health systems as complex yet adaptive systems designed to undergo continuous reiterative processes of change and innovation. That process forms the basis of a six-component “Country Integrated Design” model (1-Improvement Design; 2-Implementation; 3-Sustainability; 4-Scale-up; 5-Institutionalization, and 6- Learning) that has been proven to effect positive outcomes over the three decades of USAID work (through the ASSIST and preceding projects) in the field of health system strengthening and quality improvement in health care services delivery . To achieve desired outcomes, all six components of the model must be integrated in planning improvement interventions. Thus, the process necessitates the engagement of all stakeholders throughout implementation. Depending on the scale of the problem and/or context the process can take up to several years. A typical model of ASSIST and other global health partners includes setting up, staffing country offices, and a step-by-step implementation approach. For the ESC countries, ASSIST’s task was to implement the improvement activity in multiple countries within a single year. Registering URC field offices within the countries and establishing project teams to locally support the operation was not feasible in the given timeframe. This constraint posed a significant challenge for logistics and implementation. To overcome the challenge, ASSIST undertook an innovative RMPP-MAPS which allowed simultaneous planning and execution of multiple startup processes in all four countries (instead of in linear, sequential fashion) and using real-time data or “feedback loops” to track progress.

Capturing any contextual similarities and differences early in the project is critical for the success of RMPP-MAPS. For the latter purpose, the team administered scoping surveys to national and regional stakeholders in advance, followed by initial country visits. Given the similar socio-economic, health care infrastructure or development levels, the common language, and geographic proximity, the RMPP-MAPS approach was applied universally across all four countries. The main differences, as relevant to the startup, were in the availability of local human resources (e.g., shortage of nurses) and qualified vendors for outsourcing operational and logistical tasks. In addition, the IT infrastructure readiness level, such as availability of computers, internet connectivity, or even availability of electricity in health facilities, varied within and between islands. The infrastructure-related challenges were due to differing severity of destruction caused by devastating hurricanes that passed through the region the prior year. For example, Dominica’s health infrastructure suffered more severe damages than the infrastructure in the other three countries.

Despite the above differences, the RMPP-MAPS was able to integrate real-time feedback, joint brainstorming, and adaptive implementation strategies to address unique contextual challenges and ensure simultaneous implementation of startup of activities and timely initiation of facility-level QI activities.

In documenting and evaluating this novel approach, URC has published observations and recommendations for similar rapid startup of donor-funded short-term technical assistance activities. The scope of this paper is to report on the findings of the evaluation of the startup of STTA, its activities, implementation, facilitating and constraining factors, and to capture lessons learned.

Results of the start-up phase

The start-up activities in all four countries were conducted simultaneously except for the country visits, which were conducted sequentially by the start-up team visiting one country after the other. All four countries committed to taking on the work and to allocating the human resources needed for it. In the start-up discussions, differences between the resource availability in different countries became clear. For example, some countries had shortages in staff required to act as coaches, while others did not. These were factored into the workplan through arrangements with the Caribbean Regional Nurses Association (CRNA). By the end of the start-up phase all four countries embarked on implementing the workplan.

Methods

Evaluation design

A process evaluation design and a qualitative study methodology were employed to address the evaluation objectives . The evaluation was retrospective with primary data collected through in-depth interviews with persons or key informants (KI) involved in the startup and secondary data retrieved from the review of project documents. Documents reviewed included: (1) startup documents obtained from the ASSIST Project (e.g., meeting records, activity plans, timelines, presentations, trip reports, scoping survey, etc.), (2) online public resources (e.g., USAID Zika program overviews, press-releases, ESC country-specific information, literature related to Zika outbreak in the region, summary of relevant work by implementing partners, etc.), and (3) documents obtained from program managers (e.g., Maternal and Child Survival Program reports and documents). Primary and secondary data were triangulated as necessary to document and develop a cohesive picture of the activity. See Extended Data for the documents reviewed.

Identification and engagement of key informants

KIs were defined as individuals who had direct or indirect knowledge, experience, and/or involvement with any stage of the project development or startup activities. The USAID ASSIST Project provided a list of recommended KIs to the independent research consultant (B.A.) who conducted the interviews. The list included technical partners, country program planners, project coordinators, chairs and members of community and hospital nursing divisions, and maternal and child health committees. An initial email was sent to all prospective participants by the USAID ASSIST Project to introduce the consultant and inform the eligible KI about the evaluation. The consultant then contacted each KI via email, provided details about the evaluation purpose and rationale, and invited him or her to participate in an interview via Zoom (or in some cases, over the telephone). When necessary, the USAID ASSIST Project team facilitated contact with stakeholders to confirm their participation as a KI.

In-depth individual interviews with key informants

The case study approach and evaluation or learning framework guided the development of key informant interview (KII) tools. Based on initial discussions with ASSIST program officers, the evaluation focus, objectives, main topics of interest were defined followed by identification of all stakeholders and key informants involved at each step of the startup. The interview guide was then structured to have sections around identified topics with questions and probes to facilitate detailed responses as necessary to answer evaluation questions of interest.

A semi-structured interview guide ( Extended data ) consisted of sections on the focus and objectives of the evaluation, timeline, and activities that took place during the defined startup timeframe. Probe questions focused on identifying and describing startup activities in chronological order, elucidating respondent’s views or understanding of the approach and methods employed, perceptions related to timeline or speed of the process, discussion of facilitating and hindering factors, capturing lessons learned and future recommendations. Project documentation was used to more precisely construct activities and identify additional topics of interest for inclusion in the interview guide. Questions were adapted according to the type of respondent (national or regional partner, implementing partner, or leading STTA provider) and to fit the extent of participants’ involvement in the startup. All interviews were conducted in English and lasted approximately 45 minutes. The interviews were conducted by an independent, external research consultant (B.A.), who was not part of USAID ASSIST Project and was not involved in a project startup or implementation. The consultant holds a Ph.D. in Epidemiology and has expertise in public health research and program evaluation..

Data analysis

All interviews were audio-recorded. Audio files were uploaded into password-protected computer files. Automated transcripts were then generated from interviews and transcribed using transcription software ( https://otter.ai). Transcripts were then manually checked and edited to ensure accuracy. The accuracy of transcription software was dependent on the respondent’s English accent (e.g., native English speakers in US vs. English of respondents from Caribbean countries), technical factors such as whether the interview was taken via phone, an online communication platform, or in-person, as well as recorded sound quality, presence of background noise, etc., and varied between 80 and 90 percent. Transcripts were not sent back to interviewees.

Thematic or pattern-based coding was applied to the data manually. Data were systematically grouped according to recurrent patterns and themes as related to central topics being evaluated (by B.A.), and codes were applied to identify segments of the interviews where these themes were discussed. The information obtained from the desk review was triangulated with the data retrieved from in-depth interviews to validate the information collected and more precisely map startup activities.

Ethical statement

Informed consent was obtained from each KI prior to the interview via email (consent form available as Extended data ). In addition, consent was obtained orally prior to recording each interview. Participants were informed of the voluntary nature of participation and the right to withdraw at any time or decline questions they did not wish to answer. Participants were also informed that their responses would be confidential (i.e., specific quotes or answers not linked to any specific respondent in the report). Interview audio, voice, and transcript records were stored securely in password-protected electronic files.

This evaluation was performed as part of the learning component of the USAID ASSIST Project and was not considered a research study subject to IRB review.

Limitations

Recall bias is a potential study limitation as interviews took place four to six months after the startup in the middle of the implementation phase. Participants sometimes had difficulties recalling details related to activities, dates, and sequences of events, and could not always distinguish startup from implementation activities. There is also a possibility for social desirability bias in response to sensitive questions (e.g., what the participant did not like about startup). The unavailability of some respondents due to scheduling or technical challenges is also a potential limitation.

Results

Characteristics of evaluation respondents

Invitations to participate in the startup evaluation interviews were sent out to a total of 34 individuals, out of whom 28 (82.4%) were successfully interviewed. Interviews took place between January 29, 2019 and April 7, 2019. Interviewed participants comprised of key stakeholders involved in the startup, including technical assistance providers, implementers, and partners, based in USA and the Caribbean region. The distribution of respondents included: four USAID Washington staff; four URC staff working on the USAID ASSIST Project; one Jhpiego staff member who worked on the USAID Maternal and Child Survival Program (MCSP); three staff of the American Academy of Pediatrics (AAP); one representative of the Caribbean Regional Midwives Association (CRMA); one representative of USAID/Guyana (overseeing USAID’s regional activities in ESC); as well as stakeholders from the four technical assistance recipient countries. Respondent distribution is further illustrated in Table 1.

Table 1.

Distribution of respondents.

Levels	Country	# invited	# interviewed
Country-level	Antigua and Barbuda	7	5
	Dominica	3	3
	St. Kitts and Nevis	6	2
	St. Vincent and the Grenadines	4	4
Regional Level	Trinidad and Tobago: CRMA	1	1
	Guyana: USAID Regional Representative	1	1
Global Level	USA: USAID, ASSIST, MCSP, AAP	12	12
TOTAL		34	28

Notes: Six individuals did not participate, citing “family emergency”, “not being knowledgeable enough about startup activities”, and “no longer being attached to the program” as the reason for declining; three responded to invitation but were not interviewed due to scheduling conflicts or internet access/technical difficulties with Zoom CRMA, Caribbean Regional Midwives Association; MCSP, USAID Maternal and Child Survival Program; AAP, American Academy of Pediatrics.

Stakeholders’ roles

Key stakeholders and their roles and responsibilities are shown in Figure 1. The ASSIST team in collaboration with country partners played a key role in the development of the overall scope, approach, activity plan, and implementation of the STTA. Jhpiego MCSP, a collaborating partner, worked closely with the ASSIST team to coordinate approaches and activities in the region and share lessons and experiences, as well as documents and training materials for adaptation and use in STTA. AAP, the project sub-awardee, was represented by Global Health Program Officers (management team) and pediatric consultants (technical team) and was engaged to bring in the subject matter expertise, specifically, to deliver training curricula on essential care for every baby (ECEB), and developmental pediatrics within the context of Zika. CRMA, the project sub-awardee, is a regional organization with members throughout the Caribbean region and was represented by an executive team. CRMA supported the selection and recruitment of quality improvement (QI) coaches. USAID’s Zika Program guided the overall direction and scope of the STTA.

Figure 1.

Map of stakeholders and communication/relationship linkages.

Notes: The central part of the chart shows the country Ministry of Health as the principal technical assistance recipient and stakeholder and the USAID ASSIST Project as the leading provider and implementing partner of USAID. Key supporting implementing partners and other national stakeholders are arranged around the above two main stakeholders. Arrows reflect the relationship and communication flows between different stakeholders. Sidebars display the key roles each stakeholder played in the startup. USAID Guyana and CRMA are in the middle between the main stakeholders as regional liaisons and partners. MCSP, USAID Maternal and Child Survival Program; AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; MOH, Ministry of Health; PAHO, Pan American Health Organization; CMO, Chief Medical Officer; WI-HER LLC, Women Influencing Health, Education, and Rule of Law; NGO, non-governmental organization; ECD, early childhood development; ECEB, essential care for every baby; QI, quality improvement.

Startup activities and timeline

The startup consisted of multiple processes implemented in parallel in the four countries between May 11, 2018 (when the assignment was given to the USAID ASSIST Project) and September 3, 2018. In general, all activities fall under four main categories: (1) conception and initiation activities, (2) workplan and timeline development, (3) technical content development, and (4) operations. A detailed summary list of activities under each category is shown in Table 2.

Table 2.

Summary list of startup activities.

Category		Activities
Conception and initiation activities	Achieving buy-in	Identify and contact key decision-makers in Ministry of Health to gauge interest and obtain initial support Determine country, regional stakeholders and US-based partners Initiate partnerships with stakeholders and sub-contractors Maintain communication, advocacy and engagement efforts to obtain full support and commitment of all concerned stakeholders Establish in-country activity teams (focal points, coaches, facility improvement teams, etc.) Start discussions with host country governments on sustainability and institutionalizing improvement activities
	Developing focus and scope of TA	Adapt and complete Scoping Visit Assessment Tool Conduct rapid baseline assessment of newborn and well-baby care systems with the focus on assessment of babies with suspected or confirmed congenital syndrome associated with Zika, and identify gaps, needs and priorities Plan and conduct country scoping visits with USAID global and regional representatives Develop the focus and scope of short-term technical assistance based on the scoping assessment and consultation with stakeholders Reach formal agreement with country stakeholders, USAID and technical partners on the scope of work
Workplan & timeline development		Develop detailed workplan with timeline and budget in close collaboration with USAID representatives, country partners and sub-contractors Submit workplan to USAID for review and approval
Technical Content Development	Training curriculum & materials	Quality improvement capacity building materials ZIKV-oriented early childhood development and essential care for every baby training materials Clinical capacity building materials and case studies based on the most current updates on Zika clinical care Neurodevelopmental surveillance and well-child care tools Pre/post knowledge assessment tools Capacity building materials for coaching support
	Data collection	Develop a set of indicators to track the progress of improvement activity Develop data collection reference guide Set up Improvement Indicator Database
Operations	Logistics, administrative functions, communication, financial management	Develop and approve sub-awards and agreements with partners Identify logistical needs (event organizers, information technology needs, supplies) Initiate multiple competitive procurement process including procurement of computers for health facilities, identifying workshop venues, accommodation, vendors for catering and transportation, etc. Develop/coordinate scheduling of activities with multiple stakeholders Find reproduction vendors and organize printing of training materials Finalize schedule, coordinate invitation and participation in workshop Arrange to award Continuous Medical Education credits to training participants through the countries' professional review boards Manage other administrative needs (documentation, reporting, etc.) Determine and streamline communication channels and principles between different stakeholders Determine accounting procedures, estimate cost and develop budget

The timeline of key activities is shown in Figure 2. In less than four months, the USAID ASSIST Project was able to achieve buy-in from all countries, develop a work plan with approval from the USAID and the four Ministries of Health, develop technical content and capacity building materials for the first TA visit, initiate agreements with three sub-awardees and event planners, establish facility-based improvement teams in each country, determine indicators to track improvements in service delivery, and finalize financial, organizational and logistical arrangements for both the scoping and the first technical assistance visits.

Figure 2.

Timeline of key activities.

AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; MOH, Ministry of Health; SOW, Scope of Work; IID, improvement indicator database.

Key themes and issues identified from interviews

In this section, we describe and discuss some of the common themes and issues identified from interviews as related to key processes, challenges encountered, and any enabling and hindering factors.

Given the limited timeframe, it was critical to secure buy-in for USAID Zika expansion efforts from key stakeholders in the new ESC countries from the very beginning. Achieving buy-in involved: (1) describing the problem and providing a convincing rationale for why it needs to be addressed; (2) informing stakeholders of the availability of funds and technical expertise that can be leveraged by countries to co-develop and implement a solution; and (3) mapping out the project’s direct outputs/outcomes and extended benefits. Involving country stakeholders at each step was critical to achieving buy-in both at the decision making and implementation levels. However, there were several constraining factors when it came to engaging country stakeholders. First, ASSIST did not have a prior presence in these four countries. As explained by one respondent, there were "reservations" on the part of countries since they were not familiar with ASSIST or its work. Second, by the time USAID approached countries, the Zika epidemic and the heightened urgency around Zika was essentially over. Finally, the countries faced competing priorities such as dealing with the aftermath of devastating hurricanes that ravaged the region in 2017. In order to address the countries’ needs, the activity focused on detection and support of babies potentially affected by Zika through strengthening essential newborn and well-baby care systems and early childhood development programs.

According to USAID and ASSIST respondents, initially, there were delays in establishing regular communications with the countries. Different “levels of engagement” were noted, with some country partners being more responsive compared to others. These challenges were mitigated through ASSIST’s continuous engagement with the USAID representative from the regional office, who facilitated direct contact with country stakeholders via email and phone. ASSIST also coordinated face-to-face meetings at a regional conference in Trinidad and during country scoping visits to ensure full commitment from country partners.

From the countries’ perspectives, the demonstrated experience of the USAID ASSIST Project and the proposal laid out during the initial discussions were noted as influential in deciding to get on board with the activity. As one country senior Ministry of Health (MOH) officer described:

“First of all, I found (USAID ASSIST) extremely knowledgeable. They have a wealth of experience and have done extensive work in quality all over the world. So, that for us was really important to know. Second, I think that they were quite sensitive to the specificities of all cultural nuances and …were quite responsive to suggestions we made and how to go forward. It was not just cordial… after a couple of weeks, we were talking like we've known each other for years. So, I think, that speaks to the commitment both parties had to this as well as the experience that USAID ASSIST has”.

(Senior MOH officer)

Additional contributing factor for accepting the USAID Zika STTA was the recognition of the need to have ongoing Zika surveillance and up-to-date national policy documents and guidelines for developmental surveillance, screening, referral, and support of children potentially affected by Zika. Country pediatricians have noted that the existing policy documents and guidelines on Zika were developed during the height of the epidemic and did not reflect the new evidence. For example, in Antigua and Barbuda, the guidelines for infants exposed in utero to ZIKV were developed in 2016 with technical assistance from Pan American Health Organization (PAHO). This was useful for clinical providers in responding to the outbreak at the time; however, the knowledge and evidence about Zika has since expanded. Additionally, terminology, including the definition of congenital syndrome associated with Zika (CSaZ), was not reflected. Therefore, the USAID Zika technical assistance was viewed by leading pediatricians in these countries as a “fantastic” opportunity to update relevant policy documents with the help of ASSIST and AAP.

"First of all, we recognize what we should follow in terms of surveillance, that we need to ensure we have the same case definition, that we have national policies, and we recognize that there is a limitation in terms of human resources. We also recognize that there needs to be someone external, who is looking into the weaknesses within the system, and that's one of the reasons that we were happy to have technical assistance. This is how we could improve because we're good on paper, we have many things in place, but the system needed to be improved and (made) more robust."

(Country team member)

Interviews with USAID respondents revealed that USAID designed a response framework around the four major lines of effort, each addressing a specific factor in preventing and controlling the spread of the Zika epidemic. The four lines of effort were: (1) vector control; (2) social and behavior change, communication, and community engagement; (3) service delivery; and (4) research, development, and innovation. Depending on the stage of the epidemic and the needs of the countries, implementing partners were guided by USAID to focus their efforts on one or more particular lines of effort to strategically align response activities amongst the various partners. Since ASSIST started its efforts in ESC countries, nearly two to three years after the outbreak had peaked, the country priorities shifted towards improving service delivery around identification of cases of CSaZ and other developmental malformations among infants and children exposed to ZIKV in utero and strengthening newborn and well-baby care systems to address other similar emergencies.

“The disease epidemiology shifted from when the first outbreak started, …trying to minimize another outbreak …to care and support for those who have been affected. Now, we are …trying to answer the question about the transition from emergency response to long-term development.”

(USAID Zika team member)

Therefore, the focus at this stage of the response was to improve and strengthen national health systems to enhance national capacities to address public health emergencies. Based on responses from USAID and country participants, the following factors have played a role in shaping the focus, scope and approach of the STTA: (1) timing or the current phase of the Zika epidemic; (2) timeframe of the USAID Zika response contract; (3) financial and human resources; and (4) country-level factors ( Figure 3).

Figure 3.

Key determining factors for the STTA focus, scope and approach.

TA, technical assistance; CSaZ, congenital syndrome associated with Zika; STTA, short-term technical assistance; MCSP, USAID Maternal and Child Survival Program; AAP, American Academy of Pediatrics; CRMA, Caribbean Regional Midwives Association; PAHO, Pan American Health Organization; WI-HER LLC, Women Influencing Health, Education, and Rule of Law; URC, University Research Co., LLC; WPP, World Pediatric Project.

All MOH focal persons and relevant decision-makers were engaged in the process of co-developing the work plan and finalizing the technical and geographic scope of the work.

Technical partners with specific areas of expertise were contracted after the scope and the types of technical expertise that would be needed had been defined. However, some partners wished they were involved in initial discussions. According to technical partners, earlier engagement would have helped identify clear expectations, roles, and responsibilities; while giving the partners time to plan their part of work according to the country contexts. The time left after achieving country buy-ins and initial discussions with countries to define TA focus and scope was less than a month.

. The approach was based on the ASSIST’s “model for improvement” and “adaptive management” strategies. These strategies emphasize identification of gaps and testing of local solutions to problems in real time, adjusting plans throughout the process, and implementing the results of regular monitoring of improvement interventions. Among the advantages of a rapid approach, respondents noted faster implementation, early achievement of results, expedited scaling and delivery of patient-favorable outcomes into the health services, and the efficient use of resources. Disadvantages included a possible loss of specificity and the potential for missing subtle details in the design during the early stages of response. Only a few respondents raised concerns about the possibility of diminished quality of response. Respondents emphasized that a continuous feedback mechanism and real-time adaptive management and learning strategies helped to address quality or other concerns promptly. While constant feedback and inputs from all partners were actively sought and incorporated at each step, the limited time frame made it impossible to wait until all components and specific details were in place or agreed upon by all stakeholders. Rather, the strategy adopted by ASSIST was to proceed with all necessary startup activities simultaneously to avoid delays while refining and/or making adjustments along the way as needed.

The only way to improve care is to change what is happening at facilities. So, our goal was to start facility-level improvement activities at the earliest point as possible. Knowing from previous experiences that planning is most effective when it is informed by implementation, we tried to start as fast as possible and use robust continuous feedback loops at every level to learn and adapt.”

(ASSIST team member)

The small population and geographic size of the four countries facilitated the full scale of the improvement activity from the start. Except for the few community-based clinics in Barbuda and Dominica that were damaged by hurricanes, all facilities were included in the improvement intervention. To promote early adoption and institutionalization of changes and cross-sectoral collaboration ASSIST engaged health workers at all levels of the health system and encouraged the involvement of early childhood developmental programs under the Ministry of Education, and other community-based organizations (e.g., Roving Caregivers Program in Dominica). Both scaling up and institutionalization are essential components of ASSIST’s health service improvement model. These features of the STTA were noted by many country respondents as unique when compared to other technical assistance projects they had experience within their countries. According to country representatives, often in such projects, information "stays at the top" while "people at the bottom" are left poorly informed and tasks are "thrown" at them. Many pointed out that frontline and mid-level health workers (and, in some cases, community health aides and nursing assistants) had benefited greatly from training on clinical care of Zika-affected newborns and children. The science of QI empowered health workers to improve processes within the newborn and well-baby care system and equipped them with the skills and knowledge to apply improvement principles to other priority clinical-content areas.

Ensuring the participation of all stakeholders was not without challenges. The health systems in three of the four countries (except St. Vincent and the Grenadines) were suffering from severe shortages of health care workers, particularly nurses. Many nurses and other health care workers had migrated out to other countries following recent hurricanes, exacerbating pre-existing shortages. Hence, some health facilities were concerned additional project-related responsibilities would overwhelm staff members. Due to this shortage of local health workers and heavy workload, country partners and ASSIST used retired nurse midwives who were already familiar with the health system. St. Vincent and the Grenadines was the only country that did not request external help to provide on-site coaching support to health facilities. According to a MOH officer, there was no shortage of nurses in the country. St. Vincent and the Grenadines relied on the existing nurse supervisors to take on coaching responsibilities. St. Vincent and the Grenadines did, however, experience the same challenge as the other countries when it came to "pulling the staff" for a week-long technical session while ensuring no interruption in health services.

Country teams and facilities had to shuffle various schedules and calendars around clinic days or public holidays to prevent disruption of health services and accommodate project activities.

“Well, the most stressful part of it was getting the persons together…, that was kind of a tedious job getting into different entities. The time that we had in order to prepare for that first visit and the categories of individuals that they were requesting to meet with, the timeframe was a bit short in order to get all those persons together. Persons need prior notice in order to be released. We weren't giving them that prior notice. We are basically giving them “a meeting is tomorrow and I need X number of staff”. They don't have that time to prepare…”

(Country team member)

Country team leaders noted that promoting a better understanding of the improvement activity, the benefits of adopting changes, and support from ASSIST helped to overcome the initial resistance. As one country focal point put it, they had to “sell the idea” to staff. A dedicated leadership team and local champions were crucial in creating a “positive competitive environment” to motivate increased participation and involvement. The result was increased interest and demand for participation in technical learning sessions observed from the second TA visit, with more health providers attending the workshops than initially planned.

Technical content development focused on two major areas: (1) the science of QI led by the ASSIST team; and ECEB training with a focus on ZIKV led by AAP. One of the challenges identified by respondents was the lack of technical experts with expertise in both ECEB and ZIKV. According to AAP, the pool of experts trained in ECEB was limited and finding available experts to work on developing the curriculum on “a very short turnaround time” presented significant challenges. Nevertheless, the AAP team was instrumental in “pulling together” their existing materials and information resources, as well as a team of consultants to develop high-level technical tools. In mitigating short timeframe challenges, AAP and other implementing partners emphasized the importance of early involvement of technical partners in the rapid startup. Additionally, providing technical consultants with as much background or context data on local health systems and related issues in advance, was highlighted as an essential best practice.

Operations included functions related to defining administrative support (documentation, reporting, contracting, etc.), financial management (budgeting, accounting, payments, etc.), streamlining communications between multiple stakeholders, and logistical needs (procurement/supply, event organization, etc.). Given the lack of field offices and URC staff within the countries, administrative and financial management was centralized from ASSIST headquarters in Chevy Chase, MD, USA with country-specific adjustments made where necessary.

Logistical activities were outsourced to external organizations. The main challenge was identifying reliable event planning or logistics services on the ground who have intimate knowledge of local markets, suppliers, distributors, or experience with procurement management and competitive purchasing processes. For example, no reputable event planning organizations meeting the minimum required qualifications were identified in St. Vincent and the Grenadines. A unique logistical challenge for St. Vincent and the Grenadines were the delays arranging transportation for the "nurses from the Grenadines to come to the mainland" (some of the Grenadines’ nine islands are small and remote). Therefore, ASSIST contracted two independent consultants to take on the role of event planners. The ASSIST team worked with MOH focal points in all four countries to help identify and provide recommendations on potential vendors and co-implement activity to help "build local ownership and capacity."

Financial management followed USAID accounting procedures and standards according to the terms of the ASSIST cooperative agreement between USAID and URC. Planning and budgeting took place at the same time as workplan, staffing, and operational needs were being defined. Difficulties were reported in prompt costing of local activities.

Figure 1 illustrates the flow of communication between all stakeholders and partners. The ASSIST team and the focal points from the country teams were the two parties in primary contact through whom all messages from other partners were communicated. This was established to ensure consistency of messages across multiple partners. According to ASSIST, the sub-awardees were not in direct communication with country stakeholders during the early start-up phase to facilitate focused conversations about buy-in or the technical scope. However, relevant updates and decisions were shared in a timely manner. Communication was open and regular, informing all relevant stakeholders, including local stakeholders and partners. There were also regular conference calls involving stakeholders from all islands. Emails to relevant stakeholders and partners were answered within 24 hours, and urgent issues were handled through telephone, Zoom calls, and WhatsApp.

. Participants reported that given the substantial amount of work in preparation for the first TA session in early September and very short timeframe, the startup had to be rapid and fast-paced, which had both advantages and disadvantages. The rapid startup galvanized stakeholders to brainstorm and make decisions quickly. However, the rapid speed may not have allowed enough time for detailed or thorough follow-up of some tasks (e.g., not all materials were printed out in time for the first technical session). When asked about how the pace of the startup compared to other projects, most respondents indicated that they are not aware of any other projects of the scale, depth, or the speed and pace that occurred with the USAID ASSIST Project. Both ASSIST team members and partners reported exerting a great level of effort to successfully and efficiently manage multiple tasks at the same time within a short timeframe.

Lessons learned and recommendations

This evaluation revealed that a rapid startup of complex service delivery technical assistance with activities in multiple countries without field offices was feasible. Although there were several unexpected implementation bottlenecks, by the end of the startup there was a collective realization among project teams and partners that what seemed at first a “mission impossible” was, in fact, possible, and had been accomplished. In addition, the evaluation proved that in the context of a short timeframe and urgent nature of the task, defining essential startup steps and processes early in the course of activity while at the same time identifying and accounting for specific contextual challenges is critical to avoid delays and facilitate smooth execution.

Additional lessons gleaned from respondents’ comments are summarized and listed below. While some of these lessons are not unique to the experience of the USAID ASSIST Project, they were distinctly reinforced by the course of startup and proved to be valuable for future startups.

At technical assistance/implementing partner level:

To achieve buy-in:

Maintain regular dialogue with senior country-level decision-makers to achieve prompt buy-in

When developing response/designing TA:

Involve implementing partners and collaborators early

Establish efficient but flexible communication principles

When setting up operations:

Learn as much as possible about host country context and determine optimal operational mechanisms, including:

Identify knowledgeable individuals and/or entities locally to take on or share logistical responsibilities

Determine context-appropriate contracting and financial management strategies

At the country/beneficiary level:

When co-developing and co-implementing TA:

Conduct rapid assessment of the health system and service delivery needs jointly with key national stakeholders and subsequently with implementing partners

Co-develop interventions with a focus on sustaining the impact at scale, tailoring TA to country context, capacity development needs, and achieving local ownership of the activity

When implementing TA:

Ensure activities are naturally integrated within the existing system, structures, and functions and not run or perceived by staff as a “parallel” task. Create a positive environment among health workers and avoid or mitigate resistance to change

Establish routine communication with key country stakeholders, partners, and donors to update them about progress, respond to their needs, promptly communicate challenges, and seek joint solutions

Identify and mitigate implementation barriers

Gather intelligence on logistical challenges from key local stakeholders

Identify and engage reliable or experienced local vendors early (before initial scoping visit) to support logistics of field activities, avoid delays, and ensure rapid start-up of TA activities

Both TA providers and beneficiary countries should aim to find an optimal balance between country needs and/or expectations and what technical partners and donors can offer, and to identify and acknowledge various limitations (programmatic content, financial, human resources, etc.) in advance. Discussions on sustainability and transition plans as part of the startup will facilitate successful collaboration and ensure the achievement of project goals and outcomes.

Finally, the presence of committed leadership, dedicated project teams, and local champions with the ability to collaborate on project strategy and adaptation was repeatedly emphasized as a critical process. The latter point consistently comes up as an essential factor in all projects ASSIST has implemented to date globally .

Recall bias is a potential study limitation as interviews took place four to six months after the startup in the middle of the implementation phase. Participants sometimes had difficulties recalling details related to activities, dates, and sequences of events, and could not always distinguish startup from implementation activities. There is also a possibility for social desirability bias in response to sensitive questions (e.g., what the participant did not like about startup). There is also chance for respondent or KI selection and inclusion process. However, the risk is believed to be minimal as all individuals involved at any stage of the startup were identified as potential KIs, contacted and invited to participate in the study with response rate exceeding 80 percent.

Conclusions

In less than four months, the USAID ASSIST Project developed and carried out an innovative RMPP-MAPS to lay the groundwork for the implementation of a complex STTA activity in the four Caribbean countries. The STTA was implemented in full scale in all four countries and engaged health workers, educators, and caregivers at all levels of the newborn and well-baby service delivery system, promoting early institutionalization and long-term sustainability of the evidence-based interventions.

Operating under the USAID “Journey to Self-Reliance” agenda, the ASSIST Project worked closely with government leaders to achieve locally sustained results and strengthen local capacities with the end goal of a complete transition to the government at the end of the project. The combination of improvement expertise and upfront clarity on roles, responsibilities, and transition expectations facilitated teaming up with host country nationals and transferring competencies to them.

Challenges encountered in the startup stemmed from the tight timeline, the lack of in-country offices, and the shortage of healthcare staff in the ESC countries. Despite these challenges, the RMMP-MAPS model for startup facilitated a rapid launch of the implementation phase of the STTA. Dedicated project teams, country leadership, local champions, and a strong sense of partnership and country ownership were identified as critical factors in facilitating a successful startup.

Although no cost analysis was done as part of this evaluation, based on the ASSIST Project’s decades of experience in implementing health systems strengthening TA in numerous countries, the RMPP-MAPS appears to be a cost-efficient model in time and resource constrained circumstances. Similar TA under normal circumstances typically involved setting up country offices, hiring local staff, external and internal consultants with startup activities taking much longer, often over a year incurring significantly higher costs before TA is ready to launch.

In conclusion, the RMPP-MAPS is a feasible and resource-efficient model to consider by implementers, donors, and low-and middle-income countries facing temporal and financial constraints when responding to public health emergencies.

Data availability

Underlying data

In order to encourage free and open sharing of opinions and perceptions, we assured stakeholders that their responses would be confidential. Since interview questions were shaped around a participant’s specific role and involvement level in the startup, their responses were correspondingly specific to their particular experiences and views. Many organizations were represented by no more than 1–2 respondents. Hence, interview files (transcripts and audio recordings) will contain potentially identifying information and thus, cannot be made openly available. Those interested in accessing this data may contact the corresponding author, at rashadmassoud@outlook.com. Requests will be considered on a case-by-case basis and must include contact information, type of data requested, and justification (e.g., proposal) for how the data will be used and for what purposes. Any information that could potentially identify respondents or organizations will be removed from the data shared.

Extended data

Open Science Framework: Zika Startup Evaluation Manuscript Extended Data, https://doi.org/10.17605/OSF.IO/B7QGS 10

This project contains the following extended data:

Short-Term Technical Assistance startup review documents

Semi-structured interview guides

Consent form

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

The authors have responded to the comments and the article is approved for indexing from my perspective.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Implementation science, QI, MNCH.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The authors have provided in-depth responses to both reviewers' comments. However, while some of these responses are reflected in the revised manuscript, others remain in the reviewer responses. Matching these carefully crafted responses into edits would strengthen the paper and fully address the reviews' questions. For example, the qualitative analysis methods, the question about aligning the scope of the paper ("An evaluation was conducted to document the STTA startup activities, to identify enabling and constraining factors, and to capture lessons learned."), while other parts talk about the full implementation (albeit with less details than start-up) and sustainability. A specific focus of the paper at the end of the introduction could help orient the reader. Clarity on the goals and, if included, linking implementation to the start-up if the data allows, detailing a bit more about the TA. Addressing all the comments through edits (or noting if not and why) will significantly strengthen this important paper.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Implementation science, QI, MNCH.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Dear Lisa,

The evaluation was conducted on the start-up period of the project from the initial receipt of the assignment to the point at which technical activities were started. It is specifically about how we rapidly started up the project in four countries simultaneously - working with the four Ministries of health; connecting with other stakeholders in each country as appropriate; forming improvement teams in multiple facilities in each country; and creating the supervisory, learning and improvement networks between them. This study is not an evaluation of the project itself. The project is referred to only in the context of giving the reader a better understanding of the nature of what was started and why it was important to do so rapidly. I hope this addresses your question. Thank you.

Kind regards, Rashad

The previous comments have been addressed and if the remaining comments below are addressed, the manuscript may be approved.

The Methods section should include that 'inductive or thematic analysis' was adopted for the analysis and interpretation.

As the data were analyzed by one person/author, it may be mentioned in the limitation section.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Public health

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

This report presents the process and experience of the technical assistance to the four LAC countries for addressing the Zika challenge. The documentation and dissemination are important for lessons and potential application in the future.

The manuscript/article can be improved by considering the following points:

1. Data collection and analysis

1.1. How were the KI IDI guides drafted/developed? It should be mentioned.

1.2. Which method was used for qualitative data analysis? It should be mentioned.

1.3. Was the data analysis done by only one person (B.A.)? Usually, qualitative analysis is done by more than one person.

1.4. What was the degree of error with the use of the transcription software?

2. Results

2.1. How did the status of preparedness, responses, perceptions, practices, and response components vary across these countries? These should have been mentioned.

2.2. Triangulation with the record review/desk review has been mentioned in methods. The observations have not been mentioned clearly in the results section.

2.3. Some narration about the results of the start-up activities should be given - whether the countries carried on the activities as planned or not.

2.4. The conclusion mentions the TA as a resource-efficient model. Some information about the resource-efficiency should be mentioned.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Public health

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Response to Reviewer #2:

The manuscript/article can be improved by considering the following points:

1. Data collection and analysis

1.1. How were the KI IDI guides drafted/developed? It should be mentioned.

Author response:  To draft KI interview guides initial list of topics of interest to be evaluated around the startup components was developed based on preliminary discussions with ASSIST program officers. These initial conversations helped define the focus and objectives of the evaluation, develop a comprehensive list of stakeholders and key informants involved at any stage of the startup, precisely define the startup period and outline specific processes or activities that took place within the given timeframe.

Additionally, project documentation, including meeting minutes, presentations, trip reports, work plans, USAID and partners' program documents, etc., were reviewed to more precisely construct activities and identify additional topics of interest for inclusion in the interview guide.

The interview guide was structured to have sections built around identified topics and startup components with specific questions and probes drafted to facilitate getting answers for evaluation objectives. Prepared draft tools were reviewed by ASSIST, and received feedback was used to finalize the tools.

We have described the above development process in the manuscript, in brief, to keep the length of the manuscript within the recommended guidelines.

1.2. Which method was used for qualitative data analysis? It should be mentioned.

Author response:  The number of respondents was not large, under 30, but respondents varied widely by which stakeholders or countries they represented. Additionally, interviews were conducted in a free-flow conversation style which allowed a diverse range and at the same time often unique answers or specific issues raised by respondents. Therefore, to best capture the range, the grouping or coding of emerging common and unique response themes was done manually (i.e., no automated qualitative analysis software or tools were used).

1.3. Was the data analysis done by only one person (B.A.)? Usually, qualitative analysis is done by more than one person.

Author response:  Data analysis was done by B.A., who was not involved in the project and contracted to perform an independent evaluation of the activity and avoid potential bias if the evaluation was to be done by ASSIST people themselves. The evaluation was performed as part of the learning process of the project. It was not considered a research subject to IRB review, which would have, perhaps, required more rigorous data analysis or verification strategies.

However, all collected data (audio records, transcripts of interviews) were submitted to ASSIST and independently verified by supervising program officer who was not involved in the startup activities for completeness, data integrity, and record-keeping purposes according to program rules.

1.4. What was the degree of error with the use of the transcription software?

Author response:  The accuracy of the transcription software varied depending on the respondent’s English speaking style and accent (native English speakers in the US vs. English of respondents from Caribbean countries). Also, technology factors such as whether the interview was via phone, or Zoom platform, in-person, recorded sound quality, presence of background noise, etc., played a role. On average, the accuracy was between 80 and over 90%.

2. Results

2.1. How did the status of preparedness, responses, perceptions, practices, and response components vary across these countries? These should have been mentioned.

Author response:  The four countries were similar in terms of economic development level or health infrastructure state. As relevant to the startup, most differences were observed in the levels of availability or the shortage of local health care providers, particularly nurses; availability of qualified local vendors for outsourcing operational and logistical tasks; as well as relatively poor technological infrastructure in facilities, in terms of availability of computers, internet connectivity, even electricity in some cases. The latter was partly due to the differing severity of damages caused by hurricanes that passed through the region the prior year. Hence, priorities or promptness in responsiveness varied somewhat between countries but consistent follow-up and identifying bottlenecks in a timely manner helped the team address the challenges and needs without significant delay and allow simultaneous initiation of the technical phase of the TA.

2.2. Triangulation with the record review/desk review has been mentioned in methods. The observations have not been mentioned clearly in the results section.

Author response:  Because KI typically were few in numbers representing different stakeholders and each played a unique role in the project, no one informant could provide a comprehensive picture “from the start to the end.”

Therefore, triangulation or tracing and comparing data obtained from multiple sources (preliminary discussions, in-depth interviews, project documents, etc.) was done to corroborate information, construct a cohesive and accurate picture of the startup, deduce the timeline and sequence of events, or elucidate enabling and limiting factors in order to reach evaluation objectives.

2.3. Some narration about the results of the start-up activities should be given - whether the countries carried on the activities as planned or not.

Author response:  The start-up activities were implemented in all four countries as described.

2.4. The conclusion mentions the TA as a resource-efficient model. Some information about the resource-efficiency should be mentioned.

Author response:  No specific cost analysis was done as part of this evaluation. The conclusion was based on the previous experience of ASSIST program leaders and TA implementers, other stakeholders, who had decades of experience in implementing similar health systems strengthening TA in multiple countries in different regions. Similar TA under normal circumstances typically involved setting up country offices, hiring local staff, external and internal consultants with startup activities taking much longer, often over a year incurring significantly higher costs before TA is ready to launch.

The manuscript/article can be improved by considering the following points:

1. Data collection and analysis

1.1. How were the KI IDI guides drafted/developed? It should be mentioned.

Response: To draft KI interview guides, an initial list of topics of interest to be evaluated around the startup components was developed based on preliminary discussions with ASSIST program officers. These initial conversations helped define the focus and objectives of the evaluation, develop a comprehensive list of stakeholders and key informants involved at any stage of the startup, precisely define the startup period and outline specific processes or activities that took place within the given timeframe.

Additionally, project documentation, including meeting minutes, presentations, trip reports, work plans, USAID and partners' program documents, etc., were reviewed to more precisely construct activities and identify additional topics of interest for inclusion in the interview guide.

The interview guide was structured to have sections built around identified topics and startup components with specific questions and probes drafted to facilitate getting answers for evaluation objectives. Prepared draft tools were reviewed by ASSIST, and received feedback was used to finalize the tools.

We added the above additional details to the revised version.

1.2. Which method was used for qualitative data analysis? It should be mentioned.

Response: The number of respondents was not large, under 30, but respondents varied widely by which stakeholders or countries they represented. Additionally, interviews were conducted in a free-flow conversation style which allowed a diverse range and at the same time often unique answers or specific issues raised by respondents. Therefore, to best capture the range, the grouping or coding of emerging common and unique response themes was done manually (i.e., no automated qualitative analysis software or tools were used).

1.3. Was the data analysis done by only one person (B.A.)? Usually, qualitative analysis is done by more than one person.

Response: Data analysis was done by B.A., who was not involved in the project and contracted to perform an independent evaluation of the activity and avoid potential bias if the evaluation was to be done by ASSIST people themselves. The evaluation was performed as part of the learning process of the project. It was not considered a research subject to IRB review which would have, perhaps, required more rigorous data analysis or verification strategies.

However, all collected data (audio records, transcripts of interviews) were submitted to ASSIST and independently verified by supervising program officer who was not involved in the startup activities for completeness, data integrity, and record-keeping purposes according to program rules.

1.4. What was the degree of error with the use of the transcription software?

Response: The accuracy of the transcription software varied depending on the respondent’s English-speaking style and accent (native English speakers in the US vs. English of respondents from Caribbean countries). Also, technology factors, such as whether the interview was via phone, or Zoom platform, in-person, recorded sound quality, presence of background noise, etc., played a role. On average, the accuracy was between 80 and over 90%. We added this information to the manuscript.

2. Results

2.1. How did the status of preparedness, responses, perceptions, practices, and response components vary across these countries? These should have been mentioned.

Response: The four countries were similar in terms of economic development level or health infrastructure state. As relevant to the startup, most differences were observed in the levels of availability or the shortage of local health care providers, particularly nurses, availability of qualified local vendors for outsourcing operational and logistical tasks, as well as relatively poor technological infrastructure in facilities in terms of availability of computers, internet connectivity, even electricity in some cases. The latter was partly due to the differing severity of damages caused by hurricanes that passed through the region the prior year. Hence, priorities or promptness in responsiveness varied somewhat between countries but consistent follow-up and identifying bottlenecks in a timely manner helped the team address the challenges and needs without significant delay and allow simultaneous initiation of the technical phase of the TA.

2.2. Triangulation with the record review/desk review has been mentioned in methods. The observations have not been mentioned clearly in the results section.

Response: Because KI typically were few in numbers representing different stakeholders and each played a unique role in the project, no one informant could provide a comprehensive picture “from the start to the end”. Therefore, triangulation or tracing and comparing data obtained from multiple sources (preliminary discussions, in-depth interviews, project documents, etc.) was done to corroborate information, construct a cohesive and accurate picture of the startup, deduce the timeline and sequence of events, or elucidate enabling and limiting factors in order to reach evaluation objectives.

2.3. Some narration about the results of the start-up activities should be given - whether the countries carried on the activities as planned or not.

Response: Results of the start-up phase

The start-up activities in all four countries were conducted simultaneously except for the country visits, which were conducted sequentially by the start-up team visiting one country after the other. All four countries committed to taking on the work and to allocating the human resources needed for it. In the start-up discussions, differences between the resource availability in different countries became clear. For example, some countries had shortages in staff required to act as coaches, while others did not. These were factored into the workplan through arrangements with the Caribbean Regional Nurses Association (CRNA). By the end of the start-up phase all four countries embarked on implementing the workplan.

The above paragraph was added to the article to address this point

2.4. The conclusion mentions the TA as a resource-efficient model. Some information about the resource-efficiency should be mentioned.

Response: No specific cost analysis was done as part of this evaluation. The conclusion was based on the previous experience of ASSIST program leaders and TA implementers, other stakeholders, who had decades of experience in implementing similar health systems strengthening TA in multiple countries in different regions. Similar TA under normal circumstances typically involved setting up country offices, hiring local staff, and external and internal consultants, with startup activities taking much longer, often over a year, incurring significantly higher costs before TA is ready to launch.

The authors report on a rapid approach to provide the start-up across 4 countries in the Eastern and Southern Caribbean to begin work to improve detection of and care for children affected by Zika virus. The work is embedded in the ASSIST project, which brings deep experience in establishing quality improvement work across a range of settings, expertise which is clearly reflected in the development of this rapid approach (RMPP-MAPS) to provide short term TA reflective of the timeline of overall funding.

The report is important in providing some important strategies for how to rapidly implement the key steps and operations needed to move rapidly, although this is relative to usual times for non-emergency interventions which would require even more rapid work (about 4 months). The strengths of the work include using an external evaluator, and the range of KIIs used. However the report could be significantly strengthened in a number of areas which need clarification.

Methods: What, if any, frameworks were used for the qualitative coding (and KII guide development)? How were contextual factors captured and compared across the countries for similarities and differences which may have influenced implementation success of the start-up? This is described in a few areas (nurses coming from the Grenadines, coaches from existing resources versus retired nurses for example), but would be very helpful for others to be described and any association with variability of RMPP-MAPS implementation. In addition, how did ASSIST decide who to suggest for KIIs? Was there any chance of bias (should include in your limitations-which also typically go into discussion sections).

Results and Discussion

The scope of the evaluation is a bit confusing if this is just about start-up (as described in the results and abstract) or overall program implementation, which is described in the discussion. The authors should include information on how this approach was successful in rapid start and the outcomes of the STTA, or modify the language about the definitions of success.

For example, in the conclusions, I do not see the relevance of the decades from research to implementation, as this was more about implementing and strengthening established EBIs? Also some data on if the EBIs were indeed implemented would strengthen the paper. Particularly given the sentence in the preceding paragraph that the broad engagement of stakeholders resulted in “promoting early institutionalization and long-term sustainability”.

For the knowledge to be more transferable, more details of how in addition to what was done would be helpful particularly in getting leadership ownership, developing a culture for change and other areas to which ASSIST brought expertise. How did this expertise influence the success and what would another organization need to have (or not need to have) to be able to adopt the RMPP-MAPS. For example, highlighting that this was done without country offices is important for replication. What else? What partners should they choose?

The focus on engaging a broad range of stakeholders is admirable and important. A bit more detail in how the stakeholders were identified and if the calls involving stakeholders were with all of them or more specific groups (page 11).

The approach is described as "resource-efficient”, but no data on cost is included. That would be important as this is indeed an important factors on which TA is not always evaluated.

Finally, the report meets many of the Consolidated criteria for reporting qualitative research (COREQ) and with a small effort would help strengthen the paper.

Minor:

Page 8; The authors write the approach was based on the ASSIST “model for improvement” and “adaptive management”. This needs a reference.

Typo: Experience top of page 10 should be experienced.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Implementation science, QI, MNCH.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The authors report on a rapid approach to provide the start-up across 4 countries in the Eastern and Southern Caribbean to begin work to improve detection of and care for children affected by Zika virus. The work is embedded in the ASSIST project, which brings deep experience in establishing quality improvement work across a range of settings, expertise which is clearly reflected in the development of this rapid approach (RMPP-MAPS) to provide short term TA reflective of the timeline of overall funding.

The report is important in providing some important strategies for how to rapidly implement the key steps and operations needed to move rapidly, although this is relative to usual times for non-emergency interventions which would require even more rapid work (about 4 months). The strengths of the work include using an external evaluator, and the range of KIIs used. However the report could be significantly strengthened in a number of areas which need clarification.

Methods: What, if any, frameworks were used for the qualitative coding (and KII guide development)?

Response: The case study approach and associated evaluation or learning framework guided the development of KII tools. In particular, based on initial discussions with ASSIST program officers, the evaluation focus, objectives, main topics of interest were preliminarily defined followed by identifying all stakeholders and key informants involved at each step of the startup. The interview guide was then structured to have sections around identified topics with questions and probes to facilitate detailed responses as necessary to answer evaluation questions of interest. Thematic or pattern-based coding was applied to the data manually.

How were contextual factors captured and compared across the countries for similarities and differences which may have influenced implementation success of the start-up? This is described in a few areas (nurses coming from the Grenadines, coaches from existing resources versus retired nurses for example), but would be very helpful for others to be described and any association with variability of RMPP-MAPS implementation.

Response: Contextual similarities and differences were systematically identified and documented via scoping survey by the TA team during initial visits to countries. Specifically, similarities across countries in terms of socio-economic or health infrastructure development levels allowed the simultaneous and universal application of the approach. As relevant to the startup, the differences observed were more pronounced in the levels of availability of local human resources (e.g., shortage of nurses) or qualified vendors on the ground for outsourcing operational and logistical tasks. In addition, some differences were observed in the IT infrastructure readiness level, such as availability of computers, internet connectivity, or even availability of electricity in certain facilities due to differences in the destruction level caused by hurricanes that passed through the region the year before (e.g., Dominica’s health infrastructure was severely damaged compared to other countries). Therefore, appropriate specific adjustments were made to the startup planning and implementation to address any unique to specific country challenges. As indicated in the paper, to deal with these contextual differences, ASSIST integrated into its RMPP-MAPS real-time feedback mechanisms, joint brainstorming, and adaptive implementation to address challenges promptly and initiate facility-level QI activities in all four countries at the same time.

In addition, how did ASSIST decide who to suggest for KIIs? Was there any chance of bias (should include in your limitations-which also typically go into discussion sections).

Response: All individuals who were involved at any stage in the start-up were identified as KI, contacted and invited to participate in the study. The response rate was over 80%. Therefore, we believe the risk for respondent selection bias is minimal. Thank you and we included this information to the limitations which were also moved to the discussion section as suggested.

Results and Discussion

The scope of the evaluation is a bit confusing if this is just about start-up (as described in the results and abstract) or overall program implementation, which is described in the discussion. The authors should include information on how this approach was successful in rapid start and the outcomes of the STTA, or modify the language about the definitions of success.

For example, in the conclusions, I do not see the relevance of the decades from research to implementation, as this was more about implementing and strengthening established EBIs? Also, some data on if the EBIs were indeed implemented would strengthen the paper. Particularly given the sentence in the preceding paragraph that the broad engagement of stakeholders resulted in “promoting early institutionalization and long-term sustainability”. “The RMPP-MAPS model for startup facilitated a rapid launch of the implementation phase of STTA. The STTA was implemented in full scale in all four countries and engaged health workers, educators, and caregivers at all levels of the newborn and well-baby service delivery system promoting early institutionalization and long-term sustainability of the evidence-based interventions”.

Response: The scope of the evaluation was limited to startup only. However, since the success of the startup phase is inevitably linked to the success of the implementation phase, we tried to mention in some form the effect on the implementation phase or the STTA itself. Regarding EBI, we removed the corresponding part on “decades of research to implementation” to avoid any confusion. We modified the language in the manuscript as shown below:

For the knowledge to be more transferable, more details of how in addition to what was done would be helpful particularly in getting leadership ownership, developing a culture for change and other areas to which ASSIST brought expertise. How did this expertise influence the success and what would another organization need to have (or not need to have) to be able to adopt the RMPP-MAPS? For example, highlighting that this was done without country offices is important for replication. What else? What partners should they choose?

Response: Operating under the USAID “Journey to Self-Reliance” agenda, the ASSIST Project worked closely with government leaders to achieve locally sustained results and strengthen local capacities, with the end goal being that the work would fully transition to the government by the end of the project. This was reiterated in conversations over the life of the activity, and plans for sustainability and transition were discussed regularly. The combination of improvement expertise and upfront clarity on roles, responsibilities, and transition expectations facilitated teaming up with host country nationals and transferring competencies to them.

The focus on engaging a broad range of stakeholders is admirable and important. A bit more detail in how the stakeholders were identified and if the calls involving stakeholders were with all of them or more specific groups (page 11).

Response: Key contacts and stakeholders in the country and the region were identified in conjunction with USAID and then in conjunction with the key contacts in the country for other partners within the country.

The approach is described as "resource-efficient”, but no data on cost is included. That would be important as this is indeed an important factors on which TA is not always evaluated.

Response: No specific cost analysis was done as part of this evaluation. The conclusion was based on the previous experience of ASSIST program leaders and TA implementers, other stakeholders, who had decades of experience in implementing similar health systems strengthening TA in multiple countries in different regions. Similar TA under normal circumstances typically involved setting up country offices, hiring local staff, external and internal consultants with startup activities taking much longer, often over a year incurring significantly higher costs before TA is ready to launch.

Finally, the report meets many of the Consolidated criteria for reporting qualitative research (COREQ) and with a small effort would help strengthen the paper.

Minor:

Page 8; The authors write the approach was based on the ASSIST “model for improvement” and “adaptive management”. This needs a reference.

Response: We added references.

Typo: Experience top of page 10 should be experienced.

Response: Corrected.