Evaluation of rapid telehealth implementation for prenatal and postpartum care visits during the COVID-19 pandemic in an academic clinic in Springfield, Massachusetts, United States of America.

BACKGROUND AND AIMS: COVID-19 forced healthcare systems to implement telehealth programs, facilitated in Massachusetts by a policy requiring insurers to reimburse for telehealth visits. Prior studies suggest that telehealth is effective for obstetric care, but little is known about its implementation in response to policy changes in underserved communities. We utilized the RE-AIM framework to evaluate telehealth implementation in a large academic urban obstetric practice that serves a medically underserved population.
METHODS: RE-AIM elements were assessed through retrospective review of electronic health record (EHR) data for all obstetric encounters between March 19 and August 31, 2020 and review of clinic implementation processes. Data extracted included demographics, number and type (in-person or telehealth) of prenatal visits, prenatal diagnoses, delivery outcomes, and number and type of postpartum visits. Data were analyzed using descriptive statistics.
RESULTS: A total of 558 patients (60.6% Hispanic; 13.2% primary language Spanish) had 1788 prenatal visits, of which 698 (39.0%) were telehealth visits. A total of 209 patients had 230 postpartum visits, of which 101 (48.3%) were telehealth visits. The Reach of the intervention increased from 0% of patients at baseline to 69% in August. Effectiveness measures were limited but suggested potential for earlier diagnosis of some prenatal conditions. Adoption was high, with all 30 providers using telehealth, and the telehealth was found to likely be feasible and acceptable based on uptake. Increases in the percentage of telehealth visits over time and continuation post-lockdown suggested maintenance was potentially achievable.
CONCLUSIONS: The COVID-19 pandemic has changed traditional approaches to healthcare delivery. We demonstrate that the use of the RE-AIM framework can be effective in facilitating implementation of telephone visits in a large academic urban obstetric practice after state-level policy change. This may be of particular importance in settings serving patients at higher risk for maternal morbidity and poor birth outcomes.

INTRODUCTION

At the onset of the COVID‐19 pandemic, obstetric practices in states that experienced major surges in COVID‐19 cases needed to act quickly to develop innovative approaches to delivering high‐quality obstetric care while minimizing the potential exposure of patients to the novel virus. Telehealth was quickly viewed as an important potential alternative to in‐person prenatal and postpartum services. Prior to and during the early stages of the COVID‐19 pandemic, delivering prenatal care via telehealth was found to be associated with comparable health outcomes and improved patient satisfaction and engagement when compared with in‐person care. , , , , Prior studies also suggest that care delivered via telehealth is associated with improved obstetric outcomes, especially concerning the surveillance and management of gestational hypertension, the reduction of smoking, as well as establishing higher rates of initiation and maintenance of breastfeeding in the postpartum period. Telehealth theoretically has the potential to address some inequities in perinatal care by improving access for patients who face barriers such as limited transportation or difficulty finding child care, but it also risks introducing new inequities related to differential access to technology. , , , Ensuring equitable access to telehealth has the potential to reduce healthcare inequities worsened by the COVID‐19 pandemic. Despite the potential benefits of having a telehealth option for eligible prenatal and postpartum visits, poor payer reimbursement, patient preferences, and general provider discomfort with telehealth limited the uptake of telehealth services in obstetrics before and during the early stages of the COVID‐19 pandemic. , ,

The first confirmed COVID‐19 case in Massachusetts occurred in Boston on February 1, 2020. As cases continued to increase throughout the commonwealth, a state of emergency was issued on March 10, and on March 17 emergency orders from the governor were issued limiting in‐person gatherings. System‐wide policy changes went into effect for healthcare settings as well, including but not limited to the expansion of coverage for telehealth visits by insurers beginning with visits dated on March 12, 2020 in addition to the mandate of payment parity between insurers. ,

Healthcare organizations in many, if not all, states have implemented some version of telehealth during the pandemic, and it appears likely to remain part of the healthcare delivery infrastructure post‐COVID. , Understanding and disseminating best practices for implementing and sustaining such programs for individuals in populations with disparate maternal mortality and morbidity outcomes is critical to optimizing the effectiveness of telehealth for prenatal/postpartum care. We used the RE‐AIM (reach, effectiveness, adoption, implementation, and maintenance) implementation framework , to evaluate the rapid implementation of telehealth services for prenatal and postpartum care during the COVID‐19 pandemic in a large urban academic practice. The RE‐AIM framework was selected because of its utility in evaluating multiple elements of implementation of an evidence‐based intervention such as telehealth.

METHODS

Study setting and population

This study evaluated the implementation of telehealth services for patients who received their prenatal or postpartum care at a large urban academic obstetrics and gynecology practice located in Springfield, Massachusetts in the United States, between March 19, 2020, and August 31, 2020. The clinic is located in a tertiary care hospital and provides care for both low‐ and high‐risk pregnancies; high‐risk pregnancies are co‐managed by maternal‐fetal medicine specialists. The clinic serves a population comprised of 60% Latina/x identifying patients, the majority of whom had low incomes as evidenced by 89.9% being insured by Medicaid, a government‐funded health insurance plan available only to people below certain income levels. The clinic serves as the regional referral center for high‐risk obstetrical patients as well as lower‐risk patients for the surrounding communities for whom no referral is needed.

Description of implementation

Before the COVID‐19 pandemic, the clinic utilized traditional prenatal and postpartum care models, which included only in‐person visits at standard visit intervals. Patients began to be scheduled for either in‐person or telehealth appointments beginning on March 19, 2020 in response to the declaration of a state of emergency and the initiation of significant restrictions on in‐person visits to healthcare providers. Decisions about whether an appointment would be in‐person or via telehealth were made based on patient preference, medical complexity, and/or the need for ultrasound/bloodwork or other congruent in‐person services.

Implementation strategies included communication from clinic leadership about the introduction of telehealth, provider and clinic staff education about options for the provision of prenatal and postpartum care, and appropriate billing. Communication with providers and staff regarding telehealth implementation took place via e‐mail, memos, technical instruction regarding use of phone systems to contact patients, and mandatory virtual meetings. Video equipment was available if patients had the technology to support video telehealth visits, otherwise visits were conducted via audio‐only telephone calls. New note templates in the electronic medical record also were made for telehealth visits, including the necessary information to meet billing guidelines. Providers were trained to properly confirm patient identity prior by confirming patient's full name and date of birth before engaging in any conversations with protected personal health information.

Data collection

We evaluated implementation of the telehealth program and selected health outcomes and processes through retrospective chart review of all pre‐ and post‐natal care visits conducted between March 19, 2020, and August 31, 2020. These dates were chosen to coincide with the mandated lockdown restrictions in the state of Massachusetts that began on March 19, 2020 and began to be lifted at the end of May and throughout the month of June. Data were collected in July and August to evaluate the maintenance of telehealth as an option for prenatal and postpartum visits after state‐mandated restrictions had been significantly eased. The following data were extracted for each prenatal and postpartum encounter using a structured data collection instrument: patient age, race/ethnicity, visit type, and pregnancy, delivery, and prenatal care outcomes as relevant to the encounter using REDCap , , electronic data capture tools. REDCap (Research Electronic Data Capture) is a secure, web‐based application designed to support data capture for research studies.

Application of RE‐AIM framework:

Reach (the proportion of a target population that participated in the intervention). Reach was assessed based on the percentage of visits across the specified time interval for each visit type (telehealth and in‐person). Reach was further assessed based on population‐level demographic information (eg, age, primary language spoken, insurance type, race, gravida, parity, and singleton vs multiple gestations).

Effectiveness (the positive and negative outcomes of the intervention). Effectiveness was assessed by exploring various health outcomes in relation to the number of telehealth visits a patient had. Prenatal outcomes included hypertensive diseases of pregnancy (including gestational age at diagnosis), presence of pre‐existing mental health disorders, and gestational diabetes (including gestational age at the time of diagnosis). Postpartum outcomes included gestational age at delivery, induction of labor, mode of delivery (induction vs spontaneous vaginal delivery, operative [vacuum or forcep assisted] vaginal delivery or cesarean section), live birth vs intrauterine fetal demise (IUFD), the percentage of patients who received a prescription for any contraceptive methods prior to 8 weeks postpartum and for long‐acting reversible contraception (LARC), exclusive breastfeeding rate, Edinburgh postnatal depression scale (EPDS) scores, number of postpartum presentations to the emergency department or triage unit, and the percentage of patients who completed a 2‐hour glucose tolerance test before 8 weeks postpartum among patients meeting criteria for gestational diabetes.

Adoption (a measure of the temporal pattern of implementation or the proportion of settings implementing the intervention). Adoption was measured as the percentage of total telehealth visits compared with the total number of all visits over time during the study interval of March 19 to August 31, 2020. The percentage of visits conducted in‐person or via telehealth were classified by gestational age ranges (<21 weeks, 22‐25 weeks, 26‐29 weeks, 30‐34 weeks, 35‐36 weeks, and >37 weeks) and for postpartum visits conducted between 1 and 28 days postpartum (“2 week visit”) and 29 to 56 days postpartum (“6 week visit”) and was recorded to assess the changes in telehealth visit rates over time. These time intervals were chosen based off of the traditional prenatal and postpartum care model in the United States.

Implementation (the extent to which the intervention was delivered as intended). Implementation outcomes included the acceptability and feasibility of this intervention. Acceptability was measured indirectly by comparing the number of telehealth visits over time, as the decision to proceed with telehealth at subsequent appointments was jointly made by the provider and patient at the preceding appointment. Feasibility was measured by the total initial uptake of telehealth services in the initial months of this intervention (March and April), as telehealth was not a service previously offered in this clinical setting.

Maintenance (the sustainability of the intervention). Maintenance was measured as the total number of visits conducted via telehealth after the initial lockdown restrictions were eased at the end of June, and is displayed as the number of telehealth visits in the subsequent months of July and August.

Statistical analysis

Data were analyzed using descriptive statistics, including means and SDs for normally distributed continuous variables or medians and percentiles for skewed continuous variables. Frequencies and percentages were utilized for describing categorical data. Graphical approaches were also used to illustrate the RE‐AIM constructs. This study was determined to be exempt by the study site's Institutional Review Board (Reference: 1535131‐2). This manuscript was created within the framework of the SQUIRE (Standards for Quality Improvement Reporting Excellence) and StaRI (Standards for Reporting Implementation Studies) guidelines. ,

RESULTS

A total of 558 prenatal patients and 209 postpartum patients who had 2028 outpatient visits were analyzed. Across both types of patient visits, mean age was 27 years, 60.2% identified as Hispanic, with 13.2% having a primary language of Spanish. The majority of patients were insured through Massachusetts's Medicaid program (89.0%). Additional demographic information is presented in Table 1. Delivery data were available for 110 of the prenatal patients during the study interval (Table 2). The results below are organized by RE‐AIM construct and are summaritvely organized in Table 3.

TABLE 1

Characteristics of patients with pre‐ and post‐natal encounters (March 19‐August 31, 2020)

	Prenatal	Postpartum
	(N = 558)	(N = 209)
Age at first contact
Mean (SD)	27.53 (5.84)	27.19 (5.71)
Gravida
Median (iqi)	3.0 (2.0, 4.0)	2.0 (1.0, 4.0)
Parity
Median (iqi)	1.0 (0.0, 2.0)	1.0 (1.0, 2.0)
Nulliparous
Multiparous	396 (71.0%)	168 (80.4%)
Nulliparous	162 (29.0%)	41 (19.6%)
Plurality
Singleton	548 (98.2%)	206 (98.6%)
Twins	10 (1.8%)	3 (1.4%)
Primary language spoken
English	457 (81.9%)	179 (85.6%)
Spanish	77 (13.8%)	25 (12.0%)
Other	24 (4.3%)	5 (2.4%)
Race
Asian/Pacific Islander	17 (3.0%)	5 (2.4%)
Black/African‐American	104 (18.6%)	45 (21.5%)
Native American	3 (0.5%)	0 (0.0%)
White/Caucasian	297 (53.2%)	101 (48.3%)
Unknown	137 (24.6%)	58 (27.8%)
Hispanic
No	223 (40.0%)	78 (37.3%)
Yes	334 (59.9%)	131 (62.7%)
Missing	1 (0.2%)	0 (0.0%)
Insurance
ACO	384 (68.8%)	143 (68.4%)
MCO	46 (8.2%)	20 (9.6%)
MassHealth	72 (12.9%)	25 (12.0%)
Private	44 (7.9%)	17 (8.1%)
Other	12 (2.2%)	4 (1.9%)
Prenatal in‐person vs telehealth visits
All in‐person	142 (25.4%)	‐
All telehealth	64 (11.5%)	‐
Mixed	352 (63.1%)	‐
Postpartum visit type
In‐person	‐	108 (51.7%)
Telehealth	‐	101 (48.3%)

Abbreviations: ACO, accountable care organization; iqi, inter‐quartile range; MCO, Managed care organization.

TABLE 2

Delivery outcomes by number of prenatal telehealth visits

	Total	All in‐person	At least one or more telehealth visit
	(N = 110)	(N = 35)	(N = 75)
Induction of labor
No	74 (67.3%)	25 (71.4%)	49 (65.3%)
Yes	36 (32.7%)	10 (28.6%)	26 (34.7%)
Delivery type
Cesarean section	31 (28.2%)	8 (22.9%)	23 (30.7%)
Vaginal delivery	76 (69.1%)	27 (77.1%)	49 (65.3%)
Forcep/vacuum delivery	3 (2.7%)	0 (0.0%)	3 (4.0%)

TABLE 3

Summary of results based on RE‐AIM indicators

RE‐AIM indicator	Definition	Measure	Figure/Table
Reach	Proportion of a target population that participated in the intervention.	Demographic breakdown of population characteristics and proportion of telehealth visits compared to total visits before and after telehealth implementation.	Table 1 and Figure 2
Effectiveness	Positive and negative outcomes of the intervention.	Comparison of various pregnancy related outcomes in relation to the number of telehealth visits.	Tables 2, 4 and 5
Adoption	Measure of the temporal pattern of implementation.	Percentage of telehealth visits compared to total visits on a month by month basis throughout the study time period.	Figure 1
Implementation (acceptability and feasibility)	Extent to which the intervention was delivered as intended.	Indirect comparison of the number of telehealth visits over time, with particular focus on interval after state‐mandated lockdowns began to lift (May and June), as well as the breakdown of percentage of telehealth vs total visits broken down by gestational age to assess for whether certain prenatal/postpartum care intervals are more acceptable than others.	Figures 1 and 2
		Total initial uptake of telehealth services in the initial 2 months of the intervention (March and April).	Figure 2
Maintenance	Sustainability of the intervention.	Comparison of the number of telehealth visits over time compared with the number of total visits.	Figure 2

Reach: Of the 2018 total prenatal and postpartum visits conducted during the study period, 698 visits (34.4%) were conducted via telehealth. Looking exclusively at postpartum visits, of the 209 patients who had at least one postpartum visit, 108 (51.7%) had only in‐person visits and 101 (48.3%) had at least one telehealth visit. A total of 8 of the 698 total telehealth visits occurred via video conferencing, with the remaining occurring via telephone only.

Effectiveness: The overall prevalence of hypertensive disease of pregnancy and gestational diabetes mellitus and estimated gestational age at the time of diagnosis by visit type is shown in Table 4. Findings included that the median gestational age at diagnosis of hypertensive disease of pregnancy was approximately 37 weeks among those receiving prenatal care via any amount of telehealth, compared with approximately 35 weeks among those receiving all of their prenatal care in person. The median gestational age at diagnosis of GDM was 31.8 weeks among patients who had only in‐person prenatal visits (n = 142), compared with 27.7 weeks among patients who had at least one telehealth prenatal visit (n = 416). Of note, blood pressure and blood sugar monitoring occurred via patient self‐report during telephone visits of self‐collected and recorded values.

TABLE 4

Hypertension and gestational diabetes (GDM) diagnoses among prenatal patients by number of prenatal telehealth visits

	Total	All in‐person	At least one or more telehealth visit
	(N = 558)	(N = 142)	(N = 416)
Hypertension
Number of patients diagnosed	22 (3.9%)	8 (5.6%)	14 (3.4%)
Median EGA at diagnosis (range a )	36.8 (9.1, 39.6)	35.3 (15.7, 39.6)	37.0 (9.1, 39.3)
GDM
Patients diagnosed	56 (10.0%)	14 (9.9%)	42 (10.1%)
Median EGA at diagnosis (range a )	29.6 (9.0, 37.1)	31.8 (11.7, 36.1)	27.6 (9.0, 37.1)

Abbreviation: EGA, estimated gestational age.

Range = minimum, maximum.

In terms of intrapartum results, of the patients who had all in‐person prenatal visits, 28.6% (10/35) underwent induction of labor compared with 34.7% (26/75) who had at least one in‐person telehealth visit (Table 2). Vaginal delivery rates of 77.1% (27/35) among patients who had all in‐person prenatal visits compared with 66.7% (49/75) among patients who had at least one telehealth prenatal visit.

Finally, postpartum outcomes by visit type are shown in Table 5. These outcomes included patient‐reported contraception use at 6 weeks postpartum of 79.6% among patients who presented for only in‐person postpartum care visits compared with 71.3% of patients who had at least one postpartum telehealth visit. Patient‐reported breastfeeding rates at 6 weeks were 25.0% and 23.8%, respectively, and OB triage/ED visit rates were 13.0% compared with 7.9%.

TABLE 5

Postpartum outcomes by in‐person vs telehealth postpartum visit

	Total	In‐person	Telehealth
	(N = 209)	(N = 108)	(N = 101)
Hypertensive disease of pregnancy
Yes	28 (13.4%)	17 (15.7%)	11 (10.9%)
No, never met criteria	146 (69.9%)	66 (61.1%)	80 (79.2%)
No, was diagnosed before delivery	35 (16.7%)	25 (23.1%)	10 (9.9%)
Days from delivery to HTN Diagnosis weeks (n = 28)
Median (range a )	1 (0, 6)	1 (0, 6)	1 (0, 2)
At risk for postpartum depression
EPDS < 10	105 (50.2%)	77 (71.3%)	28 (27.7%)
EPDS ≥ 10	27 (12.9%)	20 (18.5%)	7 (6.9%)
Missing	77 (36.8%)	11 (10.2%)	66 (65.3%)
Any contraception prescribed
No	51 (24.4%)	22 (20.4%)	29 (28.7%)
Yes	158 (75.6%)	86 (79.6%)	72 (71.3%)
LARC placed (n = 157)
In‐hospital placement	38 (24.1%)	26 (30.2%)	12 (16.7%)
Postpartum placement	28 (17.7%)	19 (22.1%)	9 (12.5%)
Neither	92 (58.2%)	41 (47.7%)	51 (70.8%)
Breastfeeding (at 6 weeks postpartum)
No	158 (75.6%)	81 (75.0%)	77 (76.2%)
Yes	51 (24.4%)	27 (25.0%)	24 (23.8%)
2 h GTT (by 8 weeks postpartum)
Not indicated	189 (90.4%)	96 (88.9%)	93 (92.1%)
No	18 (8.6%)	10 (9.3%)	8 (7.9%)
Yes	2 (1.0%)	2 (1.8%)	0 (0.0%)
Any triage/ED visits
No	187 (89.5%)	94 (87.0%)	93 (92.1%)
Yes	22 (10.5%)	14 (13.0%)	8 (7.9%)

Abbreviations: ED, emergency department; GTT, glucose tolerance test; HTN, hypertension; LARC, long‐acting removable contraception; PEDS, Edinburg postnatal depression scale.

Range = minimum, maximum.

Adoption: Every provider (n = 30) participated in the initiation and maintenance of the provision of telehealth services as evidenced by universal uptake and continued participation throughout the study period of all providers in the provision of telehealth services. Additionally, Figure 1 shows the percentage of telehealth and in‐person visits by gestational age range. Two time points were noted to have a larger percentage of in‐person visits: 26 + 0 weeks to 29 + 6 weeks and 35 + 0 weeks to 36 + 6 weeks. The majority of visits between 22 + 0 and 25 + 6 weeks were via telehealth (57.6%), and the majority of visits were in‐person for all other gestational age windows. Approximately half of the patients had a telehealth postpartum visit regardless of the number of postpartum visits they had: one visit at approximately 2 weeks postpartum (53.3%), one visit at approximately 6 weeks postpartum (47.4%), or visits at both 2 and 6 weeks postpartum (52.4%). The extremely low rate of uptake of video telehealth visits was cited by providers to be due to difficulty in both providers and patients use of technology.

FIGURE 1

Number and percentage of telehealth visit vs in‐person visit by estimated weeks of gestation and number of weeks postpartum

Implementation: Figure 2 reveals an increase in telehealth visits over time from a baseline of zero telehealth visits to 83 telehealth visits over the last 12 days of March (37% of all visits during those 12 days in March). By the end of the study period, over 300 prenatal and postpartum telehealth visits were conducted in August (69% of all visits during August) entry, resulting in subsequent near‐universal documentation of EPDS scores in the EHR.

FIGURE 2

Total number of telehealth and in‐person prenatal outpatient appointments during the pandemic in 2020

Finally, as demonstrated in Figure 1, visits surrounding 28 weeks gestational age and 36 weeks gestational had lower proportions of telehealth visits (30.1% and 23.2% respectively) compared with other visit intervals that had higher proportions of telehealth usage ranging from 34.3% to 57.4%.

An unintended benefit of evaluating the implementation of telehealth visits in the study clinic was that it detected issues with quality of care that existed before the pandemic, including failure to provide all routine prenatal and postpartum care as intended. In the course of evaluating the implementation of telehealth in this obstetric clinic, we discovered that 36.8% of patients did not have EPDS screening documented at their postpartum visits. A root cause analysis revealed a high number of missing EPDS screens and determined that providers believed the medical assistants (MA) were entering this information into the EHR, and the MAs believed the providers were entering this information into the EHR. Adjustments were made to the MAs' workflow to standardize EPDS data.

Maintenance: Figure 2 reveals total in‐person and telehealth visit data from the months of July and August. The targeted intervention ended in June when the state of Massachusetts began to ease restrictions, and more healthcare services began to be regularly offered in person. The continued use of telehealth visits at this point is demonstrated by the even higher number of telehealth visits, 245 visits in July (46% of all visits) and 305 visits in August (69% of all visits).

DISCUSSION

The results of this study suggest that rapid implementation of telehealth visits for prenatal and postpartum care is feasible in a practice serving a low‐income, majority Latina/x patient population using a parsimonious set of implementation strategies. Measures of reach and adoption suggested that uptake of the telehealth intervention was ubiquitous in the practice and that implementation outcomes measured suggested acceptability and appropriateness of telehealth. Although additional evaluation post‐COVID will generate a better understanding of maintenance of telehealth, this study's early assessment suggested that maintenance of telehealth visits once in‐person restrictions were lifted in the state of Massachusetts was achievable and feasible.

The risks associated with in‐person medical encounters due to the COVID‐19 pandemic created an unprecedented need for rapid innovations in healthcare delivery. The pandemic has also illuminated and exacerbated long‐standing racial and ethnic inequities in health and healthcare. Telehealth was technologically feasible before the pandemic, but the need to limit in‐person visits coupled with state‐level policies to facilitate rapid adoption of telehealth led to a dramatic increase in remote healthcare delivery. Many medical specialties responded to the COVID‐19 pandemic by implementing telehealth services. , Given widespread patient dissatisfaction with the pre‐pandemic model of prenatal care and recent calls by the American College of Obstetric and Gynecology to make substantial changes in the postpartum care, it may not be surprising that there has been rapid adoption of telehealth in obstetric care during the pandemic. , , , , , , The evidence base for the effectiveness and safety of telehealth in obstetric and gynecologic care made it suitable for widespread dissemination and implementation. This study addressed a gap in knowledge by using the RE‐AIM framework to evaluate the real‐world rapid implementation of telehealth services in a large, academic, urban obstetric practice that serves a lower‐income, racial/ethnic minority population.

The very low rate of video compared with telephone encounters for telehealth visits suggests that feasibility, accessibility, and acceptability may differ based on the mode of telehealth delivery. Although no formal evaluation of these visits was performed, staff and patients reported difficulty with setting up the video platform, which resulted in a rapid abandonment of this service. A growing body of literature suggests that telehealth may actually increase socioeconomic and racial/ethnic inequities in healthcare because of the greater difficulty affording devices needed to easily participate in video telehealth and higher likelihood of living in neighborhoods with lower accessibility to high‐quality internet services. , More studies are needed to assess and address user issues including accessibility of WiFi internet, confidence using digital technology, sufficient cellular plans, privacy, and interest in video visits.

The payment parity of telehealth services in Massachusetts during the pandemic also may have played a role in the ability to rapidly implement these services. Not only did the executive order issued by Governor Baker require payment parity reimbursement for in‐person and telehealth visits during the pandemic, but it also mandated private insurers reimburse at least at Medicaid levels for telehealth services. This order had significant impact on large, urban, academic centers, such as the center in which the current study took place, whose patient population is insured almost exclusively by Medicaid for prenatal and postpartum care. This order likely played an important role in limiting, but not fully preventing, the exacerbation of healthcare disparities that a vulnerable patient population already faces and that might otherwise have been worsened by the pandemic. As previous studies have demonstrated, it is doubtful that statewide policy changes alone cannot remove all of the barriers to telehealth use in general, and specifically in diverse populations. ,

This study's strengths include the large number of patients (n = 662), assessment of both pre‐ and postpartum visits, use of detailed EHR data, and its location in a state with one of the earliest major surges of COVID‐19 cases in the country. Although the study might have benefitted from an experimental design that allowed for causal inference, it was not possible to do so given the rapid changes required by the pandemic surge and the stresses on the healthcare system at the time. Alternatively, this study illuminates implementation considerations in a real‐world scenario. Future studies that compare implementation strategies in different contexts will extend the knowledge gained from the current study. Estimations of cost or cost‐effectiveness will also be important to understand the long‐term financial impacts of telehealth and its sustainability should it continue to be used for a substantial portion of prenatal and postpartum visits. The study was undertaken at a time when healthcare organizations were in crisis in Massachusetts and healthcare providers under substantial duress, so formal surveys and interviews to evaluate provider satisfaction with telehealth were not conducted but would be important in future studies as well. Previous studies have reported high patient satisfaction with the implementation of telehealth in obstetrics care, and given that patients had an active role in deciding whether or not they had in‐person or telehealth visits in this study, suggesting that patient satisfaction might be high in this setting. , Ascertaining patient's satisfaction with prenatal and postpartum telehealth visits among patients in lower‐income and racial/ethnic populations will be important to be able to tailor services to patient's needs and better understand the role telehealth may play in addressing inequities in maternal health care and disparities in maternal morbidity and mortality.

The RE‐AIM framework was a useful tool for evaluating implementation and maintenance of a new intervention in the obstetric setting and should be considered for use more broadly. Given the extensive racial and ethnic disparities in obstetric care, future studies should determine the impact of telehealth specifically on these disparities. If telehealth is found to be associated with improved outcomes, it will also be important to identify the mechanisms through which it has an effect, such as removal of barriers to access to care (ie, transportation) or increased self‐efficacy through joint decision‐making.

While prior studies have addressed the initial implementation phases of telehealth during the COVID‐19 pandemic, we employed a theory‐based framework to evaluate the implementation and maintenance of telehealth in response to COVID‐19 associated policy changes in an urban, low‐income, majority Latina/x patient population. This study focused primarily on practice and organizational factors influencing implementation and maintenance, but the effect of public policy on equity in obstetric care should continue to be evaluated in systematic ways to better inform policymakers and healthcare providers about policy impacts.

CONFLICT OF INTEREST

The authors declare that they have no competing interests.

AUTHOR CONTRIBUTIONS

Data Entry: Keith Reisinger‐Kindle, Shiva Niakan, Colby Cayton

Data Analysis/Table Creation: Alexander Knee

Senior Oversight: Neena Qasba, Sarah L. Goff

All authors read and approved the final manuscript. All authors contributed to manuscript writing. All authors have read and approved the final version of the manuscript.

Keith Reisinger‐Kindle had full access to all of the data in this study and takes complete responsibility for the integrity of the data and the accuracy of the data analysis.

TRANSPARENCY STATEMENT

The primary author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

University of Massachusetts‐Baystate's IRB granted an exemption after review of the protocol. Reference: 1535131‐2.