Assessing provider performance of intrapartum care using simulated encounters and clinical vignettes: A comparison study from Tanzania.

OBJECTIVE: To compare clinical vignettes and objective structured clinical examinations (OSCE) as methods for assessing the quality of intrapartum care among skilled providers in rural primary-level health facilities in Tanzania.
METHODS: Cross-sectional study conducted at six health facilities in the Simiyu region of Tanzania. Providers were assessed using OSCE and clinical vignettes in spontaneous delivery, neonatal resuscitation, and management of postpartum hemorrhage. Trained researchers used a structured clinical checklist. The frequencies of items are presented as percentages and the agreement of the methods of assessment are reported using kappa statistics (high: kappa > 0.80, moderate: kappa = 0.60-0.80, low: kappa < 0.60).
RESULTS: Most healthcare providers were female (60.7%), registered nurses by training (29.0%), and worked in a dispensary (56.1%), with an average age of 33 years and an average of 7.4 years of experience in their respective professions. Five items had high agreement between OSCE and clinical vignettes: postpartum vital signs every 15 min, oxytocin within 1 min of birth, diagnosis of postpartum hemorrhage, elevating legs of the mother, and deciding on manual compression of the uterus.
CONCLUSION: OSCE and clinical vignettes should be viewed as complimentary to one another in the assessment of provider knowledge and skill, with priority given to OSCE, particularly in intrapartum care.

INTRODUCTION

Achieving quality of care has been a longstanding healthcare goal of the United Nations and WHO as part of the Sustainable Development Goals. To measure the performance of healthcare services, researchers perform quality assessments to identify gaps in provider knowledge and skill. Over the years, efforts have focused on improving access to healthcare services, particularly in maternal and child health services in low‐ and middle‐income countries (LMICs). Studies have shown that increased coverage of maternal health services alone, including deliveries in health facilities, is not successful in reducing neonatal or maternal mortality. , , , Greater emphasis is being placed on the quality and delivery of these services and their contribution to reducing maternal and child mortality. ,

Many efforts in the global health community have focused on validating measures of quality assessment that are sustainable and feasible for developing healthcare systems. Many methods for quality assessment exist, including direct observation of care provision, clinical vignettes, and objective structured clinical examinations (OSCEs). Although direct observation is considered the gold standard when assessing the knowledge and skill of health providers, it is expensive, time intensive, and may not always be feasible to implement. ,

An OSCE is an effective alternative to direct observation that primarily assesses provider skill. During an OSCE, a patient scenario is simulated by a trained actor, interviewer, or mannequin, and the provider's performance is evaluated. Clinical vignettes traditionally assess health provider knowledge and clinical decision making using written standardized clinical scenarios. Clinical vignettes are more cost‐effective than direct observation and OSCEs. , Clinical vignettes have been reported as a valid method for quality assessment in LMICs, although not specifically in intrapartum care settings. Studies have shown that clinical vignettes have predicted provider competence and performance. , , However, some evidence suggests that providers may not follow the exact course of action in real life practice that they verbalize in a similar hypothetical scenario. , ,

Amref‐Tanzania, in partnership with Amref Health Africa, is an African‐based non‐profit organization that manages, facilitates, and sponsors programs focused on African health issues. They implemented the Uzazi Uzima II project in Tanzania's Simiyu region, with a goal of reducing maternal and newborn mortality and morbidity through expanding basic and comprehensive emergency newborn care services. We developed and tested tools to evaluate the quality of maternal and child health care as part of this project. In the present study, we compared skills and knowledge measured by OSCEs and clinical vignettes to determine whether clinical vignettes could replace OSCEs as a more practical method of evaluating the quality of intrapartum care in low‐ and middle‐resource settings.

MATERIALS AND METHODS

Study design and sample

A cross‐sectional study was conducted to evaluate the quality of intrapartum care provided by clinicians trained by the Uzazi Uzima II project. Clinicians were defined as healthcare workers that are government appointed and provide direct patient care in the intrapartum setting, including medical doctors, clinical officers, registered nurses, nursing officers, enrolled nurses, assistant medical officers, nursing assistants, and medical assistants.

The study was conducted in four health centers and 20 dispensaries in the Simiyu region of Tanzania between November and December 2019. Dispensaries are staffed by a clinical officer or nurse and are expected to provide basic emergency obstetric care. Health centers are higher level of care facilities staffed by physicians and are expected to provide basic and comprehensive emergency obstetric care. For this sub‐study, we focus on the data collected through the OSCEs and clinical vignettes.

Assessment procedures

All 77 clinicians who provided intrapartum care at the 24 facilities were eligible for inclusion and elected to participate in the study. All providers were administered the OSCE and clinical vignette on the same day. Clinicians were randomly assigned to be assessed via OSCE or clinical vignette first using a random number function in Microsoft Excel. The randomization was carried out for each facility such that equal numbers of providers were assessed by OSCE or clinical vignette first.

For OSCEs, health providers were asked to demonstrate their clinical reasoning and or perform specific skills within three clinical scenarios using MamaNatalie simulators. For clinical vignettes, the same providers were asked to verbalize their clinical reasoning or identify interventions they would perform if indicated within the three same clinical scenarios.

The clinical scenarios included: (1) management of a normal vaginal delivery with postnatal care; (2) management of a woman with postpartum hemorrhage; (3) management of a newborn with asphyxia. The clinical scenarios were developed based on the training materials developed by the Tanzania Ministry of Health for training health providers to provide basic emergency newborn care services as part of the Uzazi Uzima project.

Data collection and management

Trained researchers recorded the actions performed or verbalized by the respondents using a checklist for each scenario. These checklists were developed from training protocols used as part of the project, with generated tools and more detailed descriptions available online. , Tools were pretested and then piloted with providers who were not part of the sample before use in Simiyu.

All data collectors were trained medical personnel (medical officers, nurse officers, nurses, nurse‐midwives, and/or clinical officers) who had previous experience in data collection and had completed a weeklong training on the survey tools, protocols, and methods. Data were recorded on paper and entered in an electronic data capture system, Open Data Kit (ODK), using Android tablets (ODK Collect). Data were uploaded daily into a password‐protected server hosted at the Ifakara Health Institute.

Analysis

The actions listed on the checklist were categorized as performed or not performed and are presented as proportions. To assess the agreement between clinical vignettes and OSCEs, we calculated the kappa statistic at the provider level. We used thresholds of less than 0.6, between 0.6 and 0.8, and more than 0.8 for the kappa to be considered as low, moderate, or high, respectively. A high kappa value indicated that the action was captured to the same extent by clinical vignettes and OSCEs. As Cohen's kappa is underestimated and overestimated whenever the prevalence or bias is high (margins are unbalanced), we adjusted the kappa for prevalence and bias using the methods of Brennan and Prediger. Brennan and Prediger‐adjusted kappas are presented with 95% confidence intervals.

Ethical considerations

The study was reviewed by institutional ethics review boards of the participating institutions. Johns Hopkins Bloomberg School of Public Health and Ifakara Health Institute both gave ethical clearances. The Tanzanian National Institute for Medical Research Institutional Ethics Committee also reviewed the protocol and gave a favorable opinion. The other participating institution, Amref Tanzania, did not officially review the protocol, but gave appropriate permission based on their institutional guidelines. Providers were included in the study only if they provided written informed consent.

RESULTS

Demographics

In total, 77 providers participated, with the majority being 30–40 years old (42.5%). Most participating providers were female (61.3%) and nearly a third were registered nurses by training (28.8%). Most participating providers worked in a dispensary (56.3%) and had 3–10 years of work experience in their respective professions (62.5%) (Table 1).

TABLE 1

Demographics of participating intrapartum healthcare providers in objective structured clinical examination and clinical vignette assessments in three different clinical scenarios in Tanzania health facilities (n = 77)

Variable	n (%)
Female gender	49 (61.3)
Age (years)
<30	33 (41.3)
30–40	34 (42.5)
>40	13 (16.3)
Marital status
Single/widow	22 (27.5)
Married/in union	58 (72.5)
Education level
Certificate	44 (55.0)
Diploma and above	36 (45.0)
Professional
Medical doctor	1 (1.3)
Clinical officer	15 (18.8)
Registered nurse, nursing officer	23 (28.8)
Enrolled nurse	21 (26.3)
Assistant medical officer	1 (1.3)
Nursing assistant/medical attendant	14 (17.5)
Clinical assistant	5 (6.3)
Work experience (years)
<3	19 (23.8)
3–10	50 (62.5)
>10	11 (13.8)
District
Bariadi DC	8 (10.0)
Bariadi TC	5 (6.3)
Busega	24 (30.0)
Itilima	13 (16.3)
Maswa	14 (17.5)
Meatu	16 (20.0)
Type of health facility
Dispensary	45 (56.3)
Health center	35 (43.7)

Infection control

On both OSCEs and clinical vignettes, most providers performed or reported washing hands with soap and water (72.7% vs. 67.5%, kappa = 0.55), putting on personal protective equipment (75.3% vs. 61.0%, kappa = 0.45), and putting on sterile gloves prior to performing a delivery (77.9% vs. 63.6%, kappa = 0.45) in OSCEs and clinical vignettes, but with low agreement for all (Table 2).

TABLE 2

Comparison of actions demonstrated on objective structured clinical examination (OSCE) or verbalized in clinical vignettes (CV) for infection control, vital signs, management of second stage labor, management of third stage labor, neonatal resuscitation, and postpartum hemorrhage (n = 77)

	n (%) OSCE	95% CI	n (%) CV	95% CI	Kappa	95% CI
Infection control
Washes hands with soap and water	56 (72.7)	[64.5, 79.6]	51 (67.5)	[59.9, 74.3]	0.55	[0.33, 0.72]
Puts on personal protective barriers	58 (75.3)	[62.9, 84.6]	47 (61.0)	[49.5, 71.5]	0.45	[0.24, 0.65]
Puts on sterile gloves	60 (77.9)	[68.6, 85.1]	49 (63.6)	[55.1, 71.4]	0.45	[0.24, 0.65]
Vital signs
Intrapartum
Blood pressure	63 (81.8)	[70.7, 89.3]	57 (74.0)	[61.6, 83.5]	0.66	[0.49, 0.83]
Respiratory rate	48 (62.3)	[48.6, 74.4]	46 (59.7)	[49.0, 69.7]	0.53	[0.33, 0.72]
Postpartum
Blood pressure	60 (77.9)	[67.0, 86.0]	57 (74.0)	[64.6, 81.6]	0.63	[0.45, 0.81]
Pulse	50 (64.9)	[57.4, 71.8]	48 (62.3)	[51.1, 72.4]	0.63	[0.45, 0.81]
Assess for uterine tone	34 (44.2)	[31.6, 57.5]	31 (40.3)	[31.4, 49.8]	0.50	[0.30, 0.70]
Monitor every 15 min postpartum	22 (28.6)	[17.3, 43.4]	21 (27.3)	[15.4, 43.5]	0.84	[0.72, 0.97]
Second stage labor
Control the birth of the head	62 (80.5)	[62.6, 91.1]	56 (72.7)	[54.7, 85.5]	0.63	[0.63, 0.45]
Wipe the eyes and mouth of baby	52 (67.5)	[58.2, 75.6]	41 (53.2)	[44.8, 61.5]	0.61	[0.61, 0.41]
Check for nuchal cord	25 (32.5)	[24.4, 41.7]	19 (24.7)	[17.0, 34.4]	0.58	[0.58, 0.39]
Allows baby's head to turn spontaneously	67 (87.0)	[73.8, 94.1]	59 (76.6)	[59.8, 87.9]	0.63	[0.63, 0.45]
Guidance of head and posterior shoulder	66 (85.7)	[72.4, 93.2]	47 (61.0)	[45.8, 74.4]	0.37	[0.37, 0.16]
Support baby's body as it exits vaginal canal	72 (93.5)	[81.2, 98.0]	64 (83.1)	[72.8, 90.0]	0.66	[0.66, 0.49]
Third stage labor
Apply counter traction in upward direction	65 (84.4)	[74.1, 91.1]	62 (80.5)	[65.4, 90.0]	0.50	[0.30, 0.70]
Hold firm and steady tension with cord	61 (79.2)	[61.5, 90.1]	57 (74.0)	[57.2, 85.9]	0.68	[0.52, 0.85]
Deliver placenta slowly with both hands, turning the placenta	58 (75.3)	[64.3, 83.8]	49 (63.6)	[52.2, 73.7]	0.61	[0.42, 0.79]
Give mother oxytocin intramuscularly within 1 min of birth	72 (93.5)	[84.5, 97.4]	69 (89.6)	[78.1, 95.4]	0.89	[0.79, 1.00]
Massage uterus until firm after placenta delivers	56 (72.7)	[59.8, 82.7]	44 (57.1)	[45.2, 68.3]	0.37	[0.16, 0.58]
Examine the vulva, perineum, and vagina for laceration/tears	66 (85.7)	[76.3, 91.8]	55 (71.4)	[60.8, 80.1]	0.66	[0.49, 0.83]
Neonatal resuscitation
Check Apgar score	68 (88.3)	[73.6, 95.3]	68 (88.3)	[72.7, 95.5]	0.50	[0.30, 0.70]
Place baby on back and clean, warm surface	50 (64.9)	[51.4, 76.4]	35 (45.5)	[34.7, 56.6]	0.42	[0.22, 0.63]
Provider indicates neonatal resuscitation protocol	65 (84.4)	[67.1, 93.5]	55 (71.4)	[54.6, 83.9]	0.68	[0.52, 0.85]
Extend neck and open airway	46 (59.7)	[41.2, 75.9]	40 (51.9)	[39.4, 64.2]	0.55	[0.36, 0.74]
Suction mouth	63 (81.8)	[70.6, 89.4]	64 (83.1)	[68.7, 91.7]	0.66	[0.49, 0.83]
Placement of bag valve mask	71 (92.2)	[84.0, 96.4]	67 (87.0)	[78.2, 92.6]	0.71	[0.55, 0.87]
Check seal by ventilating 2–3 times	46 (59.7)	[48.0, 70.5]	43 (55.8)	[44.6, 66.5]	0.53	[0.33, 0.72]
Ventilate at 40 breaths/min for 1 min	22 (28.6)	[19.5, 39.7]	29 (37.7)	[23.3, 54.6]	0.58	[0.39, 0.76]
Assess for spontaneous breathing	46 (59.7)	[45.6, 72.5]	43 (55.8)	[43.2, 67.8]	0.50	[0.30, 0.70]
Check pulse/heartrate	34 (44.2)	[34.4, 54.4]	30 (39.0)	[30.2, 48.5]	0.34	[0.13, 0.56]
Postpartum hemorrhage
Provider makes correct diagnosis of postpartum hemorrhage	64 (83.1)	[70.9, 90.9]	64 (83.1)	[71.5, 90.6]	0.88	[0.78, 0.98]
Elevate legs of woman by raising foot of bed	3 (3.9)	[1.4, 10.2]	1 (1.3)	[0.1, 10.4]	0.95	[0.87, 1.00]
Draw blood for hemoglobin and cross‐matching	19 (24.7)	[15.6, 36.7]	30 (39.0)	[26.7, 52.8]	0.66	[0.49, 0.83]
Decide on bimanual compression of uterus	32 (41.6)	[27.7, 57.0]	30 (39.0)	[27.3, 52.1]	0.92	[0.83, 1.02]
Give second dose of oxytocin	56 (72.7)	[62.5, 81.0]	55 (71.4)	[59.3, 81.1]	0.76	[0.61, 0.91]

Abbreviation: CI, confidence interval.

Vital signs

Most providers measured or reported measuring the blood pressure of the mother during OSCEs and clinical vignettes with moderate agreement (81.8% vs. 74.0%, kappa = 0.66). Most providers assessed respiratory rate during intrapartum care in OSCEs and clinical vignettes with low agreement (62.3% vs. 59.7%, kappa = 0.53). During postpartum care, most providers measured blood pressure in OSCEs and clinical vignettes with moderate agreement (77.9% vs. 74.0%, kappa = 0.63). Most providers assessed pulse in OSCEs and clinical vignettes during postpartum care with moderate agreement (64.9% vs. 62.3%, kappa = 0.63) (Table 2).

Second stage labor

Most providers either performed in OSCEs or verbalized in clinical vignettes, actions associated with the management of second stage labor (Table 2). However, less than a third of providers checked for a nuchal cord in OSCEs and clinical vignettes with moderate agreement (32.5% vs. 24.7%, kappa = 0.58). A higher proportion of providers demonstrated controlling the birth of the head in OSCEs than in clinical vignettes with moderate agreement (80.5% vs. 72.7%, kappa = 0.63). More providers demonstrated wiping the eyes and mouth of the baby (67.5% vs. 53.2%, kappa = 0.61), allowing the baby's head to turn spontaneously (87.0% vs. 76.6%, kappa = 0.63), and supporting the baby's head as it exits the vaginal canal (93.5% vs. 83.1%, kappa = 0.66) in OSCEs than in clinical vignettes with moderate agreement. More providers provided guidance of the head and posterior shoulder in OSCEs than in clinical vignettes with low agreement (85.7% vs. 61.0%, kappa = 0.37).

Third stage labor

More providers performed counter traction of the placenta with low agreement (84.4% vs. 80.5%, kappa = 0.50), holding tension with the umbilical cord with moderate agreement (79.2% vs. 74.0%, kappa = 0.68), delivering the placenta slowly with both hands/turning the placenta with moderate agreement (75.3% vs. 63.6%, kappa = 0.61), giving the mother oxytocin intramuscularly within 1 min of birth with high agreement (93.5% vs. 89.6%, kappa = 0.89), and examining the vulva/perineum/vagina for lacerations with moderate agreement (85.7% vs. 71.4%, kappa = 0.66) in OSCEs than verbalized in clinical vignettes. Agreement was low for massaging the uterus until firm after the placenta was delivered, but more providers performed this skill in OSCEs compared with clinical vignettes (72.7% vs. 57.1%, kappa = 0.37) (Table 2).

Neonatal resuscitation

Most providers checked or verbalized checking the Apgar score in both OSCEs and clinical vignettes, but with low agreement (88.3% vs. 88.3%, kappa = 0.50). More providers performed placing the baby on its back with low agreement (64.9% vs. 45.5%, kappa = 0.42), indicating the neonatal resuscitation protocol with moderate agreement (84.4% vs. 71.4%, kappa = 0.68), extending the neck and opening the airway with low agreement (59.7% vs. 51.9%, kappa = 0.55), proper placement of the Ambu bag and mask with moderate agreement (92.2% vs. 87.0%, kappa = 0.71), checking the seal with low agreement (59.7% vs. 55.8%, kappa = 0.53), assessing for spontaneous breathing with low agreement (59.7% vs. 55.8%, kappa = 0.50), and checking a pulse with low agreement (44.2% vs. 39%, kappa = 0.34) via OSCEs compared with clinical vignettes. Fewer providers performed suctioning the mouth with moderate agreement (81.8% vs. 83.1%, kappa = 0.66) and ventilating at 40 breaths per min with low agreement (28.6% vs. 37.7%, kappa = 0.58) in OSCEs compared with clinical vignettes (Table 2). Additional newborn care skills are highlighted in Tables S1 and S2.

Postpartum hemorrhage

More providers performed elevating the legs of the woman by raising the foot of the bed with high agreement (3.9% vs. 1.3%, kappa = 0.95), giving a second dose of oxytocin with moderate agreement (72.7% vs. 71.4%, kappa = 0.76), and deciding on bimanual compression of the uterus with high agreement (41.6% vs. 39.0%, kappa = 0.92) in OSCEs compared with clinical vignettes. Fewer providers obtained a hemoglobin and cross‐matching in OSCEs compared with clinical vignettes with moderate agreement (24.7% vs. 39%, kappa = 0.66) (Table 2). Additional postpartum maternal care indicators assessed by OSCEs and clinical vignettes are highlighted in Table S3.

DISCUSSION

Our study found that for most intrapartum care skills there was low to moderate agreement between OSCEs and clinical vignettes. Most of the actions assessed in the clinical scenarios involved tactile skills. Our study found a higher proportion of providers demonstrating these actions, such as supporting the baby as it exits the vaginal canal during second stage labor, examining the vulva, perineum, and vagina for lacerations during third stage labor, and proper placement of the bag valve mask during neonatal resuscitation, in OSCEs rather than verbalizing them through clinical vignettes. More knowledge‐based skills, such as ventilating at 40 breaths per minute during neonatal resuscitation, were more often caught in a clinical vignette rather than OSCE.

These findings are consistent with what is expected from these two methods of assessment. The OSCE offers a simulative experience that resembles real life cases where tactile skills can be performed. During a clinical vignette, an action is verbalized or written, which allows providers to indicate their understanding of the scenario and describe actions that should be taking place. , There were only five actions where agreement was high between OSCEs and clinical vignettes: monitoring postpartum vital signs every 15 min, giving oxytocin intramuscularly within 1 min of birth, correctly diagnosing postpartum hemorrhage, elevating the legs of the woman by raising the foot of the bed during a postpartum hemorrhage, and deciding on manual compression of the uterus during a postpartum hemorrhage.

There were several actions either performed or verbalized by high proportions of providers in both OSCEs and clinical vignettes that did not have high agreement. Most of these actions involved the management of second and third stage labor. Certain neonatal resuscitation skills also had high performance in clinical vignettes and OSCEs. It is possible that these skills had high performance because they were more commonly enforced or traditionally learned while training intrapartum care providers.

Implications

Our findings support that the OSCE is overall better for the assessment of required skills for an intrapartum care provider because most of the actions assessed were tactical or hard skills, as is required of most clinical scenarios within intrapartum care. The few knowledge‐based skills that were assessed were better captured by clinical vignettes, indicating that OSCEs and clinical vignettes are complimentary to one another as methods of assessment. Healthcare educators should consider modifying OSCE‐based assessments within a low‐resource setting to allow providers to verbalize their diagnoses or indicate during the assessment their clinical reasoning. Providing these opportunities can allow the assessment to better capture knowledge‐based skills. Although the OSCE is sufficiently more difficult to facilitate in a LMIC setting, our findings support that global health organizations should invest in providing and funding health facilities with resources that support simulation‐based learning, including provision of patient simulators, like MamaNatalie mannequins, and training supervising health providers to be able to perform these assessments.

Limitations

One of the greatest limitations of the study is the small sample size of 77 providers, which limits the precision of the kappa value and has limited power to provide estimates of performance. Second, providers completed both assessments on the same day, which may have influenced the performance in the subsequent assessment. Similarly, staff received anecdotal reports from heath workers that cheat sheets were passed after the first assessment was given. We also were unable to perform direct observation with a sufficient sample size to compare clinical vignettes and OSCEs.

CONCLUSION

We observed that very few skills within intrapartum care have high agreement between OSCEs and clinical vignettes. Clinical vignettes and OSCEs seem to be complimentary methods for intrapartum quality‐of‐care assessment and should be considered as such in low‐resource settings, with preference given to OSCE for intrapartum care providers while supporting and advocating for resources to facilitate these assessments. Intrapartum care requires knowledge, strong clinical decision making, and practiced skills, including managing delivery of a newborn and potential complications. We should continue to advocate for methods of assessments that capture the most needed skills and support efforts that provide the resources to do so.

CONFLICTS OF INTERESTS

The authors have no conflicts of interest.

AUTHOR CONTRIBUTIONS

AMPY was responsible for the data analysis, data interpretation, and drafting all sections of the manuscript. DM contributed to data interpretation and drafting and editing the manuscript. DM, EF, EH, AMK, and MM conceived, conducted, and reported on the parent study and edited the manuscript. DM and MM posed the research question for this manuscript and supported AMPY in the analysis and interpretation of the data. All authors provided feedback on the drafts of the manuscript.

TablesS1‐S3

Click here for additional data file.