Electronic Fetal Monitoring Credentialing Examination: The First 4000.

Objective  Recognized variability in fetal heart rate interpretation led the Perinatal Quality Foundation (PQF) to develop a credentialing exam. We report an evaluation of the 1st 4000 plus PQF Fetal Monitoring Credentialing (FMC) exams. Study Design  The PQF FMC exam is an online assessment for obstetric providers and nurses. The exam contains two question types: traditional multiple-choice evaluating knowledge and Script Concordance Theory (SCT) evaluating judgment. Reliability was measured through McDonald's Total Omega and Cronbach's Alpha. Pearson's correlations between knowledge and judgment were measured. Results  From February 2014 through September 2018, 4,330 different individuals took the exam. A total of 4,057 records were suitable for reliability analysis: 2,105 (52%) physicians, 1,756 (43%) nurses, and 196 (5%) certified nurse midwives (CNMs). As a measure of test reliability, total Omega was 0.80 for obstetric providers and 0.77 for nurses. There was only moderate correlation between the knowledge scores and judgment scores for obstetric providers (0.38) and for nurses (0.43). Conclusion  The PQF FMC exam is a reliable, valid assessment of both Electronic Fetal Monitoring (EFM) knowledge and judgment. It evaluates essential EFM skills for the establishment of practical credentialing. It also reports modest correlation between knowledge and judgment scores, suggesting that knowledge alone does not assure clinical competency.

In essentially every labor and delivery (L&D) unit in the United States, Electronic Fetal Monitoring (EFM) is considered a basic competency for all nurses and providers of obstetrical care. 1 Despite ongoing debates about its accuracy in predicting neonatal outcomes, EFM is utilized in the overwhelming majority of deliveries in the United States. 2 3 Standard EFM definitions and guidelines were established in a 2008 conference sponsored by the Eunice K. Shriver National Institute of Child Health and Human Development (NICHD), the American College of Obstetricians and Gynecologists (ACOG), and the Society for Maternal-Fetal Medicine (SMFM). 4

A subsequent ACOG Practice Bulletin presented these definitions and guidelines. The purpose of the bulletin was not only to improve communication among clinicians who care for patients in labor but also to improve outcomes of perinatal morbidity and mortality. 5 The Joint Commission, which accredits and certifies over 20,000 health care organizations and programs in the United States, had previously identified inadequate interpretation of fetal monitoring as a root cause of 34% of sentinel events in L&D. 6 Specifically implicated in these sentinel events were communication issues (72%) and EFM orientation and training processes (40%).

Training in EFM interpretation is a foundational component of obstetrical resident education. Similarly, EFM training is routinely built into the basic obstetric orientation of nurses. Many textbook chapters, research articles, and continuing medical education (CME) courses are devoted to various aspects of EFM. However, a reliable, standardized approach to validate EFM competency was needed. To address this need, the Perinatal Quality Foundation (PQF) launched a Fetal Monitoring Credentialing (FMC) exam in 2014. The PQF's aim is to provide meaningful credentialing for EFM interpretation to those providing obstetrical care. 7

Examinations developed by other organizations are available and focus primarily on EFM knowledge or they target specific caregiver groups. In contrast, the PQF developed separate but similar exams for such caregiver groups: nurses (registered nurses [RNs]) and obstetric providers (physicians and certified nurse midwives [CNMs]). While the questions may be very similar between the two exams, they differ by reflecting each group's different roles and responsibilities in EFM interpretation and management decisions. The FMC exam is novel in that it assesses competency not only through traditional knowledge-based questions but also unique judgment-based questions.

The judgment-based questions employ Script Concordance Theory (SCT), a method of testing that measures the consistency of a clinician's organization of knowledge for clinical action with that of experts. 8 This approach is particularly suited to evaluation of clinically uncertain scenarios that lack a clearly defined course of action. 9 The unique format of SCT questions is unfamiliar to most obstetric practitioners, although it has been validated and used in other disciplines. 10 11 12 13 A typical question presents a clinical situation, along with a previously determined course of action. Then new information is introduced, and the examinee is asked to rate his or her agreement with the previously determined action. Agreement is measured through a Likert-type scale. The questions are structured to mimic the dynamic clinical situations frequently encountered on L&D. 10

The SCT is useful in situations where there is no clear single best approach and individual clinicians presented with the same clinical scenario may act somewhat differently. This is often the situation found on L&D when delivery decisions involving a fetal heart rate (FHR) tracing entail multiple patient-specific factors such as parity, estimated time until delivery, and other clinical morbidities. One clinician may decide that the risk of an adverse outcome balanced with the anticipated time to achieve a vaginal delivery warrants proceeding with a cesarean section. In contrast, another may believe it reasonable to allow labor to proceed while continuing to monitor progress and the FHR tracing.

We report an evaluation of the first 4000 plus PQF FMC exams.

Methods

Data Collection

The PQF FMC exam is administered online at https://fmc.perinatalquality.org/ . Nurse and obstetric provider exams each contain ∼70 scored questions, of which 40 are classified as knowledge and 30 are classified as judgment (SCT) questions. Physicians and CNMs take the same provider exam, as their decision-making responsibilities on L&D are similar. An examinee that fails the initial exam is offered an opportunity to retake an exam similar in both length and content. Over time, some questions have been modified to improve validity based on feedback from examinees and ongoing statistical analysis. For example, wording was changed to make questions more clear. Statistical analysis included application of Item Response Theory, to estimate each question's difficulty, along with its ability to differentiate among examinees. 14

To support continuing evolution of the exam, three to five unscored new questions are evaluated for future use. Each question is associated with a higher-level objective associated with important aspects of EFM interpretation and management. There are five such objectives:

Interpretation of tracing per National Institute of Child Health and Human Development (NICHD) Guidelines

Characterization of Category II

Evaluation of Category II management

Assessment of Category III risk

Evaluation of Category III management

Each objective is addressed with a mix of both knowledge and judgment questions with the exception of the first objective, for which only knowledge questions are utilized. Relatively more questions are allocated to the objectives evaluating Category II tracings because Category II patterns account for the majority of tracings encountered in clinical practice.

At the beginning of each exam, demographic information is collected and includes clinician type (medical doctor [MD], RN, and CNM), years of clinical experience, degree for RNs (associate, bachelor, master, PhD), level of training (such as resident, fellow, attending), medical specialty (such as OB/Gyn generalist, OB hospitalist, MFM, family medicine), board certification, and faculty status.

Data filters were applied to identify valid first-time examinees. When necessary for reliability calculations, incomplete examinations were excluded. Incomplete exams often arise when the examinee does not reach the last question(s) because of the exam's time constraint. Over time, monitoring of individual question performance led to dropping one RN and two MD exam questions due to poor statistical performance. For reliability analysis, these changes in exam content (e.g., removed questions) were retrospectively applied to all records.

Exam Scoring

Knowledge questions are comprised of two types, either a single best or multiple response answer (e.g., “Select all that apply”). For the multiple response questions, partial credit is granted when some, but not all responses are correct. Similarly, partial credit is deducted when some, but not all responses are incorrect. The final score is based on a composite of the knowledge and judgment components.

SCT questions are designed to reflect the inherent variability routinely experienced on L&D. Daily decision-making requires the obstetric provider to choose between allowing labor to continue and expediting delivery for fetal indications. As such, experts are not expected to always agree on the “best” course of action. Consistent with this expectation, there were no questions included in the exam where the experts were in complete agreement. Consequently, judgment (SCT) questions are subject to the aggregate scoring method. 15 In aggregate scoring, the answer chosen by most experts is awarded full credit. Remaining answers receive partial credit commensurate with the percentage of experts who chose the answer. Fig. 1 shows an example SCT question along with credit applied and an explanation of the expert responses from the exams study guide.

Fig. 1

Example of Script Concordance Theory Question from the Fetal Monitoring Credentialing Study Guide.

Thirty-four experts (nineteen MD/CNM, fifteen RN) were chosen based on recognized excellence in the area of EFM through publication or established superior clinical performance and leadership. Selection of experts followed guidelines previously established in the literature. For example, sample panels of 15 to 20-members were previously shown to correlate highly with a larger population of experts. 16 Selected experts came from both academic and community backgrounds demonstrating a breadth of experience in EFM, consistent with prior recommendation for such breadth over narrow niches of expertise. 9 Breadth of experience was confirmed by statistical evaluation of potential experts' impacts on reliability indices.

Participation was confidential and voluntary.

Statistical Analysis

Reliability of the exam was calculated through coefficients known as Alpha and Omega. (Cronbach's) Alpha measures the reliability of a test with respect to one underlying concept. Alpha is considered to be biased low when an exam is known to measure multiple concepts. 17 Although not an optimal measure for this multiobjective exam, Alpha is reported here because of its frequent and familiar use. (McDonald's total) Omega measures the reliability of a test with respect to both a general, common factor and several supporting, specific factors.

Negative impacts on reliability coefficients are expected from several skewed items. 18 These items test important basic concepts and are usually answered correctly (knowledge questions) or answered in agreement with the experts (SCT questions). Further reductions of Alpha and Omega are expected in exchange for the exam's construct validity. 19 For example, while reliability coefficients could be increased simply by adding questions of redundant content, doing so would compromise the breadth of skills that the exam is intended to address within a reasonable time frame. Hence, we aim to satisfy, but not to exceed commonly accepted standards of reliability: 0.7 as minimal, and 0.8 as good. 20

For purposes of comparison, knowledge and judgment scores were normalized to mean zero and a standard deviation of one. Pearson's correlations between knowledge and judgment scores were measured.

Results

The exam was introduced in early 2014. From introduction through September 30, 2018, 4,330 different individuals took the exam. Of these, 4,196 were determined to be valid first attempts. After exclusion of an additional 139 exams with some unanswered questions, 4,057 were suitable for reliability analysis ( Fig. 2 ). Those suitable attempts include 2,105 (52%) physicians, 1,756 (43%) nurses, and 196 (5%) CNMs. The majority of examinees registered through their organizations representing a broad range of practice settings, from midwife only to small community hospitals to large academic centers. Approximately a third registered as individuals through the PQF website. Exam completion over time is shown in Fig. 3 . The pass rates for physicians/CNMs and nurses were 96.4 and 91.0%, respectively. The examinees' characteristics are detailed in Table 1 .

Fig. 2

Data filters. CNM, certified nurse midwife; MD, medical doctor; RN, registered nurse.

Fig. 3

Exam completion for MD/CNMs and nurses since introduction. CNM, certified nurse midwife; MD, medical doctor; RN, registered nurse.

Table 1

Examinee characteristics

n (% total)	MD	RN	CNM
	2,181 (52%)	1,804 (43%)	211 (5%)
Experience (% total ) a
Years experience (mean ± SD)	15.2(11.1)	10.7(9.8)	17(11.2)
n for years experience (% total)	1,483(35.3%)	1,625(38.7%)	198(4.7%)
Not reported (all exams)	890(21.2%)
Education (% exam )
Associate's degree		449 (24.9%)	4 (1.9%)
Bachelor's degree		1,131 (62.7%)	3 (1.4%)
Master's degree		154 (8.5%)	172 (81.5%)
None		3 (0.2%)	1 (0.5%)
Other		65 (3.6%)	26 (12.3%)
PhD		2 (0.1%)	5 (2.4%)
MD	1,979 (90.7%)
DO	202 (9.3%)
Clinical role (% exam )
Ob/Gyn generalist	1,026 (47%)
Maternal fetal medicine Subspecialist	202 (9.3%)
Ob/Gyn hospitalist	102 (4.7%)
Family medicine doctor	83 (3.8%)
Fellow	65 (3%)
Resident in training	673 (30.9%)
Nurse midwife (CNM)			211 (100%)
Labor and delivery nurse		1,746 (96.8%)
Nurse practitioner		29 (1.6%)
Other1	30 (1.4%)	29 (1.6%)
Board certified (% exam ) a
Yes	1,247(57.2%)
Resident year (% exam ) a
PGY1	310 (14.2%)
PGY2	155 (7.1%)
PGY3	107 (4.9%)
PGY4	87 (4%)
Faculty status (% exam ) a
Yes	774 (35.5%)	372 (20.6%)	85 (40.3%)
Other	870 (39.9%)	175 (9.7%)	145 (68.7%)

Abbreviations: CNM, certified nurse midwife; MD, medical doctor; PGY1, postgraduate year 1; RN, registered nurse; SD, standard deviation.

Not available for all examinees, category percentages shown may sum to less than 100%.

Reliability

Alpha and total Omega were measured as follows, with reported values comprising the 4000+ exams administered to date:

The MD/CNM exam Omega is 0.0.80; the RN exam Omega is 0.77.

The MD/CNM exam Alpha is 0.77; the RN exam Alpha is 0.75.

Knowledge, Judgment, and Exam Objectives

Knowledge and judgment scores vary considerably with only moderate correlation between the two: 0.38 for physicians/CNMs and 0.43 for nurses. Scatter plots of the normalized knowledge and judgment scores are shown in Fig. 4 .

Fig. 4

Variation and distribution of knowledge scores compared with judgment scores. MD, medical doctor; RN, registered nurse.

Discussion

Analysis of initial 4000 plus PQF FMC exams utilizing knowledge and judgment components demonstrates its reliability. The overall pass rates were high; however, wide variability was noted in both knowledge and judgment scores. In addition, only moderate correlation was noted between knowledge and judgment scores.

The exam combines traditional questions that test basic knowledge concepts related to fetal monitoring with unique questions designed to assess obstetrical clinical judgment. The pass rates show the majority of individuals practicing on L&D have the necessary basic knowledge and clinical judgment skills to adequately interpret FHR tracings, although a wide range of both knowledge and judgment scores among both physicians and nurses were noted. We believe this range suggests a likely benefit from ongoing training and assessment. Analysis revealed only moderate correlations between individual knowledge and judgment scores. The lack of a strong correlation implies that an individual's basic knowledge of FHR monitoring principles does not automatically translate into practical clinical skills and judgment. This lack of translation leads to questions about the utility of traditional assessments that would focus only on knowledge.

A working definition of professional competence is “the habitual and judicious use of communication, knowledge, technical skill, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” 21 An important component of competence includes the integration of basic knowledge with clinical judgment, an aspect not reliably assessed in traditional multiple-choice questions in a knowledge-only format. That format also cannot reliably incorporate the inherent ambiguity that often accompanies actual clinical practice.

Training and assessment of FHR monitoring interpretation have been almost accomplished primarily through didactics and observation. 22 After residency completion for physicians, or L&D orientation for nurses, there is no national standard to assure ongoing competency. To address this lack of standard, Berkowitz et al 7 articulated the case for a credentialing exam pointing out that individual obstetric providers and nurses who had been interpreting FHR tracings for years believed they were competent without any objective supporting evidence. At the same time chart reviews showed differences among these same providers' FHR interpretation. In 2005, the Medical Center Insurance Company, insuring several East Coast academic medical centers, required all caregivers to pass an EFM exam to work on L&D. Pettker and Grobman 23 in an expert review of quality and safety obstetrics also suggest that the standard of assuming competency based on completion of training or orientation program is inadequate.

A strength of the PQF credentialing exam is that it aims to address such inadequacies by using SCT questions to measure integration of core EFM knowledge with clinical judgment. This type of question simulates real life by incorporating ambiguity routinely encountered in daily practice. Assessment using SCT has evaluated decision-making during gynecologic surgery 12 as well as surgical specialties and radiology. 10 11 13 In addition, the large number of individuals taking the exam allows robust statistical measurement of its reliability.

Despite the theoretic benefits of the PQF credentialing examination, one of the limitations is that it cannot overcome the well-known inability of FHR monitoring to predict accurately fetal compromise and long-term neurologic outcome. 24 However, EFM can identify fetuses at risk of acidemia and it currently remains a foundational component of current labor management. 13 25 26 As such, the exam can strengthen knowledge and judgment surrounding management of FHR tracings. While there is not yet direct evidence that the exam is associated with improvement in clinical outcomes, comprehensive safety programs that have included training in FHR monitoring interpretation have demonstrated a decrease in adverse events and medicolegal costs. 27 28 Another limitation of the exam is that it only assesses individual performance and not participation in teamwork, which is an important skill necessary for a highly functioning L&D unit.

Opportunities for future work include analysis of knowledge and judgment scores by provider attributes, such as experience, training, and other characteristics collected in the FMC registration process. Given collection of over 4,000 exam results, opportunities for enhanced reporting include percentile rankings to suggest opportunities for focused training based on individual results. To assure the success of such training in the context of SCT, reliable methods to augment clinical judgment will need to be developed.

Conclusion

Analysis of more than 4,000 PQF FMC exams demonstrated a statistically reliable exam. For both physicians and nurses, knowledge and judgment scores are only moderately correlated, with such scores demonstrating clinically important variation. We believe that the modest correlation, together with the variation in scores, identifies opportunities for continued training and monitoring of competency with respect to both knowledge and judgment. Consequently, we can envision opportunities for training that address each a through a different mechanism. Lectures could improve knowledge and provide a foundation for judgment, while clinical simulation could advance development of judgment. Through improvement in both knowledge and judgment, FHR performance in L&D could be optimized.