Developing a Knowledge Test for a Neonatal Ethics Teaching Program.

Objective The innovative Neonatal-Perinatal Medicine (NPM) Ethics Teaching Program at the University of Ottawa provides NPM trainees with vital foundational knowledge required to manage ethically contentious clinical scenarios frequently encountered in practice. In this study, our aim was to develop a knowledge test to assess the impact of the NPM Ethics Teaching Program on trainees' knowledge about ethics. Study design Using an iterative four-step process, we developed a test for assessing pre- and post-training knowledge of NPM ethics. We first created a blueprint of the test, identifying its purpose, length, and format. We then weighted the learning outcomes of the NPM Ethics Teaching Program sessions to determine the number of questions that would be asked to assess to each learning outcome. Next, we populated the question bank and constructed a draft test. We obtained feedback from content experts on the draft test and piloted the draft test with former trainees from the NPM Ethics Teaching Program. Results We developed a pre- and post-knowledge test in NPM ethics consisting of 44 multiple choice questions (MCQs), each with five response options. The test takes approximately 60 minutes to complete. It took roughly 15 months to design and pilot the NPM ethics test. Conclusions This test can aid in the assessment of the amount of NPM ethics gained by trainees and contribute to the identification of areas for improvement in teaching and in the overall ethics program. Further iterations of the test will allow for additional assessment of its validity and the efficacy of the teaching program. Given the lack of structured evaluative ethics teaching programs in NPM nationally, this project will act as another step towards the introduction of our NPM Ethics Teaching Program to other Canadian NPM residencies.

Introduction

With medical and technological advances and increased survival rates among babies with extreme prematurity and complex medical conditions, difficult ethical situations frequently arise in clinical practice. To manage these situations, postgraduate trainees in Neonatal-Perinatal Medicine (NPM) must have knowledge of intricate communication skills, and ethical principles and frameworks. In recent years, the NPM Ethics Teaching Program at the University of Ottawa was redesigned [1] to better address several of the Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS roles and work continues to ensure the program fits with the CanMEDS competency-based framework [2].

The CanMEDS framework stresses the importance of knowledge of each of CanMEDS seven roles and asserts that ethical competence requires assessment using multiple modalities including written knowledge tests [3]. Clinical competence integrates multiple resources including declarative, procedural and conditional knowledge [4]. Although competency is complex and multidimensional and may be best assessed by direct observation in clinical practice, knowledge is still the root of any competency and should be appropriately assessed [5]. Reliable assessment of trainees’ knowledge helps ensure high-quality educational programming and clinical practice [6,7]. We therefore require assessment tools to evaluate our Neonatal Ethics Teaching Program. However, tools do not presently exist for assessing NPM postgraduate trainees’ knowledge in the area of neonatal ethics [8]. While a review of assessment methods for professionalism revealed 49 tools addressing a variety of ethical issues [9], all were targeted to medical students or adult specialty trainees and were not applicable to post-graduate level of training in NPM. To date, only two published tools assessing knowledge related to NPM ethics exist to examine pediatrics residents [10,11]. Despite some questions from these two tools focusing on NPM ethics, they were developed for pediatrics residents who have notable differences in the content of their training program.

This gap in the literature and the importance of assessment to demonstrate knowledge gains highlights the need for a new tool to assess trainees’ knowledge in NPM ethics. This paper details the development of a knowledge test to measure pre- and post-training postgraduate trainees’ knowledge of medical ethics related to NPM.

Materials and methods

The Children’s Hospital Eastern Ontario Research Ethics Board and the Ottawa Hospital Research Ethics Board approved the study.

We used a multi-step, iterative process to develop the knowledge test [12,13]. To ensure that the items on the knowledge test were of equal difficulty, we used the same items on both occasions; pre- and post-training. There is a two-year period between the administration of each of the two tests which should diminish, if not eliminate, the risk of recall of test items. Recognizing that validation is an ongoing process [14], we also employed a detailed methodology to develop several lines of validity evidence.

The study team consisted of two health education researchers with expertise in assessment of tool design, two academic neonatologists with an interest in ethics (GM, EF), a neonatologist with a Master’s degree in communicative ethics (TD), and a lawyer with a Master’s degree in bioethics (Mr. Paul Muirhead).

Step 1: Creation of the blueprint for the knowledge test

The aim of this step was to develop a blueprint or systematic outline to identify content for the pre- and post-knowledge test. To develop this blueprint, the study team planned three half-day sessions over the course of two months to discuss the purpose, length, format, and content weighting for the test. The paragraphs below provide additional information on each of these topics.

a) Identification of the purpose of the test

As the initial goal in the development process, it was necessary to clearly identify the purpose of the test and to ensure that all team members agreed upon and understood this purpose.

b) Determination of the length of the test

We then aimed to determine how many questions our test should include. Determining the appropriate length of the test necessitated that we take into consideration the purpose of the test, the amount of time allotted for it, and the implications of using multiple choice questions (MCQ) (noting the general rule that 50, well-constructed, four-option MCQs take approximately 60 minutes to complete [15]).

c) Selection of the appropriate format for the test

After considering various knowledge test formats including true-false or short answer questions, the study team confirmed that MCQs would provide adequate evidence of trainees’ knowledge learning. Although short answer questions have the advantage of making students generate answers without prompting from an MCQ’s stem or response options, MCQs were chosen because they allow incorporation of specific key learning points from the neonatal ethics teaching curriculum into the response options, they are easier to mark on repetitive iterations and they remove the negative potential for a rater’s subjective interpretation to influence the marking.

d) Weighting of the sessions’ learning outcomes and content

In order to place appropriate weight on each question asked in the knowledge test, the team considered the emphasis and teaching time that the NPM Ethics Teaching Program devoted to each session as well as each session’s learning outcomes. Three of the team members (GM, EF and TD) independently ranked the learning outcomes and then, with the assistance of the educational experts, reached consensus on the ranking through discussion and examination of dedicated teaching time to each learning outcome. In addition to the relative teaching time devoted to the learning objectives during each session, another main criterion that guided the ranking process was the perceived importance of each learning outcome for the learner to be able to competently navigate ethically challenging clinical situations. This process ensured that the test proportionately represented each learning outcome and its relevant materials [15] while also considering the clinical importance of each learning outcome for a future neonatologist.

e) Development of the blueprint for the test

To conclude Step 1, we listed the learning outcomes in their weighted order and estimated the number of questions to be assigned to each of them. This information provided the necessary details to finalize the blueprint.

Step 2: Populate the question bank for the knowledge test

Prior to developing the test items, we reviewed the literature on MCQ development to ensure that we developed high-quality MCQs and avoided question and response option flaws [16]. Recognizing that the four program instructors (GM, EF, TD, PM) had the best understanding of the learning outcomes and content that made up the sessions for the program, we asked each of them to use the blueprint, with the weighted outcomes, to brainstorm an initial set of MCQs and corresponding response options related to their respective sessions. The instructors also examined other published assessment instruments, including those from Lynch, et al.’s review [9], to see how others have assessed specific ethical concepts and, as appropriate, adapted existing items for their own purposes.

Once the instructors developed their draft questions, a Research Assistant (RA) arranged two half-day brainstorming and review sessions. The RA collated and distributed the draft questions for review prior to the first session. The objectives of these sessions were to: (a) review the draft questions and rework them as needed, (b) generate additional questions if required, and (c) construct a draft knowledge test. Once the objectives were met, the RA formatted the constructed draft test and circulated it to the research team for further review and approval.

Step 3: Obtain feedback on questions from external content experts

We invited a total of three external content experts in the domain of medical education expert, theoretical bioethics, and clinical neonatology and bioethics to review and provide feedback on the draft test. The RA emailed these three external experts written information on the program, written instructions on how to complete the review, an electronic copy of the test, and a debriefing questionnaire. We asked them to provide feedback on the clarity, understandability, and level of difficulty of the test using the debriefing questionnaire. We asked these experts to: (a) comment on the length and the time allocated for the test; (b) provide feedback on the appropriateness and relevance of the test questions and answer options, and; (c) identify any questions they felt were missing or should be removed from the test [17,18]. Finally, we invited the experts to comment directly on the knowledge test to indicate any concerns or recommend changes directly related to specific questions or answer options.

Step 4: Pilot the knowledge test with former trainees from the NPM Ethics Teaching Program

Once step 3 was concluded, we piloted the knowledge test with a convenience sample of four former trainees from the NPM Ethics Teaching Program to improve the quality and utility of the test [12]. Three program instructors (GM, EF, TD) organized a mock testing session where the pilot participants completed the knowledge test as they would if they were formally taking the test. Before the test, each volunteer participant was handed a test package that included an information letter, a consent form, written instructions on how to complete the test, the draft knowledge test, and a debriefing questionnaire. Once the participants were ready to begin the test, they were given exactly 60 minutes to complete it. After completing the test, they were asked to complete the debriefing questionnaire to provide feedback. Similar to the expert review, we asked pilot participants to provide feedback on the clarity, understandability, and level of difficulty of the test. We asked them to comment on the length of the test as well as the amount of time allocated to complete it. We also asked the pilot participants to reflect on the NPM Ethics Teaching Program and to provide feedback on the appropriateness and relevance of the test questions and answer options. Finally, we asked them to identify any questions or response options they felt were missing or should be removed from the test.

The RA entered individual test responses for each pilot participant into SPSS statistics for Windows, Version 21 and corrected them using the test answer key. The RA then developed an aggregated report indicating pilot participant responses and correct answers for each question as well as the overall test scores and completion times. The study team reviewed all test responses and flagged test items that were answered incorrectly by three or more pilot participants for further review. We used the pilot participant feedback and test results to revise and improve the knowledge test.

Results

Step 1: Creation of the blueprint

The creation of the blueprint for the knowledge test took three half-day sessions over the course of two months. Through focused discussion, we decided that the purpose of the test is to determine the amount of knowledge that postgraduate trainees gain from the NPM Ethics Teaching Program. We chose MCQ format for its suitability to the learning outcomes and its ease and consistency in scoring results; we also agreed on the 60 minutes available time for test completion. Table 1 lists the top five ranked learning outcomes of the NPM Ethics Teaching Program; Figure 1 shows the weighted distribution of the independently ranked learning outcomes.

Table 1

Top 5 ranked learning outcomes of NPM Ethics Teaching Program.

NPM: Neonatal-Perinatal Medicine.

#	Description
1	To apply ethical approaches to clinical situations
2	To define five key bioethical principles
3	To recognize the different clinical and ethical approaches to generating care plans in neonatology
4	To describe ways to resolve disagreements regarding treatment options
5	To identify principles of good communication with patients

Figure 1

Weighted distribution of learning outcomes.

Step 2: Populate the question bank

It took approximately six months for the four program instructors to become familiar with MCQ development in order to create their MCQ’s and corresponding response options. At the conclusion of the two half-day sessions to review the questions and responses, the participants generated a primary version of the knowledge test that included 44 MCQs related to 24 of the 29 learning outcomes.

Step 3: External content experts review

Including the invitation process, four months were necessary for the medical education expert to provide feedback on the structure, readability and clarity of the MCQs. Table 2 summarizes the feedback from responses of the two content experts. Their written answers (not shown) to additional open-ended questions provided reasoning for the responses in the feedback form, allowing us to further improve the quality of the knowledge test.

Table 2

Summary of content experts’ feedback.

*One expert did not choose ‘agree’ or ‘disagree’ but wrote many comments suggesting they were not in complete agreement or complete disagreement.

**One expert did not choose ‘agree’ or ‘disagree’ but wrote ‘Variable’.

Please consider the following statements and add a checkmark under either ‘Agree’ or ‘Disagree’.		Agree	Disagree
	The test items were at an appropriate level of difficulty.	0	1*
	I found this test fair.	0	1*
	The clinical scenarios reflected my experiences in Neonatal-Perinatal Medicine.	2	0
	The test items were clear.	0	2
	The test items were easily understandable.	0	1**
	I thought the test was an acceptable length.	2	0
	The time for completing the test was adequate.	2	0

Step 4: Pilot phase

Overall, the scoring percentages of the four pilot participants on the draft test were 80%, 68%, 64% and 52%, respectively. Figure 2 demonstrates the response selection of the participants. Sixteen questions were answered correctly by all four participants; five questions were not answered correctly by any of the four participants. Table 3 summarizes feedback on the test from the pilot participants. Based on the pilot participants’ response selection and their feedback, a final review of the test was completed. The review focused on the construct of the questions, grammatical errors and item-specific comments. From the review, seven of the 44 MCQs had their stem altered and 13 had one or more of their responses modified. It took two months to complete the pilot testing, to review the feedback and to create the final version of the knowledge test.

Figure 2

Frequency (out of 44 items) of different percentages of correct response selection by former NPM Ethics Teaching Program participants (n = 4).

NPM: Neonatal-Perinatal Medicine.

Table 3

Summary of pilot participants’ feedback.

*One participant did not choose ‘agree’ or ‘disagree’ but wrote “Some were clear” and “Some were easily understandable”.

Please consider the following statements and add a checkmark under either ‘Agree’ or ‘Disagree’.		Agree	Disagree
	The test items were at an appropriate level of difficulty.	4	0
	I found this test fair.	4	0
	The clinical scenarios reflected my experiences in Neonatal-Perinatal Medicine.	4	0
	The test items were clear.	3*	0
	The test items were easily understandable.	3*	0
	I thought the test was an acceptable length.	4	0
	The time for completing the test was adequate.	4	0

Discussion

We believe that this systematically developed knowledge test in neonatal ethics should help to fill in, at least partially, an important gap in the literature – the lack of well-designed assessment tools to evaluate knowledge and competencies to navigate ethically sensitive situations in neonatology [3,8,19]. The test will be applicable to NPM and other training programs that share the common and relevant learning outcomes of our NPM Ethics Teaching Program and assist in assessing medical ethics knowledge in the era of competency-based medical education. The rigorous well-structured design process and the improvements made after external expert’s feedback supports the content validity of the neonatal ethics knowledge test. However, the small sample size of this pilot study did not allow for testing the reliability and construct validity of our test.

Medical educators must assess their trainees’ learning upon the completion of training initiatives, as they are accountable to these trainees and the patients and families whose care will be entrusted to them [15]. Although not the best strategy to assess a multifaceted competency, a knowledge test is necessary to ensure high-quality educational programming and clinical practice. It allows the opportunity to evaluate if the program’s curriculum is well taught [6,7] and that learners are retaining the information in, at least, the short term while they are actively participating in our program. In addition, assessment directs and influences trainees’ learning, even increasing the potential for learners to truly search for meaning and understanding [7].

The role of written tests remains important because the ability of trainees to demonstrate the understanding of key concepts is essential for their clinical competency [3,20]. Structured ethics training programs require an evaluative component including assessment tools such as written tests. Existing tools do not assess the specific knowledge in NPM ethics or meet our specific objective to evaluate the efficacy of the NPM ethics teaching program. The Socio-moral Reflection Measure of Gibbs, used by Self, et al. for first-year medical students, focuses on assessing trainees’ social-moral judgement skills [21] rather than their knowledge of key ethical concepts. Although trainees in NPM may demonstrate these judgement skills, they also require the necessary knowledge of key ethical concepts to deal with ethically challenging situations [22]. Kesselheim, et al. developed an ethics knowledge test for pediatrics residents but only five questions were related to neonatology, which was insufficient to meet our goals [8]. Many other ethics assessment tools focus on attitudes and behaviour rather than knowledge [9].

A review of 20 years of literature [9] found only nine examples of written knowledge tests. Although some of the content in these knowledge tests may be applicable to NPM trainees, they were intended for medical students (n = 7) or other adult specialty trainees (n = 2). Notably, Sulmasy, et al. [23] did create a 15-item MCQ test with face validity and a Cronbach’s alpha of 0.76-0.86, but nearly all the MCQ dealt with specific adult and/or American medico-legal issues. Another written test only assessed medical students’ reasoning in justifying an ethical decision as a means to explore knowledge gain [24].

A major strength of our study is that the test reflects our NPM ethics teaching program learning objectives and accurately measures the efficacy of our program in supporting ethics knowledge gain. There were several limitations with our test and its development. The use of only MCQs minimizes the depth of knowledge assessment in some areas of ethics. Complete assessment of certain learning outcomes and competencies require additional validated knowledge testing methods, such as short answer questions and structured oral examinations. Additionally, we acknowledge that the relatively high number of learning outcomes in our NPM Ethics Teaching Program makes it very difficult to assess each one with a single 44-item MCQs knowledge test. By using our ranking methodology, though we ensured that the test focused on the most important learning outcomes, the test did not assess several other low ranked outcomes. Finally, we did not collect detailed demographic information regarding the pilot participants (such as their background education in ethics and clinical exposure) that could act as a confounder through its influence on the participants’ responses and opinions during the test development.

This work is part of a multi-phase research program designed to formally evaluate our teaching program. Work in progress includes the provision of the knowledge test as a pre- and post-assessment of NPM trainees’ knowledge prior to and after their participation in our NPM Ethics Teaching Program. Only three to five trainees are admitted at the University of Ottawa NPM residency program per year. Given the small sample size, data will be accumulated for at least three cohorts prior to analysis. By determining the amount of knowledge that trainees gain from our NPM Ethics Teaching Program, as educators, we will understand which concepts in the program are taught well and which ones need revision, additional teaching time, or alternative teaching methods.

Our next step includes assessing how trainees use their knowledge when navigating complex and challenging clinical situations; this is a critical learning outcome to measure. In addition to our knowledge test, we are in the process of developing a communication assessment tool for ethically sensitive scenarios that will be used during our teaching sessions but also at bedside when trainees are directly interacting with parents. We will be able to evaluate the relationship between their knowledge in neonatal ethics and their performance during these very sensitive situations.

Conclusions

Our systematically developed knowledge test in neonatal ethics is the first published dedicated assessment tool of NPM trainees’ knowledge related to specific learning outcomes and their expected competencies in dealing with ethically challenging clinical situations. Although the test was designed around a unique teaching program, it may be useful to other medical training programs based on common learning outcomes. Future reiteration of the test will allow for evaluation and improvement of our NPM Ethics Teaching Program.