Stepwise Simulation Course Design Model: Survey Results from 16 Centers.

BACKGROUND AND OBJECTIVES: In 2016 we published a stepwise evidence-based model (subsequently named SimSteps) for curriculum development (CD) of simulation-based courses. The current study aimed to assess the uses, user friendliness, and perceived effectiveness of this model and its worksheet and to obtain suggestions for improvement.
METHODS: We sent e-mail invitations for a 14-question web-based survey to 13 health professionals who requested the supplemental worksheet of the stepwise model and 11 authors who cited the model's publication in 14 articles. The survey included quantitative and qualitative items.
RESULTS: Sixteen (67%) from seven countries and six professions responded. Ten (63%) used the model: six for both course and faculty development, three for course development only, and one for faculty development only. Both users and nonusers found the model and worksheet applicable and user friendly and agreed that they guided use of a systematic, comprehensive approach to CD. 94% (15 of 16) agreed that they helped CDers integrate educational effectiveness criteria, develop more objective learners' assessment tools, and enhance validity for their courses. Sixty-nine percent (11 of 16) agreed that model and its worksheet helped CDers include nontechnical skills in courses. The highest reported role in enhancing program evaluation results was in the gain of knowledge (five of eight, 63%) and least was clinical outcomes (two of eight, 25%). All respondents would recommend the model and worksheet to a colleague.
CONCLUSION: Respondents find the stepwise model and its worksheet user friendly and helpful in developing simulation curricula of high educational standards. Future studies should include larger sample size, objective measures of impact, and longer-term follow-up.

INTRODUCTION

Simulation is an important teaching and learning method that can shorten health professionals' learning curves.[1] Simulation has long been used to teach provider skills, and its use in health professions education is expected to expand with the increasing use of milestones and entrustable professional activities to reach decisions regarding trainees' level of proficiency and ability for safe independent practice.[2,3] Simulation has also been used to enhance patient safety[4] by decreasing individual human errors[5] and enhancing health care systems' performance.[5-8]

Whereas the use of simulation is rapidly growing, the lack of standardization in simulation-based course design, implementation, learner assessment, and course evaluation has hindered the gathering of multiinstitutional evidence of the transfer of learnt skills to the workplace.[9,10] Moreover, the lack of rigorously established outcome measures has made it difficult to measure the return on investment of simulation. Gardner et al.[11] identified three areas that, if standardized using validated frameworks, could lead to the highest return on investment: debriefing, curriculum development (CD), and assessment.

In 2016 we published a stepwise systematic model for the design of simulation-based courses (subsequently named SimSteps) that integrated criteria of educational effectiveness for simulation (e.g., deliberative repetitive practice to proficiency)[12] into Kern's six steps of curriculum development.[13] The model was developed based on literature review, input from a panel of international experts, and consensus development.[9] Guidelines and a worksheet ( and ) for applying the model were made available on request. Since then, the model has been cited in more than 45 publications (as retrieved from Google Scholar Citations), and we have received communications from multiple centers in several different countries showing interest in the model and asking for the supplemental worksheet.

The goals of the current study were to assess the uses, user friendliness, and perceived effectiveness of the model and worksheet from those who showed interest in them and to obtain suggestions for further improvement.

METHODS

This was a cross-sectional survey study.

Participants

In January 2018, we sent e-mail invitations to participate, with an embedded link to the survey, to: (1) 13 health professionals who requested the supplemental worksheet and guidelines of the stepwise model that were mentioned to be available upon request in the model publication and (2) 11 of the authors who cited our stepwise model's publication in 14 of their articles. At the time of data collection of the current study, the number of citing articles of our stepwise model publication was 16 as retrieved from Google Scholar Citations. We included all citing journal articles in different languages but excluded self-citing articles and those from outside the health professions field. As such, we excluded 2 articles, one was a self-citing article by one of the model's authors (R.S.) and the other was a non–health care article that cited the model while describing the design of a biology course. We sent two further e-mail reminders in February and April 2018. Responses were received in the duration between January 2016 and May 2018.

Survey

We developed a 14-question Web-based survey (Survey Monkey, Palo Alto, CA) with quantitative and qualitative items. In addition to the respondents' demographic data, the survey questions covered the following categories regarding our published stepwise model for curriculum development[9] and its template worksheet: (1) attendance of a training workshop, (2) use of the model to develop simulation-based (SB) courses, (3) applicability for developing respondents' courses, (4) effect on enhancing the educational effectiveness of the developed SB courses, (5) impact on enhancing the outcomes of the developed SB courses (6) use in faculty development, (7) user friendliness, and (8) recommendation to others. In addition, there was an open-ended question about providing suggestions/comments for enhancement.

Analysis

Descriptive analysis was used to quantify responses. Frequency counts and mean values were calculated using the Web-based survey analysis tool of Survey Monkey. Fisher exact test was used to evaluate the associations between the categorical outcomes and the attendance of a training workshop and use of the model in course design and in use in faculty development variables. The statistical significance level was set at P ≤ .05. One of the investigators (N.K.) carried out the qualitative analysis of the open-ended suggestions and comments question by categorizing and coding responses into themes. A second investigator (D.K.) reviewed the coding. No discrepancies were reported.

RESULTS

Response Rate and Respondent Characteristics

Sixteen of the 24 invitees responded (67%). Of the 13 who requested the supplemental worksheet and guidelines between January 2016 and May 2018, 11 responded (85%).

Responses were received from seven different countries and six different professions, although most of the respondents were from institutions located in the United States and were physicians. Education was a focus of work for most respondents (94%), although almost all had other foci as well. Eight respondents (50%) had leadership and administrative positions as directors, codirectors of simulation centers, education operations manager, and assistant dean or board members of national scientific associations. shows the characteristics of the survey respondents. Half of the study's participants (eight of 16) had attended a training workshop on applying the stepwise model for CD to designing simulation courses.

Uses of the Stepwise Model and Its Worksheet

Ten of the 16 respondents (63%) reported using the model and its supplemental materials. A statistically significant relationship was found between respondents' use of the model for course design and its use in faculty development (P < .05; Fisher exact test). Eighty-six percent of those who reported using the model for faculty development (six of seven) also reported using it for course design. The 67% who reported using the model for the design and review of all or some of their simulation-based courses (six of nine) also reported using it for faculty development. shows the total use, nonuse, and intent to use by the study respondents.

Respondents who used the model to develop courses have used it to design and review courses for different specialties, skills/procedures, and learners at various levels of training. shows examples of the courses that the respondents used the model to develop.

Applicability of the Stepwise Model and Its Worksheet

None of the respondents, whether having used the stepwise model, intending to use it in the future or not having used it, found it not applicable to the courses they develop. Most of the respondents (13 of 16, 81%) described it as very applicable, and three respondents (19%) found it moderately applicable.

According to the opinions of 12 respondents (75%), the fill-in worksheet of the stepwise model was moderately user friendly. Three respondents considered it very user friendly (19%), whereas only one respondent (6%) viewed it as slightly user friendly. No one found it user unfriendly.

Helpfulness of the model and worksheet: guiding curriculum developers (Table 6).

Regarding the helpfulness of the stepwise model and its worksheet in guiding CDers to enhance the educational effectiveness of the simulation-based courses, all 16 respondents (100%) agreed or strongly agreed that the stepwise model and worksheet help/guide CDers to use a systematic and comprehensive approach for curriculum development for simulation. Ninety-four percent (15 of 16) agreed or strongly agreed that it helps the curriculum CDers to integrate criteria of educational effectiveness of simulation into the course development process, develop more objective assessment tools of learners performance (e.g., checklists with clear outcome measures and metrics), plan for course implementation as an essential step of curriculum development, and enhance the validity of their developed simulation-based training course(s).

Sixty-nine percent (11 of 16) agreed or strongly agreed that the stepwise model and its worksheet help/guide CDers to argue for the need for the course and gain educational management support, and 63% (10 of16) to ssure the inclusion of nontechnical skills (e.g., communication skills, teamwork skills) in the developed course. shows the respondents' level of agreement on the helpfulness of the model and its worksheet in guiding CDers.

Enhancing Course Evaluation Results ()

Eight of the nine respondents who used the stepwise model and worksheet in developing at least one of their simulation courses recorded opinions regarding their role in enhancing the evaluation results of the courses that they developed. The highest reported role was in the gain of knowledge (five of eight, 63%), followed by learners' satisfaction (four of eight, 50%). On the other hand, 63% of the respondents were not sure or had no evaluation results available regarding gain of skills, workplace transfer, and clinical outcomes of full procedures”.

Role of the model in enhancing course evaluation results.

Recommendation of the Model to Others

All respondents mentioned that they would recommend the model and its worksheet to a colleague either as an excellent model (seven of 16, 44%) or a good model (nine of 16, 56%).

Comments and Suggestions for Improvement

Five respondents provided answers to the final open-ended question asking about comments and suggestions for improvement. Comments included that the model and worksheet were applicable across health professions, were easy to follow, and sped the process of simulation courses design. The model was also noted to have value for research. Using a template was believed to be a good tool for planning that ensured consistency and inclusion of all steps of the course design process.

Respondents' suggestions for improvement included development of a light version of the worksheet to encourage completion, especially by surgeons, and provision of hands-on-training to assist unexperienced users.

DISCUSSION

Initial feedback from users shows that the stepwise model and worksheet have been used to develop variety of courses for diverse groups of trainees, at variable levels of training, in different countries. This is in addition to its use for faculty development. Both users and nonusers perceived the model and worksheet as user friendly and helpful in providing a systematic and comprehensive approach for the design and objective assessment of simulation courses. The model connects the courses to true health care needs with inclusion of the relevant cognitive, technical, and nontechnical skills.

Nontechnical skills (e.g., communication, teamwork) acquisition is important for health care team functioning and patient safety.[14] However, despite the availability of validated courses (e.g., TeamSTEPPS[15]), the integration of these skills in simulation training and the assessment of learner's achievement of proficiency[1] is challenging.[16] Sixty-two percent of our survey respondents agreed that using the model and worksheet assures inclusion of nontechnical skills in their developed courses. Our model considers nontechnical skills training as an essential component of the simulation course. In the worksheet, there is a dedicated section for nontechnical skills that shows the peculiarity of this type of skill (e.g., for a team, necessity of reaching/communicating a shared mental model of the task prior to its execution).

The current shift toward entrustment, with gradual increase of responsibility of learners until readiness for independent practice is reached, mandates the use of objective measures of performance with clear milestone-derived metrics. Ninety-four percent of our respondents agreed that the model and worksheet help them to develop more objective assessment tools of learners' performance. Simulation can provide an ideal educational method that can contribute to the journey toward entrustment. It provides the trainees with the opportunity to practice repeatedly while receiving formative feedback until they reach proficiency.[17] These are among the principal concepts underlying our model.

In terms of course/program evaluation, respondents reported that using the stepwise model and worksheet has enhanced learners' knowledge gain and satisfaction. On the other hand, there was lack of evaluation results and/or uncertainty regarding their program's effect on the gain of technical skills, transfer of those skills to the workplace, and clinical outcomes. This might be explained by the fact that evaluation of results at these higher levels of the Kirkpatrick's pyramid[18] requires evaluation designs, evaluation methods, and/or resources not available to many institutions. Collaboration with clinical departments and quality units and longer follow-up periods are needed to detect changes in workplace transfer and clinical outcomes. Hopefully, with time our model and evolving consensus will promote more program evaluation at these levels.

Strengths of the current study include being, to our knowledge, the first assessment of an increasingly used model for simulation-based curriculum design and evaluation. The study is a multinational, multiprofessional one including user and nonuser respondents from educational institutions, health care facilities, and scientific societies. Our limitations include (1) small sample size, (2) variability in the duration between receiving the model's supplemental materials and responding to the survey, (3) elicitation of feedback about single-institutional and not multiinstitutional efforts, and (4) self-reporting by respondents as opposed to objective measurements of impact.

CONCLUSION

This study provides evidence that respondents from 16 international centers found the stepwise model (subsequently named SimSteps) and its worksheet user friendly and helpful in developing simulation curricula that meet high educational standards. Based on these results, other simulation-based educators might want to consider using this model. Future evaluation of the model should include larger sample size, objective as well as perceived measures of impact, longer-term follow-up, and comparison of outcomes before and after application of the model. Based on feedback from respondents, development of an electronic simplified version of the worksheet could be a next step toward increased dissemination and use of the model as a standardized, valued approach to simulation-based course design and evaluation.

Table 1.

Summary of Model Guidelines*

I. Problem Identification and General Needs Assessment	• Description of problem being addressed by simulation curriculum • Performance of gap Analysis to identify difference between current training activities and ideal • Done at international, national or regional rather than institutional level
II. Targeted Needs Assessment	• Collection of data on learners' existing competencies and needs at institutional level • Coordination with other curricula to integrate the simulation training into overall curriculum • Identification of stakeholders and involving them early in the curriculum design process
III. Goals and Objectives	• Identification of cognitive pre-requisites • Defining psychomotor/technical and non-technical skills/competencies at individual and team level • Developing objectives (outcome measures) with their metrics; specific quantifiable (e.g. centimeters, etc.) or unambiguously defined non-numeric values (e.g. cross check defined as assistant repeating surgeon's request verbatim)
IV. Educational Strategies	• Knowledge post-test to determine eligibility to start psychomotor training • Procedure/skill deconstruction into key steps including common and important errors • Setting criteria for expected levels of proficiency • Choice of most appropriate simulation and level of fidelity and determining Proficiency benchmark values • Training through cycles of practice with increasing complexity, recording and review of performance • Planning for faculty Development to ensure expertise in teaching and assessment using simulation
V. Individual Assessment and Feedback	• Development of assessment tool and inclusion of space for open-ended comments • Establishment of inter- and intra-rater reliability and documentation of validity evidence • Use for both formative and summative assessment • Consideration of repeating assessments to ensure maintenance of proficiency
VI. Program Evaluation	• Kirkpatrick pyramid hierarchy is applicable: clinical outcomes>clinical behaviors int practice>knowledge, skills and attitudes gain>satisfaction with learning experience • Use of aggregated learner assessments to evaluate success of program • Subjective assessment of curricular components, areas of strength and improvement and practical value of the course • If feasible, long term follow up of learners.
VII. Implementation	• Paying attention to simulation methodology and setting to ensure fidelity and planning for interprofessional training sessions • Seeking political and administrative support for allocation of resources and addressing barriers • Introduction of curriculum with consideration of piloting or phasing in as appropriate

A copy of the full guidelines is available upon request from the first (corresponding) author

Table 2.

Template Worksheet for SimSteps (the Stepwise Model for Simulation-Based Curriculum Development for Clinical Skills, a Modification of the Six-Step Approach)

Course Title: __

Step 1: Problem identification and general needs assessment (international, national, regional level)

1. Problem identification:

-

-

-

2. Current approach:

-

-

-

3. Ideal approach:

-

-

-

4. Needs identified (use gap analysis, which is the difference between current and ideal)

-

-

-

Step 2: Targeted needs assessment (for targeted learners and setting)

1. Targeted learners:

-

-

-

2. Needs assessment/targeted learners:

-

-

-

3. Needs assessment/targeted learning environment:

-

-

-

Step 3: Goals and objectives

-Competency(ies):

Course title: __

-

-

-For each competency:

-Goal(s):

-

-

-Outcome measures (specific objectives) and their metrics (quantifiable numeric values or clear definition for nonnumeric values—include correct knowledge/actions and ERRORS as based on task deconstruction reached in general needs assessment):

-By the end of the program, learners will be able to:

-Knowledge (cognitive prerequisites):

-

-

-

- Technical (psychomotor) skills:

-

-

-

-Nontechnical (team performance, communication skills, professionalism, etc.):

-

-

-

Competencies and Goals   Outcome Measures and Metrics

Step 4: Educational strategies

As based on the ideal approach identified in step 1, your targeted needs (Step 2), your objectives (Step 3), and available resources, identify:

-Content to be taught:

-

-

-

Educational methods:

1. Orientation

2. Syllabus material: e.g. textbook, handouts, online learning modules, etc.

3. Cognitive (didactic) component:

a. Knowledge pretest

b. Teaching and learning methods (e.g., interactive computer tutorials, video-recorded tutorials, or interactive live tutorials):

-

-

-

c. Knowledge posttest (to determine eligibility to begin the psychomotor component):

3. Psychomotor and nontechnical components:

a. Simulation method appropriate for outcome measures and learner level of expertise:

b. For technical procedures:

-Deconstruction into key components (e.g., steps, tasks, subtasks, skills). Inclusion of common and critical errors and how to identify/prevent/correct them if they occur. These are based on consensus among clinical subject matter experts, medical educators, and behavioral psychologists and on existing evidence of effectiveness, when available.

-

-

-

c. For nontechnical skills (e.g., team training, communication and professionalism):

-Deconstruction into key components; steps/tasks/subtasks/skills for team-related skills (e.g., TeamSTEPPS) for relevant professionalism components, etc.

-

-

-

d. Metrics to be used to quantify the steps/task/subtasks/skills and errors:

(quantitative, e.g., time in seconds, distance in millimeters, number of errors or qualitative in the form of a distinctive attribute or characteristic possessed, e.g., repeats commands, inserts the trocar).

-

-

-

e. Designing the training tool (will also be used also as assessment tool, e.g., checklist, rating scale, etc.) based on the output from task deconstruction and metrics identification.

Item    Done    Not done

f. Set the benchmark value for proficiency to be acquired; thus, simulation exercises are gradually increased in complexity in a proficiency-based progression, and each level must reach 100% proficiency benchmark value before progressing to the next level.

-

-

-

g. Review of individual learner-recorded performance at the end of a training trial.

4. Faculty development:

-Faculty development for simulation (to ensure expertise in use of simulation method), for example, in feedback, small-group facilitation, or other relevant teaching skills

-Total duration of training:

-Topics of training sessions:

-Number of sessions:

-Duration of each training session:

-Educational methods:

-

-

-

Step 5: Individual Assessment and Feedback

-Development:

1. Assessment tools:

a. Pre- and posttests for cognitive component (pretest only needed for research trials)

b. For psychomotor and nontechnical skills, edit training tool developed in step 4 to develop the assessment tool:

i. Add task and subtask benchmark values (as set by experts)

ii. Include cells for total scores and global rating scores

iii. Include space for open-ended comments (helpful for formative assessment/feedback).

iv. Establish inter- and intra-evidence for assessment tools (of >0.80); consider internal structure (homogeneity), and alternate form reliability evidence when appropriate.

v. Plan for other forms of evidence of validity when feasible (e.g., concurrent, predictive, or discriminant validity).

-

-

-

2. Pass scores:

(As noted above, it is recommended that each learner achieves a 100% score before progressing to the next level.)

-Cognitive posttest:

-Psychomotor and nontechnical skills:

3. Use:

-Formative assessment with feedback (inform learner of errors they commit during the training trial) until benchmark value is achieved)

-Summative assessment (final grade/certification of level of proficiency)

-Reassessment, often at 6–8 weeks, and retraining if necessary to ensure maintenance of proficiency.

Step 6: Program evaluation

1. What questions are you trying to answer with your program evaluation?

-

-

-

2. For learner perspectives on the curriculum, what questions are you trying to answer?…tquality of faculty performance?…tsatisfaction with content of the course?…tperceived effectiveness of educational methods?…ttechnical problems?…tetc.? What method will you use: e.g., questionnaire, focus group, etc. This type of evaluation usually uses a posttest-only design.

-

-

-

3. For effectiveness of the course in achieving desired learner outcomes (often aggregates of individual assessments):

a. What is/are your evaluation designs (e.g., posttest only, pre-posttest, control/comparison group, randomization or not)?

-

-

-

b. Evaluation methods: these will usually be the methods used for individual assessments. Will this be supplemented by other methods, such as video-review?

-

-

-

4. Revision/improvement of curriculum based on evaluation. How will you decide on revisions in the curriculum based upon evaluation results?

-

-

-

3. Review/analysis of evaluation results. What is your plan for preparing and distributing evaluation reports?

-

-

-

Step 7: Implementation

Because simulation is a resource-intensive educational methodology, the curriculum developer wants to ensure that necessary support and adequate resources for the curriculum can be obtained, that requested resources are justified, and that they are efficiently used. Curricular plans may need to be adapted based on available resources and support.

1. Political and administrative support: who are the stakeholders whose support you need (e.g., dean, department head? How will you secure their support?

-

-

-

2. Resources needed: e.g., personnel, time, facilities, equipment, funding

-

-

-

3. Administration of curriculum: what needs to be done, e.g., developing and distributing schedules and reports, collecting information, collating data, communicating information to learners and faculty; who will be responsible for each task?

-

-

-

4. Identification of barriers and solutions:

-

-

-

5. Introduction of the curriculum (consider a pilot study first, then phasing in of the full curriculum):

-

-

-

Table 3.

Characteristics of Survey Respondents

Characteristics	n	(%)a
Location
United States	9	(56%)
Saudi Arabia	2	(13%)
Denmark	1	(6%)
Netherlands	1	(6%)
South Africa	1	(6%)
South Korea	1	(6%)
United Kingdom	1	(6%)
Profession
Physician	8	(50%)
Nurse	4	(25%)
Physiotherapist	1	(6%)
Physician assistant	1	(6%)
Social work professor	1	(6%)
Education scientist	1	(6%)
Foci of workb
Education	15	(94%)
Clinical care	7	(44%)
Research	7	(44%)
Leadership and administration	8	(50%)
Safety systems design and improvement	1	(6%)
Other (did not specify)	1	(6%)

Percentages are rounded to the nearest whole number.

Multiple choices were allowed.

Table 4.

Use of the Stepwise Model in Course Design and Faculty Development (n = 16)

Variables	Yesb	Intend to use	No
Use in course design	9 (56%)	5 (31%)	2 (13%)
Use in faculty development	7 (44%)	7 (44%)	2 (13%)

Percentages are rounded to the nearest whole number.

Total number of users of the model and worksheet among respondents is 10 of 16 (63%) because six of 10 users (60%) used them for both course design and faculty development.

Table 5.

Examples of the Courses That the Respondents Used the Model to Develop

Learners	Courses
Undergraduate HPE students
Year 1 and 2 medical students	Simulations for the preclinical (basic) sciences for the health professions
Year 1 nurse practitioner students	Emergency medicine rotator simulation course
Fourth-year medical students
Graduate HPE students/trainees:
Categorical emergency medicine residents, PGY1–5	Emergency resident's simulation course
Medical residents	Thoracentesis
	Paracentesis
Surgical, anesthesiology, medical, family medicine residents	Central line insertion
Senior anesthesia trainees	Crisis resource management
Gastroenterology fellows	Fundamentals of gastrointestinal endoscopy
Master of medical education students	Module on simulation-based education
Attending physicians, practitioners and faculty
Anesthetic practitioners and consultants	Crisis resource management
Attending emergency physicians	Emergency medicine attending maintenance of certification resuscitation course
Health professions education faculty	Procedural skills
	Curriculum development of simulation-based courses
Others
Graduate social work students	Assessment and intervention skills in psychosocial oncology
Social workers

HPE, health professions education; PGY, postgraduate year.

Table 6.

Respondents' Level of Agreement on the Helpfulness of the Model and Worksheet in Guiding Curriculum Developers (n = 16)

Item	Agreementa
	Strongly Agree	Agree	Not Sure	Disagree	Strongly Disagree
	n (%)	n (%)	n (%)	n (%)	n (%)
Model helps/guides curriculum developers to:
1. Use a systematic and comprehensive approach for curriculum development	10 (63%)	6 (38%)	0 (0%)	0 (0%)	0 (0%)
2. Integrate criteria of educational effectiveness of simulation into the curriculum development process	8 (50%)	7 (44%)	1 (6%)	0 (0%)	0 (0%)
3. Use general needs assessment and problem identification to better focus the course and target it to meet an educational goal	9 (56%)	7 (44%)	0 (0%)	0 (0%)	0 (0%)
4. Argue for the need for the course and gain educational management support	5 (31%)	6 (38%)	5 (31%)	0 (0%)	0 (0%)
5. Assure the inclusion of relevant cognitive background in the developed course	5 (31%)	9 (56%)	2 (13%)	0 (0%)	0 (0%)
6. Assure the inclusion of technical skills (e.g. joint injection, chest tube insertion)	7 (44%)	6 (38%)	3 (19%)	0 (0%)	0 (0%)
7. Assure the inclusion of nontechnical skills (e.g. communication skills, teamwork skills)	5 (31%)	5 (31%)	6 (38%)	0 (0%)	0 (0%)
8. Include common errors into simulation-based training and assessment	4 (25%)	8 (50%)	4 (25%)	0 (0%)	0 (0%)
9. Develop more objective assessment tools of learner's performance (e.g., checklists with clear outcome measures and metrics)	5 (31%)	10 (63%)	1 (6%)	0 (0%)	0 (0%)
10. Introduce the proficiency-based progression process	5 (31%)	9 (56%)	2 (13%)	0 (0%)	0 (0%)
11. Plan for course implementation as an essential step of curriculum development	10 (63%)	5 (31%)	1 (6%)	0 (0%)	0 (0%)
12. Enhance the validity of their developed simulation-based training course(s)	6 (38%)	9 (56%)	1 (6%)	0 (0%)	0 (0%)

Percentages are rounded to the nearest whole number.