Continuous multidimensional assessment system of medical residents in teaching hospitals.

Background and aims: Evaluation of learners is considered as one of the most important principles in education, especially in the clinical fields. Continuous assessment can be used to provide appropriate feedback to students about their strengths and weaknesses. In this regard, this study is aimed to design a system of continuous assessment of medical residents (SCMARs).
Methods: This study was performed using a combination of qualitative methods, including focus group discussion, expert panel, and Delphi technique. The study population consisted of all the stakeholders involved in the evaluation process of medical residents in Imam Reza Hospital Complex (IRHC) in Iran. This study was conducted in three phases, including identification of subthemes and indicators, production of a primary framework for the SCMAR, and agreement on the subthemes of the SCMAR. The nominal group technique was used for generating priority information. Data analysis was performed during the agreement review stage with the Excel software version 2016.
Results: The finalized SCMAR consisted of 10 main themes and 38 subthemes. The themes included objectives, evaluators, areas, and indicators of evaluation, evaluation periods, evaluation requirements, data collection, data sources, point assignment and data analysis methods, reporting, and feedback dissemination methods. Five areas of evaluation and 11 indicators of evaluation were proposed.
Conclusion: A comprehensive assessment system that continuously evaluates the performance of Medical Residents can be used as a stimulus to improve the quality of educational processes. The present study was aimed to address this need by designing a framework for such a system.

INTRODUCTION

In the last 20 years, there have been obvious changes in the development and research for assessment methods in medical education. Students play a crucial role in the improvement of the performance of the educational and treatment system. Moreover, they are responsible for the treatment of patients, especially in unconventional shifts, in medical education centers. One of the most important and challenging issues in clinical education is student evaluation.

Therefore, comprehensive assessment helps identify the strengths and weaknesses of education and take steps to reform the educational system by strengthening the positive aspects and eliminating the shortcomings. Furthermore, clinical evaluation of students through direct observation in practical and real situations will guarantee their ability to cope and predict clinical events in specific patient conditions. Clinical assessment methods that are accompanied by feedback promote learning and are able to evaluate the student in areas that are difficult to evaluate by traditional methods. ,

Studies performed by Ogunyemi et al., Goldstein and Zuckerman, and Stark et al. about the assessment of the professional skills and behavioral competence of residents suggest that assessment requires more than one group of evaluators. In other words, multiple groups of evaluators need to be involved in the assessment. , Moreover, to improve and enhance the quality of clinical education, its status should be constantly evaluated.

According to the above‐mentioned studies, a desirable method should continuously assess the learners and guarantee the participation of all stakeholders in the process, while giving them feedback on the evaluation results. Therefore, due to the importance of assessment based on the integrated indicators and the need to facilitate and accelerate access to the collected information, Imam Reza Hospital Complex (IRHC) decided to design and implement system of continuous assessment of medical residents (SCMARs). The education and treatment managers of the hospital started to design and implement SCMAR to reduce the problems.

MATERIAL AND METHODS

This qualitative study was conducted in three phases. The research environment was IRHC in Mashhad, Iran. IRHC is a large public teaching hospital with 1000 beds. It provides medical care for more than 6000 hospitalized patients and about 19,000 patients in the Emergency Department per month. There are ~300 faculty members and 500 students in various fields of medicine in IRHC.

Phase 1: Identifying the subthemes and indicators

The main research method in this phase was focus group discussions with stakeholders involved in the medical resident assessment process. Members of the focus group discussion included six medical residents, education and health deputy of the hospital, head of the educational services department of the hospital, and two head nurses in the wards with the highest number of residents. The group had three meetings, 2 h long. They discussed the most important indicators of educational accreditation standards. Moreover, after reviewing the related literature in the focus group discussions, the requirements of clinical ethics and the performance of medical learners were discussed. The literature required at this step was prepared by the narrative review approach, using the following words and phrases in search engines Google Scholar, PubMed, Science Direct, EMBASE, and Web of Science databases, Assessment, Teaching hospital, Medical education, and medical resident.

In the meetings, each member was allowed to comment on the areas that needed improvement. Moreover, it was emphasized that the members should listen to the statements and views of the group members with respect and present their complementary opinions after they finish talking. To prevent deviation from the research objectives, the answers were guided if necessary. At this phase, the knowledge, views, and attitudes of members about the SCMAR were extracted and after reaching a consensus, the aspects of this system were identified.

Phase 2: Produce a primary framework for the SCMAR

This phase is mainly designed to produce a primary framework for the SCMAR. Its main research method was the expert group meeting. Members of the Medical Education Committee of IRHC were selected as members of the expert group. The members of the Medical Education Committee included the head of the hospital, 19 clinical academics, education and treatment deputy of the hospital, head of the hospital quality improvement office, head of the hospital clinics, head of the clinical research unit, head of the hospital education department, and head of the hospital clinical education development office. To further enrich these sessions, three other people were invited, including the deputy of hospital management and resources development, a faculty member familiar with the evaluation methods, and the director of nursing of the hospital.

In this phase, the nominal group technique proposed by Potter et al. was used in five steps. In the first step, the study was introduced, which was followed by an explanation of the session. The topic in the session, a summary of the findings of the previous stage of the study, was presented in a session for 10 min. Then, the questions and ambiguities of the participants regarding familiarity with the study were answered.

The second step was based on the protocol of the silence to generate the idea. This step was conducted in three parts: in the first part, during a session, people were asked to think about the themes of SCMAR. Their opinions in response to the following questions were gathered: “Is this theme appropriate for SCMAR?” and “Are the themes defined providing the necessary comprehensiveness to evaluate the performance of medical residents?” Participants were given 20 min to think about these questions and generate their own ideas.

In the second part, which was conducted in two sessions, participants were asked to think about the following questions, which were about the axes with at least 75% of participants agreed: “Are these subthemes appropriate for the intended theme? And are the subthemes comprehensive enough? And what other subtheme do you suggest?”

In the third part, people were asked to think about the indicators related to the field of axis and residents' evaluation indicators and to generate their ideas on the following questions: “Are these indicators appropriate?” and “Are the indicators comprehensive enough?” and “What other indicators do you suggest?”

In the third step, for each part of the previous step, ideas were created based on Ron Robin's model. Accordingly, all individuals were asked to introduce their ideas and no ideas were rejected or discussed at this step by the team leader and others. All items were displayed on the video projector screen without any discussion about the titles. This step was continued until everyone introduced all their ideas on the next topic and all ideas were recorded. The next step was a group discussion to clarify the ideas. At this step, the first volunteer was asked to talk about the opinions of her/his colleagues and discuss the issues. Afterward, everyone was given the opportunity to express their opinions.

The final step was to vote on the ideas. At this step, voting was done for each part. Items with more than 75% of the votes remained in the model and others were excluded. The result of the steps was the creation of a basic framework for continuous evaluation of medical residents, which was considered as input for the next step, which is the Delphi method.

Phase 3: The agreement on the subaxes of the SCMAR

In this phase, we conducted an expert panel and all the possible options for components in the primary framework were considered. Components of the primary framework of SCMAR, which was obtained from the prior phase, were confirmed by the Delphi method. Purposive sampling was used for data gathering. To ensure the representativeness of the panel members, expert members included the Medical Education Committee, the research management team, and members of the focus group discussion (Table 1). In addition, the snowball sampling method was used to identify other participants. In the first round, themes of the Primary Framework for SCMAR were sent to the panel members. Panel members were asked to express their viewpoints about each theme with three choices: agree (1), disagree (2), and without viewpoint (0). Themes with more than 75% Consensus were used for the secondary framework. Themes with Consensus between 50% and 75% were entered into the second round of Delphi and others were excluded. After gathering the panel member's statements, we summarized the viewpoints in an appropriate format for feedback, so that each member received an abstract of the panel viewpoints as well as a reminder of the scores that the member had assigned to each theme. In addition, the panel members were invited to a second round. Based on the findings of data analysis in the second round of Delphi, there remained no themes with an agreement between 50% and 75%. In the final step of consensus development, the panel members were invited to see the feedback and discuss their viewpoints about the appropriateness of each theme and subtheme based on their own professional decision. The interviews focused on the applicability, conformity, and relationship between themes and approaches to amend data gathering and future steps in the expansion of SCMAR. All participants were consented to participate in the study.

Table 1

Demographic characteristics of the participants (participants in the first and second round of Delphi)

Variable	First round (35 participants)	Second round (32 participants)
Gender
Female	12	11
Male	23	21
Education
Subspecialty and medical fellowship	10	9
Medical specialty	13	11
PhD	3	3
Residency	6	6
Master's degree	3	3
Occupation
Physician (including all medical categories and residencies)	24	22
Member of the hospital management team	4	4
Nurse	4	3
Academics with outstanding work in the areas of study	3	3
Work experience
Below 10 years	18	17
10–20 Years	13	12
Over 20 years	4	3

RESULTS

In this study, six participants did not participate in the second round of Delphi due to their busy work schedules. As shown in Table 2, 38 subthemes (of 52 subthemes) in the first round of Delphi gained a score higher than 75% of total points and 5 items (with an agreement between 50% and 75%) were entered into the second round of Delphi. Nine subthemes that scored <50% of total points were excluded. The panel members added a further four subthemes to the list in the second Delphi round. Finally, 10 themes and 43 subthemes were identified (Table 2).

Table 2

Main themes of system of continuous assessment of medical residents

Themes	Delphi score
	Round no. 1	Round no. 2
Objectivesa
Assessment of residents in the field of individual performance	91	–
Assessment of residents in relation to the patient	54	78
Assessment of residents in the field of equipment	80	–
Assessment of residents in relation to hospital policies	83	–
Assessment of residents in relation to other residents	94	–
Evaluatorsb
Departments chancellor	97
Head nurse	66	81
Supervisor	77	–
Hospital evaluators (from quality improvement office)	54	75
Areas and indicators of assessmentc
Individual performance: Adherence to the organizational uniforms(including dress code)	83	–
Individual performance: Respectful behavior and interaction with nonphysician staff	97	–
Individual performance: Timely attendance at the patient's bedside for a visit or consultation	91	–
Individual performance: Adherence to the five moments for hand hygiene	100	–
Patient: Respectful behavior towards the patients and adherence to the patient privacy and confidentiality	97	–
Patient: providing effective patient education	97	–
Patient: Introduction to the patient and describes your status in the treatment team before any action	97	–
Equipment: Technical skills in the use of capital equipment and adherence to the principles of maintenance of capital equipment	86	–
Hospital policies: Adherence to the requirements of the health insurance	74	84
Hospital policies: Visiting critical and emergency patients immediately after notification	94	–
Educational: The role of residents as educators	86	–
Repetition periodd
The full assessment period is 6 months and is commensurate with the Residents' rotation	71	78
In a full course, the assessment is done on a monthly	97	–
Assessment requirementse
Assessment should be based on evidence	83	–
Assessment should be done by a person familiar with educational activities and processes	69	88
Data gatheringf
Observation	80	–
Face to face interview	91	–
Document review	66	78
Data sourcesg
Departments chancellor	83	–
Head nurse	63	75
Deputy director of education of the hospital	83	–
Faculty members	77	–
Patient	71	91
Data analysish
Checklist questions with two options—Yes: >60% = 1; No: <60% = 0	60	81
Descriptive statistics in the form of frequency tables, graphs and the defined percentage for each of the educational groups and medical wards	86	–
Reportingi
The main message	86	‐
The executive brief	57	75
The full report	89	–
Feedback and disseminationj
Email	86	–
Automated SMS	66	78
Formal letter	77	–
Group meetings	71	88
Morning rounds with students through a summary of the evaluation results based on the individual, group, and educational level	86	–

Note: Superscript letters d, e, g, and i indicate that there is no item has been excluded in these sections.

In this theme, training purposes for interns was excluded.

In this theme, evaluation by other residents was excluded.

In this theme, timely attendance at the ward and saving on use of medical equipment were excluded.

In this theme, logbook and evaluation from the perspective of society were excluded.

In this theme, investigate complaints received from the department.

In this theme, WhatsApp group feedback and publication on the hospital website were excluded.

According to the designed assessment system, the required data for evaluation is collected from various sources, including the head of the ward, the deputy director of education of the hospital, the faculty members working in the ward, the head nurse, and the patients in the ward. The evaluation period is 6 months and is based on the rotation of the learners. Moreover, the data collection method is a combination of interview, observation, and document review methods (Figure 1).

Figure 1

Continuous multidimensional assessment system for medical residents.

One of the most important themes of the designed SCMAR is the theme of areas and indicators of evaluation. Based on the steps taken in this study, the evaluation indicators were defined in five areas, including individual performance, patients, equipment, hospital policies, and education (Table 3). The checklist items had two options (i.e., yes and no), which could be scored zero and one. Accordingly, the lack of the desired performance in each indicator in <60% of the evaluated items was scored 0, whereas the desirable performance in each indicator in >60% of the evaluated items was scored 1. The mechanism of data analysis was determined as descriptive statistics in the form of frequency tables, graphs, and the defined percentage for each of the educational groups and medical wards.

Table 3

Evaluated indicators of medical students

Column	Indicators	Areas
1	Adherence to the organizational uniforms (including dress code)	Individual performance
2	Respectful behavior and interaction with nonphysician staff
3	Timely attendance at the patient's bedside for a visit or consultation
4	Adherence to the five moments for hand hygiene
5	Respectful behavior towards the patients and adherence to the patient privacy and confidentiality	Patient
6	Providing effective patient education
7	Introduction to the patient and stamen of their position in the treatment team before any action
8	Technical skills in the use of capital equipment and adherence to the principles of maintenance of capital equipment	Equipment
9	Adherence to the requirements of the health insurance	Hospital policies
10	Visiting critical and emergency patients immediately after notification
11	The role of residents as educators	Educational

DISCUSSION

This study aimed to design a SCMAR to improve the performance of medical residents in various aspects. The benefits of using an evaluation system include the establishment of a regular evaluation system based on transparent axis and indicators, provision of a comprehensive and practical picture of the performance of residents at the hospital, continuous observation and attention to various aspects of performance, recruitment, and training of at least 100 evaluators at different levels, increase in the number of evaluations and prevalence of assessment by different evaluators, participation of all stakeholders the evaluation, and an increase in the satisfaction of the stakeholders.

According to previous reviews of comprehensive performance evaluation programs, they have been most influential in the axis of observation and evaluation. , The influence of the observation axis indicates the importance of a comprehensive and constructive evaluation system. In the present study, the aspects and components of the continuous evaluation system were reviewed using the ideas of experts in four basic steps to guarantee a complete evaluation. Based on the results of a study conducted in Iran using the Objective Structured Clinical Examination (OSCE) method, the feedback was positive and the students were content with this evaluation method.

In previous studies, students had stated that the OSCE test covered a wide range of clinical knowledge and competencies, identified learning weaknesses, and was equitable. One of the obtained categories was preparation as a process of adaptation; accordingly, by preparing and studying to succeed in various assessments, the students' learning improved regardless of the stress. However, the above‐mentioned evaluation system had some weaknesses; for instance, regarding the simulation of the clinical environment, it was stressful and was also held in inappropriate locations and at inappropriate times. ,

These results are consistent with the evaluation system designed in the present study in terms of direct observation; nevertheless, the evaluations in this study will be performed in a clinical environment, which is one of the highlights of the SCMAR. In the comprehensive OSCE test, the learner might have a good basic knowledge and also know how to use it in different situations, but not be able to perform well when faced with patients.

Among the main themes identified in this study were feedback dissemination methods. Continuous assessment in all shifts is conducted by the provision of feedback on the performance of medical residents to improve their performance. In a study conducted on medical students in the Neurology Department, Zhao et al. found that a combination of Mini Clinical Evaluation Exercise and Direct Observation of Procedural Skills can improve the daily evaluation of clinical skills of medical students and help them achieve their educational goals.

Similarly, in a study performed in India, Jani et al. found that implementation of a 360° evaluation through regular directional programs and provision of feedback to medical residents about their strengths and weaknesses leads to improved performance. The best element in learning is the active participation of students in the educational process and the most important point in evaluation is the provision of feedback to them. However, despite the important role of feedback in medical education, the status of feedback provision in clinical education is not desirable. Clinical professors either do not provide feedback or provide it in an inappropriate, erroneous, and unplanned form without a specific model or solution.

This study aimed to create a framework and consider the characteristics of effective feedback to benefit from the mechanisms of dissemination of feedback, including emails, letters, and group meetings, morning rounds through a summary of the evaluation results based on the individual, group, and educational level. It seems that the continuous assessment of residents promotes professionalism, teamwork, and communication. Moreover, the provision of feedback to students can be effective in teaching medical ethics. Research has shown that medical students frequently encounter moral conflicts during the course of their study, and that the greatest weakness in their communication with patients is introducing themselves. , ,

In this study, evaluation of respectful behavior towards patients and respect for their privacy and the principle of confidentiality, appropriate interaction with the staff, and introduction of themselves to the patient were among the indicators evaluated in the continuous assessment system of medical residents. This assessment system promotes professional ethics which plays an important role in communication with patients.

Limitations

Due to the outbreak of the coronavirus disease‐19 pandemic during the final sessions of the assessment system design, it was not possible to have face‐to‐face interaction and in‐depth interviews with residents to obtain their statements and comments. Therefore, discussions with residents as stakeholders involved in the evaluation process were conducted virtually via e‐mail and video call.

Implementation of a SCMAR in the form of a system or application that can be used on smartphones and tablets will greatly increase its effectiveness and establishment. This system should be web‐based, provide the ability to define access levels, define users and create a user panel, be user‐friendly, provide advice and guidance for users, perform evaluations and provide online performance feedback instantly, provide cross‐sectional and process reports, provide reports for individuals, groups, and educational categories, have an archive of evaluation records, be easily updated, send reports in the form of e‐mail and text message to managers, allow the managers to view activity reports based on the period, section, and subaxis in the relevant folder, allow incentives or disciplinary control measurements by choosing the name of the person, their position, and the type of action in question, provide warnings and reminders, and allow the graphic presentation of the information. It is suggested that an intervention study be carried out by implementing the SCMAR to generalize the obtained results with more confidence.

CONCLUSION

Evaluation in the form of a SCMAR can help the managers, faculty members, and residents of medical departments to make sure that the tasks are performed more accurately and quickly with fewer costs. It also plays an important role in the promotion and institutionalization of professional ethics and the establishment of effective communication with the patients. It should also be noted that the above‐mentioned evaluation system should be performed continuously and provide the necessary feedback to the evaluated groups and individuals. This will increase the possibility of correcting professional behaviors and activities in the evaluated individuals and also increase the effectiveness of education. This has been one of the most important goals of designing a SCMAR in this study.

AUTHOR CONTRIBUTIONS

All of the authors participated in designing the study. Mehdi Yousefi: Conceptualization and methodology, project administration, writing – review & editing. Zahra Ebrahimi: Investigation, methodology, writing–original draft, writing–review & editing. Somayeh Fazaeli: Writing–review & editing. Leila Mashhadi: Project administration, Writing–review & editing.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

ETHICS STATEMENT

This study was approved by the Research Ethics Committee of Tehran University of medical sciences, Tehran, Islamic Republic of IRAN (Ethical code: IR.TUMS.NIHR.REC.1400.012). Written informed consent was obtained from all the participants. All methods were carried out in accordance with relevant guidelines and regulations.