Utility of 360-degree assessment of residents in a Turkish academic emergency medicine residency program.

OBJECTIVES: This study was designed to test a 360-degree assessment tool for four of the emergency medicine resident competencies as outlined by the Council of Residency Directors in Emergency Medicine on patient care, communication skills, professionalism and system based practice in an academic Emergency Department.
MATERIAL AND METHODS: Using the competency framework of the American Accreditation Council for Graduate Medical Education, a 57 item-containing assessment tool was created. Based on the different exposure aspects of the involved evaluator groups, the items were integrated into seven different evaluation forms. All sixteen of 16 residents and members from each evaluator group voluntarily participated in the study. Internal consistency scores, multilayer and multilevel Kappa values were measured. Evaluator group scores and resident ranks in competency areas were compared. All evaluators were asked to comment on the applicability and usefulness of the assessment tool in emergency medicine.
RESULTS: Seven groups completed a total of 1088 forms to evaluate 16 residents. The reliability coefficient for the faculty members was 0.99 while it was 0.60 for the ancillary staff. The interrater Kappa values for faculty members, nurses and peer assessment were relevant with a value of greater than 70%. DISCUSSION AND
CONCLUSION: Our results showed that the 360-degree assessment did meet expectations by the evaluator group and residents, and that this method was readily accepted in the setting of a Akdeniz University Emergency Medicine residency training program. However, only evaluations by faculty, nurses, self and peers were reliable to have any value. Doing a 360° evaluation is time and effort consuming and thus may not be an ideal tool for larger programs.

Introduction

In 1999 the Accreditation Council for Graduate Medical Education (ACGME) initiated the “Outcome Project” in the USA, and defined a new conceptual framework to improve the quality and assessment of residency training. This conceptual framework contains six competency areas including (1) patient care, (2) medical knowledge, (3) practice based learning and improvement, (4) interpersonal and communication skills, (5) professionalism and (6) system based practice. The ACGME offered an “assessment toolboxes” for these competency areas.1, 2

In 2002 the Council of Emergency Medicine Residency Directors (CORD-EM) has defined the competencies for emergency medicine residency training by determining specific competency areas and relevant assessment methods in compliance with the ACGME proposals.3, 4 The 360-degree assessment method is recommended by the Council especially for clinical decision making, management skills, communication skills and team work and professionalism.3, 4

The 360-degree assessment, which is also referred to as “360-degree performance assessment”, “360-degree feedback” or “multisource performance appraisal” aims to collect information on the performance of an employee by using different evaluation perspectives including those at the same level in the organization chart, those above, and those at a lower level. It originates from quality work in business and its use in medical education has been reviewed by Locyer.

360-degree assessment methods have been applied in many different fields and their validity, reliability and applicability is well established.7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 It has been suggested to be one of the best methods of evaluation especially for professionalism and communication skills.

The purpose of this study was to assess emergency medicine resident competencies as defined by CORD-EM pertaining to patient care, communication skills, professionalism and system based practice. By gathering feedback from all individuals working closely with the resident. A secondary goal was to then improve their competencies by providing feedback of their deficiencies and/or favorable results.

Materials and methods

Study setting

In 1993 Emergency Medicine was declared an independent specialty in Turkey. The length of residency training is 5 years. The Emergency Medicine residency program at Akdeniz University was started in 1997. The average total number of residents is 25, with 5 or 6 residents starting each year at two different time period following the National Residency Examination for Medical Doctors a multiple choice examination held twice a year in April and September. Based on the exam scores residents are centrally assigned their specialty and residency program in August and December. There is no departmental selection process involved. During the period of assessment, there were a total of 8 faculty members, 25 residents, 20 nurses, 11 triage paramedic staff members, 8 unit clerks, and 14 ancillary staff members working in the emergency department.

Sampling property and evaluators

Only postgraduate year (PGY) 2–4 were included in the study. PGY 5 and PGY 1 were not evaluated since PGY 5 residents were going to graduate in less than a year and there would not be a chance to reevaluate them the following year with the same method. PGY 1 residents were too new to permit adequate evaluation. Six residents at PGY 4, 5 at PGY 3 and 5 at PGY 2 were included in the assessment process.

The evaluations of the 16 residents were performed by 16 residents, 8 faculty members, 10 nurses, 7 ancillary staff, 6 unit clerks, 11 paramedics and 10 patients. Nurses, administrative staff, and ancillary staff with less than a year of employment were also excluded from the evaluator pool because of insufficient exposure to the residents.

Study participants received detailed instructions on how to complete the survey instrument and on the purpose of the assessment.

The assessments were done over a two-month period. A code number was given to the evaluators ensuring confidentiality but permitting to identify their occupational group. The questionnaire forms were filled by hand; an electronic evaluation system was not used. Completed evaluation forms were dropped off in a secured box. Data from the patients was collected by our hospital-employed patient representatives. Medical record numbers of the patients who participated in the study was also obtained for cross checking purposes.

Questionnaire form

The items in the assessment instrument were specifically arranged according to the evaluators. The faculty member evaluation form consisted of 57 items encompassing the competencies of patient care, communication skills, professionalism and system based practice. Residents-peer evaluation forms contained 38 items, nurse evaluator form 33 items, ancillary staff evaluator forms 7 items, administrative assistant evaluator forms 15 items, patient evaluator forms 9 items and self-evaluation forms 38 items (Table 1). Evaluators were asked to give items a Likert scale score of 1 (never) to 5 (always). Open-ended questions were added to the end of the assessment forms. The evaluators were asked to write down their general opinion about the resident, comment on positive aspects as well as to point out areas for improvement.

Table 1

Item number according to evaluator groups.

	Patient care	ProfessionalismCommunication skills	System based practice	Total
Faculty members	22	28	7	57
Nurses	8	20	5	33
Peer–colleagues and self	2	30	6	38
Secretaries	–	15	–	15
Ancillary Staff	–	7	–	7
Patient	–	9	–	9
Paramedic	8	20	5	33

All evaluators were also asked to comment on the applicability and usefulness of the 360-degree assessment tool in emergency medicine.

Feedback process

Scores for each resident were calculated. The results were given to each resident in a sealed envelope for review.

Data analysis

Data was analyzed in two ways: Initially, 5, 4, 3, 2, 1 for quantitative data were coded into SPSS program where negative values would be 1, 2, 3, 4, 5. After this application, mean scores and standard deviation were calculated for each competency area. Internal consistency of scores was tested using Cronbach's Alpha coefficient. Inter-rater agreement was tested by multi-rater, multilevel Kappa test (http://justus.randolph.name/kappa). Evaluators' scores were compared using ANOVA statistics.

Results

A total of 16 residents were evaluated with our 360-degree assessment tool. 68 questionnaire forms were completed for each resident. A total of 1088 questionnaire forms were collected. Reply ratio was 100%. The reliability coefficient for faculty member was found to be 0.99 while it was 0.60 for ancillary staff. A coefficient above 0.80 indicates adequate internal consistency and reliability. General averages of points obtained in the questionnaire form applied to each group are shown in Table 2. Overall, nurses were the group of evaluators who gave the lowest point averages whereas paramedics gave the highest scores. There was a statistically significant difference between the nurses and the other evaluators.

Table 2

Response rate and results of statistics.

	Faculty	Nurses	Peer and self	Unit clerks	Ancillary staff	Patient	Paramedic
No of evaluator	8	10	16	6	7	10	11
No of responses	128	160	256	96	112	160	176
Mean scores
Patient care	4.0 ± 0.6	3.8 ± 0.8a	4.1 ± 0.8				4.4 ± 0.5
ProfessionalismCommunication skills	4.1 ± 0.6	3.8 ± 0.7a	4.0 ± 0.5	4.2 ± 0.6	4.1 ± 0.4	4.1 ± 0.6	4.3 ± 0.6
System based practice	4.1 ± 0.6	3.6 ± 0.8a	4.1 ± 0.6				4.5 ± 0.6
Kappa value	0.78	0.85	0.80	0.59	0.47	0.65	0.67
Cronbach's alfa	0.99	0.96	0.96	0.86	0.60	0.89	0.80

ANOVA statistics, post test's results.

Kappa value for inter-rater agreement was found to be 0.78 for faculty members, 0.84 for nurses, 0.80 for peers, 0.59 for unit clerks, 0.47 for ancillary staff, 0.65 for patients, and 0.69 for paramedics (Table 2).

Fifty-six percent (n = 32) of study participants felt that the 360-degree assessment contributed positively to the operation of the emergency department. Whereas 36.2% (n = 21) were impartial to its impact and 8.6% (p < 0.049) thought that it would have no effect on ED operation.

Participitants' views about the 360° evaluation form were given in Table 3.

Table 3

Participants' views about the assessment form.

	Difficulty			Contribution		Straightforward langualge
	Yes	Partially	No	Yes	No	Yes	No
	%	%	%	%	%	%	%
Faculty (n = 8)	0	62.5	37.5	77.8	22.2	100.0	0
Resident (n = 16)	12.5	62.5	25.0	40.0	60.0	87.5	13.5
Nurse (n = 10)	10.0	30.0	60.0	66.7	33.3	83.3	17.7
Unit clerk (n = 6)	0	83.3	16.7	100.0	0	100.0	0
Ancillary staff (n = 7)	0	14.3	85.7	63.6	37.4	81.8	18.2
Paramedic (n = 11)	9.1	63.6	27.3	31.3	69.7	93.8	6.2

Residents gave the lowest point for the appropriateness of this assessment tool using in Akdeniz University residency program. While the lowest satisfaction rates were belong to residents, the highest rates were belong to paramedics and ancillary staff (Table 4). On the other hand, most facult members and residents found the evaluation system difficult or partially difficult. The main reason was that these evaluators had to evaluate all of the core competencies and had to spend a lot of time on filling the forms. Besides, these physicians were so busy to find appropriate time to complete the questionnaires. Most unit clerks and paramedics also foung the evaluation system difficult since they were not get used to be part of that kind of evaluation system.

Table 4

Participants' views on the appropriateness of the assessment tool for Turkey and the degree of satisfaction.

	Appropriateness for Turkey		Satisfaction
	mean	Sd	mean	SD
Faculty	8.1	1.5	7.5	1.3
Nurse	7.3	2.4	8.2	2.3
Unit clerk	6.7	1.8	7.7	1.5
Ancillary staff	8.4	1.2	9.0	0.5
Paramedic	8.0	1.9	8.3	1.4
Resident	6.4	1.7	6.1	2.2

Discussion

Our study has tested the suitability and applicability of a 360-degree assessment tool for Emergency Medicine residents. We found that the reliability of the questionnaire used was high with reliability coefficients described in previous studies.8, 12, 13, 14, 18

This 360-degree assessment was used with formative purposes in mind. Weaknesses in resident behavior were easily identified and properly addressed.

In our data analysis, different Kappa values (0.47–0.85) were determined. Kappa values for faculty members', nurses' and peers' assessment showed a high inter-rater reliability (>0.70). This is possibly explained by evaluator groups' more frequent and closer exposure to the residents compared to the other groups.

Differences were found in our study in the overall scoring by the evaluators. Faculty' evaluations of resident performance in communication skill and professionalism correlated strongly with those done by ancillary staff and patients. However, kappa values of the ancillary staff and patients were quite low, therefore their evaluations should be interpreted carefully.

Among the evaluators, nurses gave comparatively lower points for all competencies compared to all other groups. Nurses have most one-on-one interactions with residents in terms of work hours. Also residents' overall behavior toward the nurses may be as a result of the hierarchical structure where residents might behave politely and intelligently toward faculty members but may not show the same respect to the nurses. Other studies describe negative correlation in peer evaluation vs. non-peer evaluations. There is evidence that peers observe one another in the same environment not only on technical skill sets but also on interpersonal abilities.19, 20 A great majority of our employees indicated that a 360-degree assessment method would contribute to the work quality of our emergency department. It is well established that such feedback is a useful tool especially for evaluating the changes in professional behavior. On the other hand, most faculty members and residents in our study found 360° evaluation difficult or particularly difficult. The main reason was the time needed to fill so many questions for all residents. In a study which the value of 360° evaluation process was assessed showed that the feasibility of this evaluation did not find new or better information when used as a single point assessment. The authors also claimed that at a juncture where time is valuable for all of us, to commit our programs to routinely adding a 360° evaluation instrument must await additional utility-related studies.

Limitations

Our study had some limitations: Though many studies on using 360-degree assessment were done on even smaller resident numbers than our study, our evaluation was applied to a relatively small group. There was a limited possibility of “bias” since residents of the same PGY level were involved in the assessment of their peers. Distribution, collection and securing data confidentiality was very time consuming and effort intensive which limits its applicability to smaller residency programs.

The most significant characteristic of our study was that it was the first 360-degree assessment method applied in the field of emergency medicine. Previous studies were written on its utility in emergency medicine. The advantages of this assessment tool are clear:

Guide residents in their areas of improvement, give the ED administration an understanding of the care rendered, and adjust residency training efforts to reflect resident competency shortcomings.

Another limitation of our study was that we did not repeat the 360° assessment over time although we gave feedback to all residents individually. A 360-degree feedback process should be repeated over time. That way, the intervention is truly a process aimed at increasing and improving clinical competencies and behaviors rather than a single event providing a one-time snapshot to the evaluators.

Conclusion

The 360° assessment of emergency medicine residents shows a high interrater reliability among those groups working closest with residents. Nurses rated residents lower than any other evaluator group. The 360° assessment was felt to be a positive contributor to emergency department functioning. Doing a 360° evaluation is time and effort consuming and thus may not be an ideal tool for larger programs.

Funding or financial support

None declared.

Conflicts of interest

None declared.