Assessment of resident physicians in professionalism, interpersonal and communication skills: a multisource feedback.

OBJECTIVE: To assess the internal validity and reliability of a multisource feedback (MSF) program by China Medical Board for resident physicians in China.
METHOD: Multisource feedback was used to assess professionalism, interpersonal and communication skills. 258 resident physicians were assessed by attending doctors, self-evaluation, resident peers, nurses, office staffs, and patients who completed a sealed questionnaire at 19 hospitals in China. Cronbach's alpha coefficient was used to assess reliability. Validity was assessed by exploratory factor analyses and by profile ratings.
RESULTS: 4128 questionnaires were collected from this study. All responses had high internal consistency and reliability (Cronbach's α > 0.90), which suggests that both questions and form data were internally consistent. The exploratory factor analysis with varimax rotation for the evaluators' questionnaires was able to account for 70 to 74% of the total variance.
CONCLUSION: The current MSF assessment tools are internally valid and reliable for assessing resident physician professionalism and interpersonal and communication skills in China.

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) recommends that residency programs to evaluate their trainees under six core competencies - patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism and system-based care 2. While the criteria of ACGME have been gradually accepted by hospitals as well as medical education organizations in China, interpersonal and communication skills are underrepresented in the medical education curriculum in China 4-5.

The ACGME suggests that assessment tools include written examinations, global ratings, 360-degree global ratings, and procedure/case logs 6. Multisource feedback (MSF), or 360-degree feedback, is used to assess physicians' competencies in a broad range of residency programs, including residency programs in family medicine and internal medicine 7-9. By reviewing MSF feedback, physicians can improve their communication skills with patients, modify their communication strategies with nurses, and improve the print material in their offices 10. Studies of MSF show that these reliable and valid feedback instruments (questionnaires) are acceptable to practitioners 11, 12. This study is intended to evaluate the psychometric characteristics of a multi-source feedback tool to assess residents' interpersonal and communication skills in China.

Method

The study received approval from the North China Center of Medical Education Development (NCCMED) and the Conjoint Health Research Ethics Board of the 19 collaborating hospitals.

Subjects

258 first year resident physicians participated at 19 hospitals in 11 provinces of China. These 19 hospitals are part of the “Reform Residency Training Program in China,” a program funded by the China Medical Board, which uses competencies and assessments derived from the Global Medical Education Requirements (Table 1). Each resident received descriptive data (mean and SD) on the results for both himself or herself and the whole group.

Table 1

The information of 258 resident physicians participated at 19 hospitals

Hospital	Resident'snumber	Gender		Division
		Male	Female	Cardio.	Diges.	ICU	ER	Endoc.	Immun.	Nephr.	Infec.	Neuro.	Phych.	Radia.	OPD	Hemen.	Others.
Hospital 1	3	0	3								2	1
Hospital 2	5	3	2			1		2		1						1
Hospital 3	18	13	6	2	2	4	2		1	1	2	2		1		2
Hospital 4	12	9	3		1			1			3	3		2		1	1
Hospital 5	6	2	4	2	2						2
Hospital 6	15	7	8	2	1		2	2		1	1	1	1	2		2
Hospital 7	32	12	20	8	3		1	3	2	6	2	1				4	2
Hospital 8	9	5	4	1	2		2	1		1	1					1
Hospital 9	17	11	6	1	2		1	2		1	5	4					1
Hospital10	21	7	14	4	3						3	1		5		1	4
Hospital 11	8	6	2	1	1						2						4
Hospital 12	25	15	10		2		3	4		2	4	1				2	7
Hospital13	17	9	8		3		2	2		3		2		1		2	2
Hospital 14	16	4	12		1					4	2	7				2
Hospital 15	22	15	7	1	1		3	2		1	4	2		1		2	5
Hospital 16	11	1	10		1	2				2	1	2					3
Hospital 17	11	1	10	4	1			6
Hospital 18	5	4	1			1						2				1	1
Hospital 19	4	3	1								3	1
Total	258	127	131	26	26	8	16	25	3	23	37	30	1	12	0	21	30

*: Hospital 1(Affiliated Hospital of Chongqing Medical University); Hospital 2(Affiliated Hospital of Shanxi Medical University); Hospital 3(Affiliated Hospital of Capital University of Medical Sciences); Hospital 4(Affiliated Hospital of Hebei Medical University); Hospital 5(Affiliated Hospital of Luzhou Medical College) ; Hospital 6(Affiliated Hospital of Nanjing Medical University); Hospital 7(Affiliated Hospital of Harbin Medical University); Hospital 8(First Affiliated Hospital of China Medical University); Hospital 9(Shengjing Affiliated Hospital of China Medical University); Hospital10(Forth Affiliated Hospital of China Medical University); Hospital 11(Affiliated Hospital of Chengde Medical College); Hospital 12(Affiliated Hospital of Binzhou Medical College); Hospital13(Affiliated Hospital of Zunyi Medical College); Hospital 14(Affiliated Hospital of Inner Mongolia Medical College) ; Hospital 15(Affiliated Hospital of Weifang Medical University); Hospital 16(Affiliated Hospital of Hainan Medical University); Hospital 17(First Affiliated Hospital of Guangzhou Medical College); Hospital 18(Second Affiliated Hospital of Guangzhou Medical University); Hospital 19(Third Affiliated Hospital of Guangzhou Medical University).

Questionnaires

The questionnaire for professionalism and interpersonal and communication skills from the Education Outcomes Service Group (EOS group) of the Arizona Medical Education Consortium was developed for attending doctors, residents (self- and peer evaluation), nurses, office staff, and patients. The goal of the EOS group was to provide assistance and support to program directors as they prepare to meet the ACGME outcome requirements. One item on the suggested list of methods for evaluating core competencies from the EOS group is the 360 Degree Evaluation. The assessment tool was refined from 2002 to 2006 by the EOS group to address new curricular elements and evaluation measurements recommended by the ACGME 13. The final questionnaire for attending doctors and residents consisted of 21, 21, and 21 items respectively (Table 2-4). The questionnaires for nurses (Table 5), office staffs (Table 6) and patients (Table 7) consisted of 26, 15 and 23 items respectively, with the same 5-point rating scale (1 = never to 5 = always). The questionnaires also included negative statements, such as “… is condescending to you or patients/families” and “… is abusive to you or patients/families rated with 1 as the perfect score (1 = never to 5 = always). All questionnaires provided respondents with the option of indicating whether they were able to evaluate the resident on the item.

Table 2

Attending doctor descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PR	ICS
A. PROFESSIONALISM
1. Demonstrates respect for the patient's
a. Culture	4.86	0.36	4.3	0.52
b. Gender	4.88	0.38	4.3	0.85
c. Disability	4.89	0.37	3.5	0.82
d. Sexual Orientation*	4.78	0.59	13.2	043
e. Age	4.88	0.38	7.8	0.83
f. Religion	4.79	0.55	3.1	0.38
2. Demonstrates respect for nurses†	4.96	0.19	3.5	0.71
3. Demonstrates respect for support staff†	4.90	0.31	3.9	0.72
4. Maintains confidentiality of patients and their families	4.80	0.42	3.5	0.59
5. Shows compassion for patients and their families	4.84	0.36	3.5	0.73
6. Seeks consultation/supervision when appropriate	4.81	0.45	3.1	0.41
7. Functions effectively as a member of the team	4.85	0.37	3.1	0.64
8. Demonstrates responsibility†	4.90	0.33	3.5	0.53
9. Completes assigned tasks	4.85	0.37	3.1	0.56
10. Manages personal stress responsibly	4.72	0.47	3.1	0.54
11. Answers pages in a timely fashion	4.81	0.39	3.5	0.54
B.INTERPERSONAL COMMUNICATION SKILLS
1. Communicates effectively with patients	4.80	0.42	3.5		0.74
2. Communicates effectively with patient's families	4.77	0.41	3.1		0.79
3. Communicates effectively with other health care professionals	4.85	0.36	3.5		0.68
4. Communicates referral information to patients	4.75	0.45	3.1		0.77
5. Maintains complete medical records†	4.90	0.32	4.3		0.56
Variance for each factor, % (total=70.87)				55.07	15.80

PR=professionalism; ICS=interpersonal and communication skills; SD=standard deviation

*: UE rate is more than 10%; †: high score.

Table 4

Resident peer descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PR	ICS
A. PROFESSIONALISM
1. Demonstrates respect for the patient's
a. Culture	4.88	0.34	1.9	0.74
b. Gender	4.88	0.35	2.7	0.79
c. Disability	4.89	0.36	2.3	0.78
d. Sexual Orientation*	4.76	0.57	11.6	0.59
e. Age	4.87	0.35	2.3	0.69
f. Religion	4.82	0.48	6.2	0.71
2. Demonstrates respect for nurses	4.87	0.33	2.3	0.44
3. Demonstrates respect for support staff†	4.92	0.29	1.9	0.62
4. Maintains confidentiality of patients and their families	4.85	0.38	1.9	0.69
5. Shows compassion for patients and their families†	4.91	0.29	1.9	0.56
6. Seeks consultation/supervision when appropriate	4.82	0.39	1.9	0.40
7. Functions effectively as a member of the team	4.87	0.36	1.6	0.55
8. Demonstrates responsibility	4.89	0.32	1.6	0.31
9. Completes assigned tasks	4.83	0.43	1.6	0.41
10. Manages personal stress responsibly	4.71	0.50	2.3	0.39
11. Answers pages in a timely fashion	4.85	0.42	1.9	0.40
B.INTERPERSONAL COMMUNICATION SKILLS
1. Communicates effectively with patients	4.81	0.39	2.3		0.59
2. Communicates effectively with patient's families	4.78	0.41	1.9		0.74
3.Communicates effectively with other health care professionals	4.80	0.41	1.9		0.74
4. Communicates referral information to patients	4.76	0.44	1.6		0.63
5. Maintains complete medical records	4.88	0.34	1.9		0.75
Variance for each factor, % (total=70.67)				51.98	18.69

PR=professionalism; ICS= interpersonal and communication skills

*: UE rate is more than 10%, †: high score

Table 5

Nurse descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PR	ICS
A. PROFESSIONALISM
1. Demonstrates respect for the patient's
a. Culture	4.86	0.47	2.0	0.82
b. Gender	4.87	0.49	3.5	0.82
c. Disability	4.87	0.46	2.3	0.84
d. Sexual Orientation*	4.74	0.64	10.1	0.69
e. Age	4.86	0.43	2.3	0.82
f. Religion	4.77	0.63	4.7	0.66
2. Demonstrates respect for nurses†	4.90	0.29	2.7	0.38
3. Demonstrates respect for support staff	4.88	0.36	1.9	0.40
4. Seeks consultation/supervision when appropriate	4.74	0.57	1.9	0.67
5. Functions effectively as a member of the team	4.81	0.48	1.9	0.66
6. Completes assigned tasks	4.85	0.42	2.7	0.29
7. Manages personal stress responsibly	4.74	0.51	3.5	0.29
8. Answers pages in a timely fashion	4.79	0.51	1.9	0.67
9. Is condescending to you or patients/families#	1.21	0.51	1.9	0.66
10. Is abusive to you or patients/families#	1.31	0.45	2.0	0.29
11. Respects patient's right to make choices regarding their care	4.68	0.70	1.9	0.29
12.Responds appropriately to the limitations imposed by the patient's illness	4.65	0.61	2.3	0.31
13. Responds in a timely fashion to nursing requests for help	4.79	0.44	1.9	0.37
14. Answers pages in a timely fashion	4.80	0.46	2.3	0.42
B. INTERPERSONAL & COMMUNICATION SKILLS
1. Communicates effectively with patients & patient's families	4.80	0.46	1.9		0.71
2. Communicates effectively with other health care professionals	4.81	0.43	1.9		0.80
3. Communicates referral information to patients	4.79	0.46	1.9		0.79
4. Maintains complete medical records	4.84	0.43	2.3		0.65
5. Listens to and considers what you have to say	4.81	0.45	1.9		0.54
6.Handles demanding interpersonal situations in a respectful and effective manner	4.79	0.53	1.9		0.61
7. Handles messages appropriately in a timely fashion	4.80	0.51	2.3		0.50
Variance for each factor, % (total=75.54)				37.72	37.82

PR=professionalism; ICS= interpersonal and communication skills

*: UE rate is more than 10%; †: high score; #: “negative” statements

Table 6

Patient descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PC	PR	ICS	SP
A. PATIENT CARE
1. Promotes health maintenance (talks about preventive care such as quitting smoking, weight control, alcohol, exercise, etc.)	4.63	0.58	1.2	0.61
2. Asks regularly about prescription and nonprescription medicine I am taking	4.72	0.52	1.6	0.62
3. Clearly explains my medical problem(s)	4.75	0.46	1.2	0.62
4. Clearly explains my treatment choices	4.75	0.48	2.7	0.78
5. Tells me about any side effects of the medicine	4.62	0.58	2.3	0.64
6. Tells me when to return for follow-up care	4.71	0.55	1.9	0.57
7. Clearly explains how to avoid my problem(s) in the future	4.73	0.48	2.3	0.67
B. PROFESSIONALISM
1. Demonstrates respect for my
a. Culture	4.82	0.45	1.6		0.68
b. Gender	4.77	0.52	2.3		0.73
c. Disability	4.79	0.57	5.8		0.82
d. Sexual Orientation*	4.63	0.80	10.1		0.80
e. Age	4.76	0.61	3.9		0.84
f. Religion	4.69	0.70	7.4		0.84
2. Is courteous to me	4.86	0.39	1.6		0.63
C. INTERPERSONAL & COMMUNICATION SKILLS
1. Listens to me	4.76	0.47	0.8			0.58
2. Spends enough time with me‡	4.61	0.61	1.6			0.50
3. Shows interest in my problems	4.58	0.60	2.0			0.60
4. Answers my questions thoroughly	4.70	0.48	1.6			0.52
5. Helps me with my fears and worries	4.72	0.50	1.6			0.76
6. Talks with me about treatment plans	4.69	0.50	2.0			0.74
7. Answers my messages in a reasonable amount of time	4.72	0.51	1.9			0.63
D. SYSTEMS BASED PRACTICE		0.61
1. Refers to specialists when needed	4.65	0.61	2.7				0.59
2. Suggests community resources for additional information and support‡	4.33	0.81	5.8				0.86
Variance for each factor, % (total=72.67)				12.11	44.59	8.78	7.19

PC=patient care; PR=professionalism; ICS=communication skills; SP=systems based practice; *: UE rate is more than 10%, ‡: low score

Table 7

Office staff descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PR	ICS
A. PROFESSIONALISM
1. Demonstrates respect for patients	4.88	0.36	3.1	0.43
2. Demonstrates respect for office staff/unit assistant†	4.90	0.33	3.1	0.30
3. Functions effectively as a member of the team	4.80	0.44	2.3	0.64
4. Demonstrates responsibility	4.86	0.39	2.7	0.61
5. Is condescending to you	1.54	1.21	3.5	0.88
6. Is condescending to patients and families	1.58	1.17	2.3	0.89
7. Is abusive to you	1.60	1.14	2.3	0.89
8. Is abusive around patients and families	1.57	1.20	2.3	0.86
9. Is courteous to patients and families	4.79	0.51	2.3	0.57
10. Responds in a timely fashion to requests for help	4.71	0.54	2.7	0.64
11. Answers pages in a timely fashion	4.77	0.46	2.7	0.76
B. INTERPERSONAL & COMMUNICATION SKILLS
1. Listens to and considers what you have to say	4.74	0.54	2.3		0.34
2. Handles demanding interpersonal situations in a respectfuland effective manner	4.79	0.47	2.3		0.59
3. Handles messages in a timely fashion	4.73	0.48	3.1		0.73
4. Accurately completes paperwork	4.78	0.44	3.5		0.81
Variance for each factor, % (total=74.62)				40.15	34.5

PR=professionalism; ICS= interpersonal and communication skills

†: high score “negative” statements; #: “negative” statements

Design

The participating residents were enrolled in the select hospitals in September 2007, and the survey was carried out in May 2008. All investigators were uniformly trained, and questionnaires were kept sealed and confidential when researchers dispatched them to evaluators. Residents were required to complete a self-evaluation. One attending doctor, 3 nurses, 7 patients, 2 resident peers and 2 office staffs were appointed by the education management department of the hospital as a group to answer questions on the survey for each individual corresponding resident physician. Global assessments of residents' performance were based on at least eight months of contact with each evaluator.

The results of collected questionnaires were inputted into our database and analyzed. We sent back the questionnaires and had them filled again when the data was found incomplete. All evaluations were conducted according to the same principles and guidelines as previous attempts.

Statistics analysis

Response rates were used to determine feasibility for each of the respondent groups. The percentage of unable-to-evaluate (UE) items, along with the mean and SD, was computed to determine the viability of items in the survey and the score profiles for every item. When the percentage of unable-to-evaluate items exceeds 10% on a survey, it suggests a need to examine the item for revision or deletion. We used exploratory factor analysis to identify the factors and numbers of factors for each questionnaire and to describe the relative variance accounted for by each factor and their coherence with each other. Reliability was assessed by use of Cronbach's alpha coefficient for each individual evaluated group, which enables an assessment of overall instrument stability. Statistical analysis was performed by using SPSS version 13.0 (SPSS Inc., Chicago, IL, USA) for Windows®.

Results

A total of 258 resident physicians participated, producing 258 self-assessments (100% return rate). A total of 258 (100%) attending doctor-assessments surveys were available for a mean of 1 per resident, 774 (100%) nurse-assessments surveys were available for a mean of 3 per resident, 1806 (100%) patient-assessments surveys were available for a mean of 7 per resident, 516 (100%) resident peer surveys were available for a mean of 2 per resident, and a total of 516 (100%) office staff surveys were available for a mean of 2 per resident.

Cronbach's alpha was calculated to determine the internal consistency and reliability of the questionnaires. There was an overall alpha of 0.913, 0.924, 0.930, 0.921, 0.901 and 0.933 respectively on attending doctor, self-evaluation, resident peer, nurse, office staff and patient surveys. The factor analysis identified 2 factors on the attending doctor, resident self, resident peer, nurse, and office staff surveys respectively: communication skills and professionalism, which accounted for 70.87% (Table 2), 71.01% (Table 3), 70.67% (Table 4), 75.54% (Table 5) and 74.62% (Table 6) of the variance respectively. There were 4 factors on the patient questionnaire: patient care, professionalism, interpersonal and communication skills, and system based practice, which accounted for 72.67% (Table 7) of the total variance.

Table 3

Resident self-descriptive statistics, unable-to-evaluate (UE) rates and rotated component matrix

Item	Mean	SD	UE	PR	ICS
A. PROFESSIONALISM
1. Demonstrates respect for the patient's
a. Culture	4.88	0.43	1.9	041
b. Gender	4.88	0.45	2.3	0.35
c. Disability	4.86	0.54	1.9	0.55
d. Sexual Orientation	4.70	0.74	9.7	0.53
e. Age	4.86	0.49	1.6	0.54
f. Religion	4.77	0.63	4.3	0.55
2. Demonstrates respect for nurse†	4.91	0.37	1.2	0.39
3. Demonstrates respect for support staff†	4.91	0.29	1.2	0.52
4. Maintains confidentiality of patients and their families	4.70	0.50	1.6	0.10
5. Shows compassion for patients and their families	4.83	0.40	1.2	0.38
6. Seeks consultation/supervision when appropriate	4.72	0.52	1.2	0.31
7. Functions effectively as a member of the team	4.83	0.38	1.2	057
8. Demonstrates responsibility	4.89	0.33	1.6	042
9. Completes assigned tasks	4.81	0.40	1.6	0.34
10. Manages personal stress responsibly	4.47	0.65	1.9	0.39
11. Answers pages in a timely fashion	4.80	0.41	1.2	0.52
B.INTERPERSONAL COMMUNICATION SKILLS
1. Communicates effectively with patients	4.70	0.50	1.9		0.68
2. Communicates effectively with patient's families	4.71	0.51	1.2		0.71
3. Communicates effectively with other health care professionals	4.78	0.46	1.2		0.71
4. Communicates referral information to patients	4.69	0.50	1.6		0.66
5. Maintains complete medical records	4.83	0.41	1.2		0.72
Variance for each factor, % (total=71.01)				32.29	38.89

PR=professionalism; ICS= interpersonal and communication skills

*: UE rate is more than 10%, †: high score.

Most items on the questionnaires could be answered by the respondents. As presented in Tables 2 to 7 the “Demonstrates respect for the patient's sexual orientation” item on the attending doctor (13.2%), resident peers (11.6%), nurses and patient (10.1%) survey had UE rates of more than 10%.

The scores for most items on the attending doctor, resident self, resident peer, nurse, patient, and office staff questionnaires were greater than 4. Low scores from the attending doctors were found in “Demonstrates respect for nurses”, “Demonstrates respect for support staff”, “Demonstrates responsibility” and “Maintains complete medical records” items. “Demonstrates respect for nurses” and “Demonstrates respect for support staff” items received low scores on the resident self -evaluations, and “Demonstrates respect for support staff”, “Shows compassion for patients and their families” and “Maintains complete medical records” items received low scores on the resident peer surveys. “Demonstrates respect for nurses” items on the nurse surveys, and the “Demonstrates respect for office staff /unit assistant” on the office staff surveys received high scores. The “time spent” and “community resources” items on the patient surveys received low scores.

Discussion

In this study, we assessed the psychometric validity and reliability of MSF assessment questionnaires which evaluated resident physicians in professionalism and interpersonal and communication skills. The evaluation was mandatory and the response rates were high, as expected. While most of the items could be answered, there were specific types of items on the attending doctor, resident self, resident peer, nurse and patient questionnaires that had UE percentages higher than anticipated. For attending doctors, resident peers, nurses, and patients, these tended to be in aspects of professionalism, specifically related to respect for patients sexual orientation. This is a sensitive subject in traditional Chinese culture and may explain the reticence of evaluators to score this domain.

Compared to traditional evaluation methods, the MSF, or 360-degree evaluation method, is more accurate and reliable 14, 15. All instruments had a high internal consistency and reliability (Cronbach's α>0.90), which suggests that both the questions and form data are internally consistent. The exploratory factor analyses with varimax rotation for the attending doctor, resident selves, resident peers, nurses, office staffs and patients questionnaires explained accounted for 70.87%, 71.01%, 70.67%, 75.54%, 74.62% and 72.67% of the total variance, respectively. The results showed that MSF assessment tools were internally valid.

Resident physicians did well in several aspects. Most resident physicians were conscientious and still learning how to become medical professionals. They respected the patient's disabilities and appreciated their colleagues' work. They completed the medical records as soon as possible.

However, results also showed that resident physicians did not pay much attention to spending enough time with their patients or suggesting community resources for additional information and support. It maybe that available community resources are not in a readily accessible and there are not a searchable format for the residents, such as an electronic database in China. In addition, resident physicians' busy daily work schedules, limiting the amount of time they have to research community health service programs and to share any pertinent information with patients in the large hospitals. Additionally, both Chinese doctors and patients tend to have lesser affinity to community health services and public health programs, as patients are often satisfied with just a visit to a larger hospital or institution. The resident physicians in this study infrequently inquired about community health services and public health programs.

There are limitations in the study. Data testing was limited to only resident physicians at the 19 collaborating hospitals which are relatively large teaching hospitals in China. The quality of physicians in these hospitals may be different from those work in smaller, community-based hospitals that are not affiliated with academic medical centers. We are not sure whether resident physicians in other parts of China would have similar performance profiles. There were only 1 attending doctor, 2 resident peer, 3 nurse, 7 patient and 2 office staff evaluators per resident physician in this study. Future research should increase the number of evaluators per resident to improve the reliability for the overall questionnaire. In addition, most teachers in China are usually not inclined to give low scores in evaluations like this, which may explain the relatively high scores 16.

A follow-up study to determine how the residents used their data, the changes they made as a result of the feedback, and their perceptions of this type of assessment is certainly warranted and was undertaken in recent resident training programs. The results of the study were collated in a second survey completed in 2009. This data will be used to provide formative feedback in a confidential manner to each resident, and suggestions for improvements will be made. The effects of such feedback and suggestions may then be reflected in the scores obtained during the following year's evaluation. In this way, a progressive improvement in professionalism and interpersonal skills and communication skills could be encouraged and measured.

Conclusions

The MSF or 360-degree feedback questionnaires for resident physicians may provide an internally valid and reliable way of assessing resident physician competencies.