Evaluation of the learning environment of urology residency training using the postgraduate hospital educational environment measure inventory.

OBJECTIVES: The educational environment plays a crucial role in the learning process. We aimed to evaluate the educational-environment perceptions of Saudi urology residents using the postgraduate hospital educational environment measure (PHEEM) inventory, and to investigate associations of their perception with stages of residency program, regions of Saudi Arabia, and main sectors of the health care system.
METHODS: We used PHEEM to measure the educational environment of Saudi urology residents. Respondents' perception was compared regarding different levels of residency training, regions of Saudi Arabia, and sectors of the health care system. Internal reliability of the inventory was assessed using Cronbach's alpha coefficient.
RESULTS: Out of 72 registered residents, 38 (53%) completed the questionnaire. The residents did not perceive their environment positively (77.7±16.5). No significant differences in perception were found among residents of different program stages or Saudi regions. Residents from different health care sectors differed significantly regarding the total PHEEM score (P=0.024) and the teaching subscale (P=0.017). The inventory showed a high internal consistency with Cronbach's alpha of 0.892.
CONCLUSION: Saudi urology residents perceived the educational environment as less than satisfactory. Perception of the educational environment did not change significantly among different stages of the program or different regions of Saudi Arabia. However, some sectors of the health care system are doing better than others.

Introduction

Urology residency training in Saudi Arabia is a 5-year structured program; it is hospital-based with central supervision from Saudi Commission for Health Specialties (SCHS) in which the 1st year is for General Surgery and Surgical Intensive Care rotation. Residents who complete their residency ultimately receive certification as a specialist.

The educational environment is an important measure which has a large impact on the satisfaction and success of medical education.1 The establishment of a supportive learning-oriented culture is of utmost significance in creating competent physicians.2 A measurement instrument of good quality to assess that learning environment is a prerequisite to allow accurate assessment of the learning environment and to identify areas requiring attention.3

The postgraduate hospital educational environment measure (PHEEM) is a self-administered 40-item inventory that assesses metrics of the level of autonomy, quality of teaching, and social support during the hospital-based training period for postgraduate students.4 PHEEM is a reliable and validated instrument to evaluate the quality assurance process as well as strengths and weaknesses within a certain educational environment.4,5

The objective of the present study is to evaluate the learning environment of urology residents and factors influencing their perception of this environment. The study also evaluates associations of educational-environment perception with stages of the residency program (R2, R3, R4, R5) and training in different regions of Saudi Arabia and different health care sectors. To our knowledge, this is the first study to evaluate the urology residency program since its implementation in Saudi Arabia.

Methods

This is a cross-sectional study using the English version of PHEEM instrument to assess the educational environment of the urology residency program. Respondents were asked to indicate their agreement with each statement using a five-point Likert-type scale, which ranged from strongly agree (4), to agree (3), unsure (2), disagree (1), and strongly disagree (0). The four negative statements (questions 7, 8, 11, and 13) were scored in reverse so that the higher the score, the more positive the environment. Information on sex, residency level, training region in Saudi Arabia, and main training hospital were also included as part of the questionnaire.

The perceptions of teaching subscale contains 15 items with a maximum score of 60; perceptions of autonomy subscale contains 14 items with maximum score of 56; and perceptions of social support subscale includes eleven items with a maximum score of 44. The summation of these scores has a combined maximum of 160 and minimum of zero. Higher levels of agreement were correlated with more beneficial educational environments. Global scores of 0–40 indicate a very poor educational environment, 41–80 indicate plenty of problems, 81–120 indicate more positive than negative but room for improvement, and 121–160 indicate an excellent educational environment.4

In 2011, after Institution Board approval, questionnaires were emailed to all residents registered in the urology training program by an independent third party, and the identity of the collected data was kept anonymous to the researchers. Global mean scores, for individual respondents, were calculated with missing values scored as 2 (the midpoint on this 0–4 scale). Scores for each item and domain were calculated and entered into a spreadsheet. Raw scores were coded and calculated.

Descriptive statistics were reported as median, mean, and standard deviation. The comparative statistics used the nonparametric method of Kruskal–Wallis6 to compare PHEEM-derived data from respondents in different levels of residency training (excluding R1 residents) as well as to compare residents from different regions of Saudi Arabia and from different main sectors of the health care system.

The reliability analysis was performed using the Cronbach’s alpha coefficient to measure the internal consistency of the whole questionnaire and each of the three subscales.7 Using the “alpha if item deleted”, Cronbach’s alpha was used to identify questions whose exclusion would improve the reliability.

Results

Out of 72 registered residents, 38 (53%) completed the questionnaire. Apart from one female, all were males with a mean age of 29.1±2.3 years. The number of residents in each residency year varied from four to 12. Out of a possible 1,520 responses to the 40 questions, only 26 (1.7%) were missing, suggesting that the questionnaire was simple and practical. Responses of residents in the 1st residency year (4) were excluded being involved in general surgery rotation year. Table 1 illustrates demographic details of the included respondents, their region of training in Saudi Arabia, and their main health care sector.

Table 1

Characteristics of study respondents

	Number	Percentage
Sex
Male	33	97.1
Female	1	2.9
Residency stage
R2	6	17.6
R3	9	26.5
R4	12	35.3
R5	7	20.6
Region in Saudi Arabia
Central	16	47.0
Western	7	20.6
Eastern	6	17.6
Northern	1	2.9
Southern	4	11.8
Sector of health service
Ministry of Health	11	32.4
National Guard Hospital	5	14.7
Armed Forces Hospital	5	14.7
Security Forces Hospital	3	8.8
University Hospital	6	27.6
Others	4	11.8

Median and mean scores (± standard deviation) for each item of the inventory for the three domains and for the overall inventory are summarized in Table 2. There were no statements which were highly rated (mean value >3), and 22 statements were poorly rated (mean value 2 or less).

Table 2

Mean and median of each question, overall and subscales scores

	Mean (SD)	Median
Q1 I have a contract of employment that provides information about hours of work	1.5 (1.1)	1
Q2 My clinical teachers set clear expectations	1.6 (0.9)	2
Q3 I have protected educational time in this post	2.0 (1.1)	2
Q4 I had an informative induction program	1.4 (1.2)	1
Q5 I have the appropriate level of responsibility in this post	2.0 (0.9)	2
Q6 I have good clinical supervision at all times	2.2 (1.0)	2
Q7 There is racism in this post	2.7 (1.0)	2.5
Q8 I have to perform inappropriate tasks	2.2 (1.1)	2
Q9 There is an informative Junior Doctors handbook	1.4 (0.9)	1
Q10 My clinical teachers have good communication skills	2.2 (0.9)	2
Q11 I am bleeped inappropriately	1.7 (0.8)	2
Q12 I am able to participate actively in educational events	2.6 (0.9)	3
Q13 There is sex discrimination in this post	2.5 (1.1)	2
Q14 There are clear clinical protocols in this post	1.4 (1.1)	1
Q15 My clinical teachers are enthusiastic	1.8 (0.8)	2
Q16 I have good collaboration with other doctors in my grade	2.5 (0.8)	2
Q17 My working hours conform to the new deal	1.7 (0.8)	2
Q18 I have the opportunity to provide continuity of care	2.5 (0.7)	2.5
Q19 I have suitable access to careers advice	2.0 (1.0)	2
Q20 This hospital has good quality accommodation for junior doctors, especially when on call	1.6 (1.4)	1
Q21 There is access to an educational program relevant to my needs	1.6 (1.0)	2
Q22 I get regular feedback from seniors	1.7 (1.0)	2
Q23 My clinical teachers are well organized	1.8 (1.0)	2
Q24 I feel physically safe within the hospital environment	2.4 (1.0)	2
Q25 There is a no-blame culture in this post	1.5 (0.8)	2
Q26 There are adequate catering facilities when I am on call	1.0 (0.9)	1
Q27 I have enough clinical learning opportunities for my needs	1.6 (0.7)	1
Q28 My clinical teachers have good teaching skills	1.6 (0.7)	2
Q29 I feel part of a team working here	2.5 (1.1)	3
Q30 I have opportunities to acquire the appropriate practical procedures for my grade	2.2 (0.9)	2
Q31 My clinical teachers are accessible	2.7 (0.9)	3
Q32 My workload in this job is fine	1.9 (0.9)	2
Q33 Senior staff utilize learning opportunities effectively	1.9 (0.9)	2
Q34 The training in this post makes me feel ready to be a consultant	1.7 (0.9)	2
Q35 My clinical teachers have good mentoring skills	2.1 (0.7)	2
Q36 I get a lot of enjoyment out of my present job	2.2 (1.0)	2
Q37 My clinical teachers encourage me to be an independent learner	2.4 (1.0)	2.5
Q38 There are good counseling opportunities for junior doctors who fail to complete their training satisfactorily	1.5 (0.8)	2
Q39 The clinical teachers provide me with good feedback on my strengths and weaknesses	1.6 (0.8)	2
Q40 My clinical teachers promote an atmosphere of mutual respect	2.1 (1.0)	1
Total score	77.7 (16.5)	75
Autonomy subscale	26.18 (6.5)	25
Teaching subscale	29.7 (7.7)	29
Social support subscale	21.9 (4.3)	22

Abbreviations: Q, question; SD, standard deviation.

There was no statistically significant difference (P>0.05) in perception scores of educational environment among residents in different training regions of Saudi Arabia or different stages of residency program.

Significant differences were identified between residents from different main sectors of the health care system regarding the total score (P=0.024) as well as the teaching subscale (P=0.017). Residents from National Guard health care sector responded more significantly positive to questions 2, 3, 12, and 21 of the teaching subscale; responded more significantly positive to questions 4 and 32 of the autonomy subscale; and responded more significantly positive to question 24 from the social support subscale (Table 3). They also responded significantly positively to questions 10 and 33 as did residents from Armed Forces and Security Forces health care sectors, respectively. Security Forces health care sector residents responded more significantly positively to question 29 of the autonomy subscale (Table 3).

Table 3

Comparison of PHEEM scores of respondents from different health care sectors

	Mean rank	P
Health sector total inventory score
Ministry of Health	11.32	0.024
National Guard hospitals	29.90
Armed Forces hospitals	21.10
Security Forces hospitals	16.67
University hospitals	17.50
Others	15.13
Health sector autonomy subscale
Ministry of Health	14.45	0.079
National Guard hospitals	29.80
Armed Forces hospitals	18.40
Security Forces hospitals	17.17
University hospitals	13.75
Others	15.25
Health sector teaching subscale
Ministry of Health	11.05	0.017
National Guard hospitals	29.70
Armed Forces hospitals	22.20
Security Forces hospitals	17.67
University hospitals	17.58
Others	13.88
Health sector social support subscale
Ministry of Health	10.32	0.079
National Guard hospitals	23.40
Armed Forces hospitals	20.20
Security Forces hospitals	16.00
University hospitals	22.67
Others	19.88
Q2
Ministry of Health	15.45	0.034
National Guard hospitals	28.00
Armed Forces hospitals	20.10
Security Forces hospitals	7.17
University hospitals	18.75
Others	12.63
Q3
Ministry of Health	18.77	0.022
National Guard hospitals	30.30
Armed Forces hospitals	15.40
Security Forces hospitals	12.17
University hospitals	11.58
Others	13.50
Q4
Ministry of Health	17.23	0.029
National Guard hospitals	29.80
Armed Forces hospitals	17.30
Security Forces hospitals	18.67
University hospitals	10.42
Others	12.88
Q10
Ministry of Health	10.77	0.035
National Guard hospitals	23.30
Armed Forces hospitals	24.50
Security Forces hospitals	15.33
University hospitals	21.83
Others	15.13
Q12
Ministry of Health	15.23	0.018
National Guard hospitals	28.40
Armed Forces hospitals	17.80
Security Forces hospitals	18.67
University hospitals	8.75
Others	22.00
Q21
Ministry of Health	9.86	0.009
National Guard hospitals	26.40
Armed Forces hospitals	17.20
Security Forces hospitals	21.67
University hospitals	16.75
Others	25.75
Q24
Ministry of Health	10.45	0.014
National Guard hospitals	28.50
Armed Forces hospitals	15.20
Security Forces hospitals	22.83
University hospitals	20.08
Others	18.13
Q29
Ministry of Health	11.45	0.048
National Guard hospitals	22.00
Armed Forces hospitals	25.80
Security Forces hospitals	15.50
University hospitals	21.50
Others	13.63
Q32
Ministry of Health	19.55	0.014
National Guard hospitals	28.00
Armed Forces hospitals	12.60
Security Forces hospitals	18.00
University hospitals	15.58
Others	7.38
Q33
Ministry of Health	11.68	0.044
National Guard hospitals	25.10
Armed Forces hospitals	20.60
Security Forces hospitals	25.83
University hospitals	18.33
Others	12.63

Abbreviations: PHEEM, postgraduate hospital educational environment measure; Q, question.

Cronbach’s alpha coefficient was computed to measure the internal consistency of the overall questionnaire and each of the three subscales. For the whole questionnaire, Cronbach’s alpha was 0.892. For the teaching and training subscale, Cronbach’s alpha was 0.847; for autonomy it was 0.750; and for the social support subscale it was 0.478. Questions 9, 17, and 32 of the autonomy domain, 16, 19, 20, 25, 26, and 36 of the social support domain, and 21, 27, and 37 of the teaching domain were not consistent with the rest of the scale and could be deleted (Table 4). On deletion of these factors from the reliability analysis, the Cronbach’s alpha coefficient increased for the entire inventory and for the social support domain to 0.911 and 0.564, respectively. Teaching and autonomy domains showed a little elevation to 0.867 and 0.757, respectively.

Table 4

Reliability analysis of the overall questionnaire

	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total correlation	Cronbach’s alpha if item deleted
Q1	76.24	260.367	0.341	0.890
Q2	76.12	258.955	0.478	0.888
Q3	75.71	253.729	0.506	0.887
Q4	76.38	253.880	0.461	0.888
Q5	75.74	256.564	0.542	0.887
Q6	75.50	256.924	0.491	0.888
Q7	75.09	259.053	0.410	0.889
Q8	75.56	260.981	0.331	0.890
Q9*	76.32	268.953	0.119	0.893
Q10	75.53	255.651	0.579	0.886
Q11	76.00	265.333	0.307	0.890
Q12	75.18	255.544	0.574	0.886
Q13	75.26	260.140	0.348	0.890
Q14	76.29	254.759	0.508	0.887
Q15	75.91	257.780	0.619	0.887
Q16*	75.21	268.411	0.160	0.892
Q17*	76.00	268.606	0.152	0.893
Q18	75.21	264.532	0.392	0.890
Q19*	75.76	267.034	0.172	0.893
Q20*	76.15	259.038	0.289	0.892
Q21*	76.15	264.796	0.247	0.892
Q22	76.09	256.143	0.538	0.887
Q23	75.94	255.390	0.588	0.886
Q24	75.32	255.074	0.515	0.887
Q25*	76.24	267.337	0.200	0.892
Q26*	76.74	273.655	–0.035	0.895
Q27*	76.18	265.544	0.301	0.891
Q28	75.71	258.093	0.546	0.887
Q29	75.21	254.532	0.530	0.887
Q30	75.59	258.977	0.496	0.888
Q31	75.00	256.788	0.550	0.887
Q32*	75.85	266.978	0.179	0.893
Q33	75.82	253.604	0.691	0.885
Q34	76.09	261.356	0.378	0.889
Q35	75.68	259.801	0.587	0.887
Q36*	75.53	270.378	0.057	0.895
Q37*	75.32	264.286	0.254	0.892
Q38	76.26	264.685	0.316	0.890
Q39	76.18	259.241	0.509	0.888
Q40	75.65	252.478	0.652	0.885

Note:

Questions 9, 16, 17, 19, 20, 21, 25, 26, 27, 32, 36, and 37 are not consistent with the rest of the scale.

Abbreviation: Q, question.

Discussion

The educational environment plays a crucial role in the learning process. PHEEM can be used to identify strengths and weaknesses of a medical residency program.5 It has been used to assess the educational environment of the residents in different studies.5,8–13 We used PHEEM as a tool to evaluate the educational environment of the postgraduate urology residency program in Saudi Arabia for the first time since its implementation in 2000.

In this study, residents in the final stages of the residency program (R4 and R5) represented more than one-half of the respondents. Although it is expected that residents in final residency stages have more ability to judge the program, the stage of the program did not seem to have an effect on how residents perceive their educational environment (P>0.05). Similar results were also seen in another study.11

Likewise, although residents from the central region of Saudi Arabia represented the majority of the respondents (47%), training region in Saudi Arabia had no effect on residents’ perception of their educational environment. A study of rotating interns in Australia showed that the rural performed better than urban locations on the teaching, autonomy, and support subscales.10 However, only one urban hospital was approached to be the comparator site and it is possible that adding more tertiary urban hospitals could change the results. We believe that no great cultural differences exist between different main regions in Saudi Arabia, especially given that all current urology training program hospitals are urban hospitals.

Residents from different main sectors of the health care system differed significantly regarding the total score (P=0.024) and the teaching subscale (P=0.017). National Guard health care sector residents showed a significantly higher satisfaction with their teachers, safety of their environment, and access to and available time for education. Similarly, residents of Security Forces health care sector significantly perceived more involvement in team work (Table 3). This mostly represents local differences among different sectors of the health care system as the urology curriculum is universally applied all over Saudi Arabia.

Our study showed no overall real positively rated points (mean score of 3.5 or more), while 22 statements were poorly rated, with a mean value of 2 or less, and should be examined more closely as they indicate problem areas.4 The lowest recorded score was 1 for item 26 (There are adequate catering facilities when I am on call), a situation that can be easily solved by proper training-site management. Poorly rated questions included questions 1, 4, 5, 11, 14, 17, 32, and 34 from the autonomy domain, which assess the presence of clear clinical protocols in the post, information provided by the contract regarding working hours, informative program, and work load in the job. Furthermore, the residents had doubts that the training in the post would make them ready to be specialists/consultants (question 34, mean score 1.7±0.9). Other poorly rated questions included questions 2, 3, 9, 15, 21, 22, 23, 27, 28, 33, and 39 from the teaching domain, which mainly define the role of teachers, ie, their teaching skills, enthusiasm, organization, and their ability to set clear expectations, as well as their ability to provide their residents with feedback on their strengths and weaknesses. Poorly rated social domain questions included questions 19, 20, 25, 26, and 38 which involve mainly hospital accommodation and catering facilities as well as no-blame culture. Similar results were shown in 101 residents in the Saudi family medicine training program.13

The total inventory score in this study was 77.7 (±16.5) which, according to the criteria proposed by Roff et al,4 reflects an educational environment with plenty of problems. Perception of the role of autonomy (26.2±6.5) showed a negative view of one’s role. Perception of teaching by the residents (29.7±7.7) indicated that teachers are in need of some retraining. The social support perception (21.9±4.3) reflected an unpleasant environment. Such results should be taken into account by curriculum planners as they consider improvements to the urology training program. Level of supervision provided by superiors, flexibility and freedom in the job, and level of participation in decision making are important areas of development and enhancement. Inclusion of residents’ perceptions of their training experience as a part of the quality assurance for accreditation might be beneficial. Course enrollments and free access to electronic journals are fundamental educational sources. Supervisors should be supportive and provide effective supervision and mentoring for residents.

Cronbach’s alpha scored high at 0.892 for the 40 statements and 0.847 for the teaching domain. It scored good at 0.750 for the autonomy domain and poor at 0.478 for the social support domain. The overall inventory showed a high internal consistency and reliability. A high reliability of PHEEM questionnaire was also shown in different studies with feasible sample sizes.4,5,9–14 However, when our data was analyzed to exclude each question in turn, using the “alpha if item deleted”, 12 questions were found to be inconsistent with the rest of the scale and could be deleted (Table 4). Furthermore, the Cronbach’s alpha coefficient for the entire inventory and its three subscales increased on deletion of these items, suggesting reduction of the questionnaire items provided that the same results could be reproduced in a larger sample size. Items 1, 7, 10, 13, 15, 25, and 28 were previously found to be uncorrelated with total score in other reports.14

Our study is limited by the small sample size and rarity of females in the Saudi urology residency program, which precluded comparison of sex perception. However, we believe that this study, being the first of its nature, represents a good chance to evaluate the current urology training programs and could help in improving them.

Conclusion

PHEEM survey is an applicable and valid instrument for assessing the educational environment of the urology residency program in Saudi Arabia. Urology residents perceived the urology residency program educational environment as less than satisfactory. Perceptions of the educational environment did not change significantly among different stages of the program or different training regions of Saudi Arabia. Some sectors of the health care system are doing better than others. The residents’ educational environment needs close attention in all of its domains. Clear clinical protocols, work load, working hours, quality of teaching supervision, and supportive hospital environment are areas which need development and enhancement. It is hoped that the information provided by this study could be used to improve the educational environment of the urology residency program in Saudi Arabia.