Assessing validity and reliability of Dundee ready educational environment measure (DREEM) in Iran.

BACKGROUND: If an institute is looking for improvement of its learning environment, a reliable and valid assessment tool is needed for measurement of the educational environment .The Dundee Ready Educational Environment Measure (DREEM) has been used in various studies to evaluate the educational environment. However, psychometric evaluations of the instrument seem necessary, for all known versions of the instrument. The aim of this study was to investigate the reliability and validity of Persian version of the DREEM in the major clinical wards in teaching hospitals affiliated to Iran University of Medical Sciences.
METHODS: This descriptive - analytical study, involved medical students (clinical stagers and interns) in 4 major clinical wards. In this study, DREEM questionnaire was reviewed in content, face validity and construct validity through confirmatory factor analysis. The reliability was calculated according to test - retest and the internal consistency was measured using Cronbach's alpha coefficient.
RESULTS: A total number of 267 questionnaires were completed by medical stagers (60%) and interns (40%) including 181 females and 82 males. The mean age of stagers and interns were 23.60 ± 1.27 and 25.45 ± 1.22 years, respectively. The total mean of the questionnaire was calculated as 96.15 (93.5375, 98.7547) out of 176, with 95% confidence interval. The face validity of the questionnaire was confirmed. The mean of content validity ratio (CVR) was calculated as 0.35, and 6 questions were omitted in this step. The content validity index (CVI) was 0.39. The reliability coefficient mean was 0.71. In confirmatory factor analysis five factors were confirmed that changed the orientation of some questions. The Cronbach's alpha coefficient of the whole questionnaire was obtained as 0.914.
CONCLUSION: The modified and validates DREEM questionnaire in Persian language with 44 items and appropriate psychometric attributes is capable of being used in assessment of clinical education environments in Iran.

Introduction

In 1998, the World Federation for Medical Education emphasized on the role of learning environment as one of the evaluation goals of medical education programs (1). Medical education environment is one of the extraordinary complexities sharing on complexities with working, specialized and training environments, however, with unique intellectual intricacies. This environment includes a host of areas that are specified by specific tasks assumed at specific times for specific objectives (2). Evaluating educational environments (both academic and clinical) is the key to achieve high-quality, student-centered curriculums (3). In other global studies, the need for measuring educational atmosphere has been noted as an important indicator in educational settings, and the necessity of its quantification has been emphasized (4). If we can determine the operational components in the educational environment, the institution atmosphere or in the programs and evaluate the attitudes of students and teachers toward them, thus, we would have a basis for their modification in order to facilitate the learning experience in relation to the educational objectives (5). The measurement action is an essential component for scientific research, whether in natural sciences, social sciences or in health sciences and it certainly plays a critical role in the health sciences (6). Quantitative measurement of learning environment requires a tool or a questionnaire. Choosing such an instrument should be based on the quality of measurement process suggesting the use of a special tool or instrument fitness for measuring educational environment. Quality or such psychometric characteristics are generally performed entitled as validity and reliability. A valid and reliable measure of learning environment leads to meaningful measurement of educational environment of an institution, and thus, a perfect tool to improve the learning environment is achieved (7). Since 1970s, some tool-oriented studies have been carried out to measure students' attitudes towards learning experiences and educational environments. Differences between educational environments have led to developing of various educational questionnaires. The first tool was Medical School Environment or MSLES, which was developed in 1970. Thence, a path to develop further tools in medical education appeared that according to a recent systematic review in 2010, its number has been reported as 19 cases (7, 8).

One of the benchmarks for measuring the environment and the atmosphere of education is Dundee Ready Educational Environment Measure (DREEM), which was developed in 1997 by Susan Roff et al. in the University of Dundee in Scotland using standard Grounded theory and Delphi procedures methods on nearly 100 educators on health specialty from around the world, and examining on over 1,000 students in different countries, it was validated in order to measure and assess the atmosphere of medical education. Employing a combination of qualitative and quantitative methods led to develop a multi-dimensional and an independent of particular culture tool (9). It was demonstrated that the DREEM have been accepted as a useful tool for gathering the feedback of strengths and weaknesses of educational environment in the educational institutions (10). Any measurement tool must have some properties to be useful for the purpose it has been built for. Specialists in measurement and evaluation for tests have considered a lot of features. The most important and emphasized ones by the professionals in examination survey include validity and reliability (11).

In Iran, the DREEM questionnaire has been frequently used to evaluate the clinical learning environment. Since the study of validity and reliability of the aforementioned questionnaire in Persian language has not done so far in the form of a specific project, we decided to examine the reliability and validity of the DREEM questionnaire in evaluation of educational environment of major clinical wards (Internal Medicine, Obstetrics & Gynecology, General Surgery and Pediatric) from the perspective of medical students (trainees and interns) in a few teaching hospital affiliated to Iran University of Medical Sciences.

Methods

This is a descriptive - analytical study. The study population was medical students (interns and stagers) from major clinical wards (Internal Medicine - Gynecology and Obstetrics - Pediatrics and General Surgery) of the Iran University of Medical Sciences. They were under education in teaching hospitals affiliated to the University (including HazratRasoolAkram, Firoozgar, Hazrat Ali Asghar and ShahidAkbarabadi hospitals). The total number in academic year of 2012-2013 has been estimated as follows: 122 students in Pediatrics; 155 students in Internal Medicine, 98 students in General Surgery and 90 students in Obstetrics and Gynecology. Data sampling was done using stratified random sampling method. Sample size was estimated according to the structure validity review using factor analysis method 5 times of the number of questions in the instrument (12), which was equivalent to 250 subjects. The number was selected randomly and proportional to the size of each groupin each stager and internship grades. First, the required permission was obtained in order to use the DREEM questionnaire developed by Susan Roff et al. at Dundee University, Scotland in 1997 and revised in 2005. Then, the questionnaire, which is originally in English, was translated into Farsi under the supervision of faculty advisors. Then, it was back translated by someone fluent in English and adapted to the original questionnaire, and the gap in compliance was corrected.

Study methodology in content and face validity

The questionnaire was given to 20 experts familiar with clinical education.

In the review of face validity, which is the questionnaire appearance and logical sequence of questions in the questionnaire and their legibility (11), a number of questions were rewritten.

The questionnaire content validity was investigated regarding two aspects of ratio (CVR) and index (CVI).

The content validity ratio (CVR) is direct linear conversion of panel members’ comments that have chosen the necessary option (13).

Two methods were used to determine the content validity:

Using the CVR formula (14) Thus, for each question was a CVR.

From the average of experts’ judgments

Given that each question was ranked on a 3-degree scale of Essential, Useful but not essential and Not necessary:

Essential: 2 Points

Useful but not essential: 0 Point

Not necessary: 1 Point

For each question, a mean score between 0 and 2 was obtained (13).

Determining a criterion for acceptance or rejection of the questions was as follows:

Acceptance of the question

1. If the CVR calculated in the formula was equal to 0.42 or higher, this number would be 16 based on Lawshe Table CVR values (14).

2. If the CVR calculated by the formula was between zero and 0.42 and the mean of judgments was equal to 1.1 or greater (In different studies, this range is different).

Rejecting the question, if the CVR value was less than zero and the mean of judgments was less than 1.1.

According to the results of the two methods, a number of questions were rejected; some were modified and the rest were accepted.

Also, reviewing the content validity index (CVI), which indicates the universality of judgments related to validity or applicability of the model, test or the final instrument, the overall CVI was calculated using the formula (13, 15).

Study of approach of consistency reliability

After conducting a pilot study, the correlation coefficients between test and retest questions were investigated using Spearman's correlation coefficient, and the questions with correlation coefficient tending toward zero were revised.

Evaluation method of internal consistency reliability

To evaluate the reliability of the internal consistency, the Cronbach's alpha coefficient calculation was used.

Study method of construct validity

The confirmatory factor analysis was used to examine construct validity during the following steps:

First step: Examining the data number proportionality using Kaiser - Meyer - Olkin (KMO) test and their homogeneity for factor analysis using Bartlett's Test of Sphericity

Second step: Extraction of the factors using principal components analysis method (16).

Statistical analysis

Data analysis was performed using SPSS software version 16. The required statistical methods include confirmatory factor analysis methods and Pearson correlation tests and Cronbach's alpha coefficient determination.

Results

267 questionnaires were completed by stagers and interns.

The questionnaire overall mean was obtained as 96.15(95% CI: 93.5375, 98.7547) from 176 that according to the interpretation of modified DREEM scores, positives points are more than negatives ones (17).

Interpretations of the factors are as follows:

Students' perception of learning: A more positive perception

Students' perception of teachers: Moving in the right direction

Students' academic self-perceptions: Feeling more on the positive side

Students' perceptions of atmosphere: A more positive atmosphere

Students' social self-perception: Not too bad (17).

Face validity

After the study of face validity of the questionnaire, the questions 7, 11, 12, 25, 27, 35, 42, 47 and 50 were amended again in writing.

Content Validity

The sum of the Content Validity Ratios (∑ CVR) was found 17.33 and Mean of CVR was computed 0.35.

The questions 9, 10, 17, 22, 39 and 42 were removed from the questionnaire.

The Content Validity Index (CVI) was found 0.39.

Reliability

The Mean of Test-Retest reliability of DREEM was 0.71 and The Consistency reliability was in an acceptable range.

Considering that the Cronbach'salpha coefficient has become more than 0.7, we can say the instrument has reliability regarding internal consistency. Meanwhile, the Cronbach'salpha coefficient of the factors was obtained in the range of 0.446 to 0.771.

Factor analysis

The KMO value was calculated as 0.892, which shows the proportion of samples number for factor analysis and the Bartlett's sphericity test became significant at significance error of 0.05 (p<0.001), indicating the homogeneity of the data for factor analysis testing.

After factors extraction, 5 factors with Eigen values higher than 1 were obtained, which covered the variance of 43.471%.

Discussion

After evaluation of the reliability and validity of the Persian version DREEM, the face validity of the questionnaire was confirmed, and the content validity index and the average of content validity ratio were obtained as 0.39 and 0.35, respectively. The mean of the CVR and CVI were low level totally. Perhaps we can say in review of content validity that the CVR value and the low average of judgments in some questions leading to their omission from the questionnaire were due to differences in the cultural context of the countries (e.g. questions 17 and 39). The overall content validity will be higher if the value of the CVI is closer to 0.99 and vice versa. Among the studies conducted in various countries, one study has been also done in Greece in 2010 by IDK Dimoliatis et al. entitles as validating the Greek translation of DREEM, in which the face and content validity of the questionnaire (qualitatively) has been studied. The face and content validity were optimized; however, due to no limitation of international results, no question was removed or added (18).

The instrument consistency reliability was better in interns than to trainees; but, both were in an acceptable range. The instrument consistency reliability was reported as average in Getulio R, De Oliveira Filho et al. study in Brazil in 2005 (19); also in Greece, the test and retest reliabilities were both obtained as 0.9. Examining the validity and reliability of Pakistani version the instrument conducted by JunaidSarfraz Khan et al. in 2011, the Spearman - Brown correlation coefficients (0.868) indicated the reliability of the analysis (20).

Review results of the internal consistency of the Persian version of the tool along with the results obtained in countries, including Brazil, Greece, Malaysia (21), China (22), Spain (23), Germany (3), New Zealand (8) and Pakistan with similar studies are given in the following table. It can be said that in all the translated versions of the instrument, the reliability of parallelism had a reasonable value Table 6.

Table 6

Comparison of Internal consistency of different versions among countries

Country	Total Cronbach’s Alfa	Subscales Cronbach's Alfa
Iran	0.914	0.446-0.771
Brazil	0.93	0.58-0.93
Greek	0.9	0.48-0.79
Malaysia	0.936	0.58 – 0.82
China	0.949	0.623-0.9
Spain	0.91
Germany	0.92 & 0.94
New Zealand	0.90 & 0.92
Pakistan	0.91

Reviewing the structure validity, the KMO test and Bartlett's Test of Sphericity provided acceptable values for performing factor analysis; in factor extraction using analysis of principal components, 5 factors were obtained. The model without rotation compared to the rotating models, indicated a better fitness with the core composition of the questionnaire based on our data. According to the questionnaire items content, it is recommended:

The questions 3-5-6-8-9-10-15-16-18-19-21-22-23-25-26-28-29-30-31-32-33-35-36-37-39-40-43 remain at the core axis and the 4-27-42 questions will be put at suggesting topics. (It should be noted that the number of items is provided after 6 questions deleting in the content validity review).

Structure validity has been also discussed in other studies. For example, in a study conducted in Brazil using the DREEM questionnaire, the construct validity was demonstrated. The factor analysis found 5 factors that explained the variance of 52% (19).

In a study in Malaysia in 2012 carried out by MuhamadSaifulBahriYusoff, the confirmatory factor analysis was used in examining the structure validity, which rejected the suggesting 5-factor structure fitness (21).

In a study conducted in 2009 in China by Jian Wang et al., 5 factors were found by analysis of principal components through Oblimin and Kaiser normalizing rotation method in performing factor analysis. The main factors names were kept, but some items in each area were changed. The five factors all had factor loadings greater than 1, and totally explained the variance of 52.186% (22).

In the Greek version of the instrument, factor analysis created significant areas; however, did not match everywhere with the original version (18).

In the Swedish version of DREEM, the exploratory factor analysis was also used due to poor fitness of the confirmatory factor analysis, which proposed 5 new factors for the tool (24).

In the Pakistani version of the tool, both exploratory and confirmatory factor analysis of areas were created; however, mismatch with the original version was high due to English to Pakistani cultural differences (20).

The German version of factor analysis showed that 5 dimensions compared to 5 areas considered by DREEM authors as necessary conditions are slightly divergent. Finally, the DREEM fitness was shown only just for the students but also for teachers in measuring learning environment (3).

According to the results of this study, the Persian modified version of DREEM consisting of 44 questions in 5 axes provided with the introduced combination in results and appropriate psychometric properties, has the capabilities of being used in assessment of our country’s medical and clinical training system, and can be used to assess the clinical training provided for medical students. The researchers are able to use the modified questionnaire for employing the questionnaire within domestic clinical environments. They can also use the original DREEM questionnaire translation to achieve the comparison with the original DREEM questionnaire in an international approach.

Conclusion

In some studies it is suggested that a few of items be deleted or edited due to those countries cultural issues or students conceptions, but in order to maintain international results no question is deleted.

Results of this study and similar studies in Sweden, Pakistan, China and Greece in oppose to previous claims shows DREEM questionnaire is not a questionnaire independent to culture.

According to the results of this study, Persian module of the DREEM including 44 questions in 5 domains with suitable valid specifications can be used in assessment of clinical educational environments in our country.

Researchers can refer to each of the questionnaire:

- The modified questionnaire for clinical educational environment surveys in our country.

- The translated original questionnaire in comparison with original DREEM in international approaches.

Table 1

Demographic data

Educational Hospital(n)	Ali AsgharRasouleAkramFirouzgarAkbarAbadi	53937447
Ward(n)	internal medicinepediatricssurgerygynecology	69883872
Age(year)	24.30±1.24Lower boundUpper bound	24.114924.4945
Gender (F / M)(n)	181 / 82
Stage (Stager / Intern) (n, %)	(161,60% )/ (106, 40%)

Table 2

Modified DREEM scores

Subscale	Mean score	95% Confidence Interval
Students' perception of learning(44 points)	21.80±6	(21.0788,22.5242)
Students' perception of teachers(36 points)	21.66±5.24	(21.0318,22.2941)
Students' academic self-perceptions(28 points)	14.90±4.55	(14.3509,15.4468)
Students' perceptions of atmosphere(40 points)	21.97±6.56	(21.1836,22.7640)
Students' social self-perception(28 points)	15.81±3.72	(15.3607,16.2572)
Total(176 points)	96.15±21.64	(93.5375,98.7547)

Table 3

CVR values Mean of judgment And the Accept or Remove results of each item

Items	CVR	Mean of judgment	Accept or Remove	Items	CVR	Mean of judgment	Accept or Remove
1	0.67	1.67	A	26	0.33	1.33	A
2	0.67	1.67	A	27	0.17	1.17	A
3	0.83	1.83	A	28	0	1	A
4	0	1	A	29	0.67	1.67	A
5	0.33	1.33	A	30	0.5	1.50	A
6	0.17	1.17	A	31	0.33	1.33	A
7	0.5	1.50	A	32	0.5	1.50	A
8	0	1	A	33	0.67	1.67	A
9	-0.5	0.50	R	34	0.33	1.33	A
10	-0.17	0.83	R	35	0.17	1.17	A
11	0.5	1.50	A	36	0.67	1.67	A
12	1	2	A	37	0.67	1.67	A
13	0.17	1.17	A	38	0.5	1.50	A
14	0	1	A	39	-0.17	0.83	R
15	0.17	1.17	A	40	0.33	1.33	A
16	0.83	1.83	A	41	1	2	A
17	-0.33	0.67	R	42	-0.17	0.83	R
18	0.83	1.83	A	43	0.83	1.83	A
19	0.33	1.33	A	44	0.83	1.83	A
20	0.67	1.67	A	45	0.67	1.67	A
21	0.33	1.33	A	46	0.17	1.17	A
22	-0.67	0.33	R	47	0	1	A
23	0.5	1.50	A	48	0	1	A
24	0.5	1.50	A	49	0.5	1.5	A
25	0.5	1.50	A	50	0	1	A

Table 4

Internal consistency reliability of DREEM

Subscale	Number of items	Cronbach's Alpha
Students' perception of learning	11	0.722
Students' perception of teachers	9	0.739
Students' academic self-perceptions	7	0.759
Students' perceptions of atmosphere	10	0.771
Students' social self-perception	7	0.446
Total	44	0.914

Table 5

Item factor loading

Scale in original Questionnaire	Suggested scale	The most factor loading in proportion to the 5 –scale domain	Scale 1	Scale 2	Scale 3	Scale 4	Scale 5	Item
1	1	0.552	0.552					1
2	1	0.472	0.472					2
5	1	0.498	0.498		-0.307			3
5	4	0.356				0.356		4
3	4	0.377	0.350			0.377		5
2	1	0.507	0.507			0.401		6
1	1	0.515	0.515					7
2	1	0.492	0.492	0.309				8
4	1	0.596	0.596					9
4	1	0.575	0.575					10
1	1	0.533	0.533					11
5	5	0.315					0.315	12
5								13
1	1	0.616	0.616					14
2	1	0.547	0.547					15
5	3	0.438	0.427		0.438			16
1	1	0.629	0.629					17
3	1	0.652	0.652					18
4	1	0.613	0.613	0.391				19
1	1	0.666	0.666					20
1	3	0.326	-0.515		0.326			21
3	1	0.445	0.445	-0.401				22
3	1	0.441	0.441	-0.415				23
5	5	0.323					0.323	24
2	1	0.683	0.683					25
4	1	0.499	0.499				-0.441	26
3	1	0.451	0.451				-0.423	27
2	1	0.476	0.476					28
4	1	0.485	0.485	0.301	0.429			29
4	1	0.577	0.577					30
4	1	0.542	0.542					31
4	3	0.318		-0.413	0.318			32
2	1	0.622	0.622					33
1	1	0.664	0.664					34
2	1	0.637	0.637					35
3	1	0.660	0.660	-0.303				36
4	1	0.655	0.655	-0.348				37
1	1	0.730	0.730					38
3	1	0.546	0.546					39
5	3	0.479			0.479		-0.322	40
1	1	0.519	0.519					41
1	4	0.484				0.484	-0.307	42
4	1	0.489	0.489					43
2	2	0.303		0.303				44