Psychometric analysis of the ambulatory care learning education environment measure (ACLEEM) in Iran.

BACKGROUND: Examining educational environment (academic and clinical) by means of a valid, reliable and comprehensive questionnaire is a major key in achieving a highly qualified student - oriented curricula. The Persian translation of Ambulatory Care Learning Education Environment Measure-ACLEEM questionnaire has been developed to support this goal, and its psychometrics has been explored in this administration in teaching hospitals affiliated to Tehran University of Medical Sciences.
METHODS: This descriptive - analytical study involved medical residents in four major clinics. In this study, the ACLEEM Questionnaire was conducted after translating and retranslating the questionnaire and examine the face and content validity, construct validity, test retest reliability and internal consistency coefficient.
RESULTS: In this study, 157 out of 192 residents completed the questionnaire (response rate 82%). The mean age of the residents was 31.81 years .The final mean of the questionnaire was calculated as 110.91 out of 200 (with 95% confidence interval). Test - retest stability of the questionnaire was between 0.322 and 0.968. The face validity of the questionnaire was confirmed. The content validity ratio was 0.64; and content validity Index was 0.78. In Exploratory factor analysis, eight factors were confirmatory that changed the orientation of some questions. The Cronbach's alpha coefficient of the whole questionnaire was 0.936.
CONCLUSION: According to the data, the Persian version of the ACLEEM questionnaire has sufficient psychometric reliability and validity to be used for conducting research, teaching and practicing the educational learning environment in ambulatory care in Iran.

Introduction

Since the 1970s, studies have been conducted to report on students' attitudes towards educational and learning environment. In 2010 Soemantri et al. (1) reviewed 178 reports related to the topic of learning environment and found that 31 studies were conducted onmedical environments. They identified several instruments that measure educational environment from students’ perspectives including the Anesthesia Theatre Educational Environment Measurement (ATEEM), the Surgical Theatre Educational Environment Measure (STEEM), and the Postgraduate Hospital Educational Environment Measure (PHEEM). Among thetools, PHEEM is specifically related to medical training atexpert level. All tools including PHEEM are questionnaires designed toassess the educational environment at hospital wards. However, none of these instruments focused on learning and teaching in ambulatory care environments. Fortunately,the Ambulatory Care Educational Environment Measure (ACLEEM) was developed in 2013 (2).

Clinical medical training mainly provides primary care. Since in Alma Ata statement, it is increasingly a known level in health care system, and currentlyit can be seen as a slogan in some documents more than the past like 2008 world health report (3). Some evidence shows that health systems with primary care havestronger health, better results, and less health cost (4-6).Moreover, international systems have taken similar procedures in to consideration, and medical schools have been advised to focus their curricula on conducting primary health care.

On the other hand, the procedure of the guarantee of residents training is increasingly important (7). The educational environment isone of the aspects of assessing the quality of training programs andpresenting information in various fields such as atmosphere, feedback, inspection at hospitals and ambulatory care wards (8).

The relationship between the primary health care with ambulatory care is undeniable; and in this condition,a lack of a special tool to assess the educational environment of ambulatory care has been reported prior to2012 (1).

ACLEEM as the first developed questionnaire was designed to measure the educational environment of ambulatory care. This instrument was designed and developed to measure the health service of graduates and post graduates by ArnoldoRiquelme et al. in 2012.

The ACLEEM is a 50-item inventory containing the following subscales: Quality of clinical teachers (12 questions), clinical activates and patient care (11 questions), protected time for non-clinical activities (5 questions), infrastructure (6 questions), clinical skills (6 questions), assessment and feedback (4 questions), Information, communication and technology (3 questions), and clinical supervision (3 questions).Each item on the ACLEES is scored on a 5 point Likert scale ranging from 0 (strongly disagree) to 4 (strongly agree), giving a possible total score of 200.

Since the study of validity and reliability of the aforementioned questionnaire in the Persian language has not been done so far in the form of a specific project, we decided to examine the psychometric properties of thePersian versionof the ACLEEM in Iran to provide a valid instrument toassess educational environment of major clinical wards (Internal Medicine, Obstetrics & Gynecology, General Surgery and Pediatric) in teaching hospitals affiliated to Tehran University of Medical Sciences.

Methods

This was a cross-sectional descriptive – analytical study. The target population of the study was medical assistants in internal medicine, gynecology & obstetric and pediatrics-general surgery clinics of clinical training centers affiliated to Tehran University of Medical sciences(HazratRasoolAkram, Firoozgar, Hazrat Ali Asghar, Shahid Akbar Abadi and Emam Khomeini Hospitals).

All residents in teaching hospitals affiliated to Tehran University of Medical Sciences were invited to participate in the study. Sampling was conducted through census.The ACLEEM questionnairewas used for datacollection, and it was translated into Persian and retranslated according to the back-translation method.The Dundee Ready Education Environment Measure (DREEM) model (9) is used forthe number of test items, and the points;its interpretation is as follows: 0-50 points (very poor) 51- 100 points (plenty of problems), 101 – 150 points (more positive than negative), and 151 – 200 points (excellent).

Translation

After obtaining permission from the author (Arnold Riquelme) the ‘forward-backward’ procedure was applied to translate the Ambulatory Care Educational Environment Measure (ACLEEM) from English into Persian. First the research team translated the questionnaire. Then an English-language expert provided the back-translation. Finally,the final versionwas compared withthe original questionnaire and was confirmed by one of the ACLEEM co-developers. Then this provisional version was tested for psychometric properties.

Determining the Face Validity of the Questionnaire

To determine the face validity of the questionnaire, the questions should be logical and should also matchwith the characteristics of the respondent. Moreover, logical frequency,suitable printing and clear and brief introduction are important inimproving the face validity of the questionnaire. Therefore, 20 experts in Iranian clinical training were asked to evaluate the face validity of the questionnairefor the Iranian context of ambulatory care learning and teaching (10).

Determining the Content Validity

Two dimensions of the content validity, content validity ratio and content validity index,wereexamined. In the former, we examined how useful and essential the test items were,and in the later, the simplicity, clarity, and relevance between the test items wereexamined. Toidentify the panelists and selecta pattern to determine the content validity ratio was developed by Lawshe (11) was used and 20 experts familiar with Iranian clinical training and teaching the medical training courses were asked to evaluate each item to indicate that an item is “essential”, useful but not essential”, “not necessary”.

QuantifyingContent Value Ratio

Two methodswereused to determine CVR.

1) Greater levels of content validity exist as larger numbers of panelists agree that a particular item is essential. Using this assumption, the followingformula wasused:

2) Computing the mean of CVR: Each item wasgiven a value based on a 3 part domain, “Essential” (2 points), “useful but not essential” (1 point), not necessary (0 point) and then the mean is computed.

Determining the Criterion to Retain or Reject the Item

a) Non-conditional retention of the test items whose CVR is equal or larger than 0.42.

b) Conditional retention of the test items whose CVR is between 0 and 0.42 and their mean is larger than 1.1.

c) The items with CVR smaller than 0 and with the mean less than 1.1 indicate that less than 50% of the panelists have identified the item as “essential”.

Determiningthe Content Validity Index

CVI was determined according to Waltz and Bausell's (12)content validity index to determine the amount of relevance, simplicity, clarity of each item in the questionnaire in a 4- part Likert (13) scale for each of the items; for example, for relevance (irrelevant (1), somewhat relevant (2), relevant (3) and entirely relevant (4)). In this study,the content validity index was computed by the division of the number of panelists with grade 3 and4 to the total number of the panelists (14).Hyrkaset al. (15)found0.79 and higherfor the retention of the items according to CVI. At the next stage, the mean of the content validity index was computed for each item and all items.

Investigating Consistency Reliability

To determine the reliability consistency, the questionnaire was given to 10 residents with similar traits to the population in the study that was analyzed with computing Cronbach's alpha coefficient. Then test – retest reliability was used to determine the consistency of the test by administering the test to the same 10 residents after a 10- day interval. Spearman’s correlation coefficient was used that led to the revision of the test items whose correlation coefficient tended to become 0(zero).

Investigating Internal Consistency of Reliability

This refers to the degree of consistency of an item with other items of the questionnaire. For this test, Cronbach’s alpha coefficient wasused.Obviously, if the Cronbach's alpha index was closer to 1, then the internal consistency of the questions wouldbe more. Cronbach advice reliability coefficient of 45% is less, 75% is moderate and acceptable, and 95% is high (16).

Investigating Construct Validity

Exploratory factor analysis was used to investigate construct validity:

Step 1: Investigating the sampling adequacy and data homogeneity

1. Kaiser- Meyer- Olkin (KMO) measures the sampling adequacy;its index ranges from 0 to 1. It is advised that the minimum amount for the factor analysis be more than 0.6.

2. Bartlett's test is used to measure the homogeneity of the test items and it should be significant for the factor analysis (p<0.05)

Step 2: Extracting the factors by using the method of the analysis of main variables. Kaiser's criterion statesthat only factors with 1 or more are used for more investigation.

Step 3: Rotation factor by the use of varimax (17).

Data Collection Instrument:In this study,the instrument used for data collectionwas ACLEEM. The first part of the questionnaire includeddemographic data such as the institute name, the specialty group name, the year of education, age, gender, marital status; and the second part of the questionnaire contained50 questions that used a five point scale from 0 (lowest) to 4 (highest). The maximum total pointwas 200. The Likert scale includedstrongly agree (4 points), agree (3 points), unsure (2 points), disagree (1 point), strongly disagree (0 points) and the items 24 and27 werecalculated oppositely.

Statistical Analysis

Data wereanalyzed using SPSS 16 statistical tests.Content validity, face validity and construct validity were appliedto evaluate the validity of the questionnaire; Cronbach's alpha coefficient was utilized to determine the internal consistency, and test- rest was used to determine thereproducibility of the questionnaire.

Results

Demographic Data

From the 192 residents, 157 responded to the study questionnaire (82% response rate). Gynecology residents had the highest response rate (91%).

The findings of this study showed that among the 157 respondents aged 26-45 years, 51% were between 26-30 years old. The mean age of the residents was 31.81±4.37 years.The third year residents constitutedthe highest percentage of the respondents (30%) (Table 1).

Table 1

Frequency Distribution of Demographic Variables

Clinic(n)	Male		female		Total
	Number	Percent	Number	Percent	Number	Percent
Surgery	15	55.9%	8	10.5%	23	65.14%
Pediatrics	8	10.5%	15	55.9%	23	65.14%
Gynecology	0	0%	40	48.25%	40	48.25%
Internal medicine	28	83.17%	43	39.27%	71	22.45%
Total	51	48.32%	107	52.67%	157	100%

ACLEEM Scores

The final mean of the questionnaire was calculated as 110.91 out of 200,and the highest mean belonged to clinical skills (65%), clinical activity and patient care (61%); and the lowest mean to clinical supervision (43%) (Table 2).

Table 2

ACLEEM Scores

Subscale	Mean score	SD	Percent of total
Quality of Clinical Teachers (48 points)	28.59	8.14	59%
Clinical Activities & Patient Care (44 points)	26.93	7.1	61.2%
Protected Timefor Non-Clinical Activities (20 points)	9.58	5.22	47.9%
Infrastructure (24 points)	11.63	4.76	48.45%
Clinical Skills (24 points)	15.55	4.18	64.79%
Assessment &Feedback(16 points)	7.80	3.74	48.75%
Information, Communication &Technology(12 points)	5.60	2.70	46.66%
Clinical Supervision(12 points)	5.18	2.08	43.16%
Total(200 points)	110.91	28.32	55.45%

Face Validity

After examining the face validity of the questionnaire, the following questions were revised: 3, 5, 10, 14, 16, 29, 5.

Content Validity

The content validity ratio was 0.64; and thefollowing test items were corrected and accepted:2, 8,10,12,26, 32, 36, 37, 38, 41, 42, 47, 48‏.

The mean of the content validity index (CVI) 0.74 (simplicity 0.78, clarity 0.77, relevant 0.69) was computed. The test items 4, 5, 10, 11, 12, 26, 30, 33, 38 were amended and retained

CVI = 0.74

Reliability

For the pilot study, the questionnaire was distributed among 10 residents with similar traits to the real samples. Cronbach's alpha of the 50 item questionnaire was 0.697. Test – retest was carried out on 10 participantswithin a 10 – day interval; its reliability was between 0.323 and 0.968.

In the main study, the Cronbach's alpha of the 50 item questionnaire was 0.936, demonstratingquality of clinical teachers with the highest score (0.917) and the clinical supervision with the lowest score (0.110) (Table 3).

Table 3

Consistency Reliability of ACLEEM

Subscale	Number of items	Cronbach's Alpha Quality of Clinical Teachers	12	0.917
Clinical Activities & Patient Care	11	0.874
Protected Time for Non-Clinical Activities	5	0.648
Infrastructure	6	0.643
Clinical Skills	6	0.865
Assessment & feedback		0.684
Information, communication& technology	3	0.603
Clinical supervision	3	0.110
Total	50	0.936

Construct Validity

The final version of the questionnaires was given to the study participants. The result of examining the construct validity using the factor analysis method is as follows: KMO≤ 0.853.

Bartlett's test of sphericity: (p=0.0001) (Bartlett’s test was significant at p<0.05).

After factors extraction, 12 factors explained 68% variance of all options, with varimax rotating for eightfactors describing59.29% of the variance. The sub-groupsshowed an acceptable homogeneity. The exploratory factor analysis revealeda retainable value. The place of some items changed with preserving the name of some original factors. Item 16 from domain 2 to 1, item 33 from domain 4 to 7, item 45 from domain 7 to 3 were moved.

Discussion

Datacollectionis one of the essential partsof any study;aquestionnaire as an instrument is evaluated for its reliability and validity. A valid and reliable evaluation tool in educational setting leads to a significant assessment of an institute.Thus, a suitable tool improvestheeducational environment (1).

In this study, the main purpose was to determine the reliability and validity of ACLEEM among residents in teaching hospitals affiliated to Tehran University of Medical Sciences.

In this study, after conducting the processes of translation, retranslation, correction and amendments, the translated versionwas given to 20 experts familiar with clinical education. When examiningthe validity and reliability of the questionnaire,we found that ithad no face validity in a small series of itemsso thiswas adjustedaccording to the medical expert’s comments. The content validity ratio was 0.64, the mean of the content validity index (CVI) 0.74 (simplicity 0.78, clarity 0.77, relevant 0.69) was also computed.The means of the CVR and CVI were moderate and acceptable. However,their value in some questions waslow due to differences in the cultural context of the countries (e.g., questions 41 and 48) so they were amended again in writing.In general, the higher the content value, the closer the CVI wasto 0.99. The questionnaire was evaluated by the medical training experts for its face validity and content validity.

In thisstudy, the reliability and Cronbach’s alphas coefficient of 50 – item were in an acceptable range totally. However, in some questions it waslow, and this may have been due to the small sample size in this study.

With respect to the construct validity,the subgroups showed good homogeneity, and the exploratory factor analysis showed a retainable value for the model 8 factor.The rotating models, compared withthe model without rotation, indicated a better fitness with the core composition of the questionnaire based on our data. We preservedthe name of some original factors, but we changed the place of some items and we recommendthat all questions remain at the core axis and questions 16, 33, 45 will be put at suggesting topics.

In this study, the total means score of all the eightdomains was 110.91 out of 200 (55 %); andconsidering the interpretation guide, positive points weremore than the negative. The Quality of Clinical Teachers domains’ score means was the highest (64.79%)and Clinical Supervision Domains’ score means was the lowest. This may indicate that despite the ability of teachers at the basic there is a fundamental flaw in evaluation system.

As regards reliability and validity of this tools have not examined yet in Iran and other countries; was not comparable with similar studies.

Finally, we conclude that the ACLEEM is a multidimensional, valid and highly reliable instrument to measure the educational environment in postgraduate ambulatory settings. Furthermore,according to the result of this study, the Persianversionof ACLEEM, with50 questions in 8 factors,can be used to assessthe medical and clinical training system inIran. Moreover, for the future studies, it is highly recommended toassess the quality of residents’ training programs andprovide them with sufficient information in various fields such as atmosphere, feedback, inspection at hospitals and ambulatory care wards.

Due to the small sample size in this study, it is suggested to use this instrument and to repeat factor analysis with a larger population to develop and rule outthe trivial problems.

Conclusion

The finalized Persian version of the ACLEEM questionnaire isa validandreliable instrument that can be used for research, and educational and practical purposes to investigate the educational environment of ambulatory care.

Acknowledgements

We would like to appreciate the Research Center for Medical Education atIran University of Medical Sciences for financially supportingthis project.