Clinical Learning Evaluation Questionnaire: A Confirmatory Factor Analysis.

INTRODUCTION: The Clinical Learning Evaluation Questionnaire (CLEQ) is a multidimensional, reliable instrument designed to measure the effectiveness of the clinical learning environment for undergraduate medical students. This study seeks to measure and examine the underlying construct along with the latent variables by conducting a confirmatory factor analysis, using structural equation modeling (SEM) so that the instrument can be utilized as an evaluation tool for the continuous improvement of educational environments and curricula.
METHODS: A cross-sectional study was carried out on 185 third- and fourth-year medical students. A confirmatory factor analysis was conducted, beginning with principal component analysis for standardized factor loadings, using varimax rotation in SPSS to explore the underlying construct of items. The constructs to which each item was tied were determined, and then the data were run through AMOS to assess construct validity through item reduction based on the modification indices, and estimates were made of the standardized residual covariance of each item in order to determine the best model fit.
RESULTS: A total of 185 students completed the CLEQ Inventory. The original six-factor structure of the CLEQ did not achieve model fit (X2=1587.475, RMSEA=0.092, RMR=0.146, GFI=0.651, AGFI=0.601, CFI=0.728, NFI=0.626). However, the suggested four-factor model of CLEQ displayed good model fit with the improvement of values (X2=86.184, RMSEA=0.052, RMR=0.062, GFI=0.903, AGFI=0.865, CFI=0.951, NFI=0.871). Internal consistency analysis showed that Cronbach's alpha values of the original six-factor model ranged from 0.68 to 0.88, while four-factor model ranged from 0.72 to 0.87.
CONCLUSION: This study did not support the proposed six-factor structure of the CLEQ tool. However, the four-factor CLEQ structure demonstrated an adequate degree of good fit and was found to be as reliable as the original structure. Further research on the predictive validity of CLEQ is required as well as a comparison of the psychometric properties across different institutions and countries.

Introduction

The clinical learning environment is the ideal educational climate offering numerous opportunities for students to learn and become competent healthcare providers.1 Learning in the clinical environment allows students to develop the knowledge they acquire in lectures into abilities and attitudes that pave the way for the achievement of clinical competence. The clinical training program begins during the clinical clerkship, in which medical students achieve competency-based learning objectives at different stages of the course. Undergraduate medical students have the opportunity to become proficient in their clinical training by practicing practical procedures, communication skills, professional skills, patient investigation and management, data interpretation, professional attitudes, and ethics.2,3

Training medical students in direct patient care is fundamental to the integration of their newly acquired skills with their existing knowledge from textbooks. However, since the 1960s, clinical-based education has exhibited many shortcomings, ranging from the inadequate supervision of learners, to insufficient time for discussion and feedback, a lack of sufficient clinical cases, and insufficiently structured clinical rotations.2–5 Overcoming such challenges is essential for the delivery of high-quality education and good patient outcomes.6,7 Indeed, the evaluation of the clinical learning environment is a crucial component for determining the quality of the clinical education and curriculum delivered. Performing such evaluations requires valid and reliable instruments.8,9

Many valuable tools are used to evaluate learning environments, including the most commonly used Dundee Ready Educational Environment Measure (DREEM), which is used by many educators across organizations and educational settings to appraise their institutions’ educational environments.10 The second most popular tool is the Postgraduate Hospital Educational Environment Measure (PHEEM), the purpose of which, is to investigate the different areas of the clinical learning environments of junior doctors.11 Third, the Clinical Learning Environment Inventory (CLEI) measures students’ perceptions of the factors related to the psycho-social aspects of learning in the clinical environment, the academic atmosphere, and facilities and their effects on the learning process.12 Finally, the Clinical Learning Evaluation Questionnaire (CLEQ) was developed for evaluating the clinical learning environment from the perspective of undergraduate medical students. This instrument was constructed as a response to the shortcomings of the existing instruments.13

CLEQ is a tool designed to measure the effectiveness of the clinical learning environment for undergraduate medical students. In contrast, the other tools focused on exploring the educational climate in general.13 The CLEQ instrument was developed based on the factors that contribute to effective clinical learning and was constructed with consideration of the findings of a qualitative study that took place at King Saud bin Abdulaziz University for Health Sciences (KSAU-HS) Riyadh, KSA. The study investigated effective clinical learning factors from the perspective of medical students and teachers.8 The initial CLEQ was intended to measure the following five factors that emerged from the study and that are often discussed in literature: (1) the diversity of clinical cases, (2) authenticity of the clinical practice, (3) the quality of direct observation and supervision, (4) the organization of the clinical sessions, and (5) the motivation to learn. Measuring the effectiveness of these five areas may positively influence students’ clinical education, which was the first aim of CLEQ.13 During the exploratory factor analysis, a sixth factor was revealed: (6) self-awareness, which refers to knowing one’s own strengths and weaknesses. It was found that this factor supported the preceding five factors and was a better fit for the data.8 Only 3 items had to be deleted due to their poor construction, after which a total of 37 items remained on the questionnaire.13

For self-developed questionnaires, verifying the items for construct validity is important, particularly when the questionnaire has not been used in past studies. This study sought to measure and examine the underlying construct along with the latent variables by conducting a confirmatory factor analysis using structural equation modeling (SEM).14 This validation method is required if the tool is to be used nationally and internationally for the purpose of assessing clinical learning environments.15

Confirmatory research or confirmatory factor analysis is used to test whether measures of a construct or a factor are consistent with the researcher’s understanding of the nature of that factor. It also verifies the number of underlying variables of the instrument (factors) and explores the item–factor relationship, in this case referred to as factor loadings.15

Researchers need measures with exceptional reliability and validity to use across different populations. Developing robust measures is a demanding and time-consuming process, and confirmatory factor analysis takes the existing measures one step further and also facilitates related research findings when the same measure is applied to a greater number of studies.16 Confirmatory research can be used for various reasons, including the development of new measuring instruments, evaluation of the psychometric properties of new measures, construct validation, and the examination of instrument effects.15,16

The present study had the following objectives:

To examine the psychometric quality of CLEQ in a sample of Saudi undergraduate medical students.

To retest the significance of specific item loadings.

To explore construct validity, as recommended by the original study.

Methods

A cross-sectional study was conducted using convenience sampling; Saudi medical undergraduates from three medical schools in their third or fourth year, ranging in age from 21 to 23 years, were invited to participate in this study. The sample size was estimated through the number of items in the questionnaire, and five samples per item were considered acceptable for the principal component analysis and confirmatory factor analysis to obtain model fit and sample size (N=185).17

Data were collected by distributing the questionnaires in the clinical areas. Informed consent was obtained from the deans of the three medical colleges before the questionnaires were distributed. Informed consent was obtained from the respondents who volunteered to respond to the questionnaire prior to its administration. Questionnaires were immediately returned upon completion. Data were analyzed using the Statistical Package for Social Sciences (SPSS v.22) and Analysis of Moment Structure (AMOS v.21). The CLEQ was developed as a tool to measure the clinical climate during the students’ clerkship and was confirmed as a valid and reliable tool after the exploratory factor analysis study.13

The CLEQ tool is based on 37 items measuring six areas of effective clinical learning based on the students’ perceptions: (1) Cases, (2) Authenticity of clinical experiences, (3) Supervision, (4) Organization of doctor–patient encounters, (5) Motivation to learn, and (6) Self-awareness. Each item is rated on a 5-point Likert scale.13 As a first step, principal component analysis by Varimax rotation was performed to test the item loadings and when it was best to remain within factors. This step is vital since an exploratory factor analysis must be conducted before the confirmatory factor analysis in order to verify the number of underlying latent variables (factors) and the pattern of observed variable–factor relationships.18,19 For construct validity, confirmatory factor analysis was completed using SEM through AMOS v.21 software to assess the data fit by looking at the fit indices, which were used as guidelines to reduce errors and improve the model-fit. First, chi-square goodness-of-fit (χ2/df) values below 500 were considered acceptable. Next, the absolute indices, which included the goodness-of-fit index (GFI) and adjusted goodness-of-fit index (AGFI), values should measure 0.9 or above to indicate model fit. In addition, the Residuals-Based Indices, namely Root mean square error of approximation (RMSEA) values, below 0.07, were considered adequate. Third, the values of incremental fit indices, namely the comparative fit index (CFI) and normed fit index (NFI), should measure 0.9 or above. The absolute fit indices determine how well the proposed theory fits the data. The calculation of incremental fit indices relies on comparisons with a baseline model.15,16

Then, the reliability of the six-model and four-model CLEQ was analyzed using Cronbach’s alpha. Finally, the correlation coefficient was computed, and the Pearson correlation coefficient was used to measure the dependence between variables.

This study was approved by the ethics committee at King Abdullah International Medical Research Center (KAIMRC), King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia.

Results

A total of 185 (100%) female (N=78) and male (N=107) third- and fourth-year students completed the CLEQ.

Exploratory Factor Analysis

First, the principal component analysis was completed with the Varimax rotation method, as in the first exploratory study.13 KMO and Bartlett’s test indicated that the data set was adequate and appropriate for use in EFA (Table 1).

Table 1

KMO and Bartlett’s Test

Kaiser–Meyer–Olkin Measure of Sampling Adequacy.		0.886
Bartlett’s Test of Sphericity	Approx. Chi-Square	3950.465
	df	666
	Sig.	0.000

The analysis was first performed on six components or factors. After interpretation, it was found that item loadings would fit better in a five-factor structure, since none of the items loaded at the sixth factor (self-awareness); item loading was assigned to the factor based on the highest loading score, with a cutoff score of 0.3 or higher. As for the factors, two factors were merged into one factor, factors 5 and 6 became one factor after items from both loaded highly together in one factor and were renamed motivation to learn (Table 2).20

Table 2

Principle Component Analysis of the Items on Five Factors

	Supervision	Authenticity	Motivation	Cases	Organization
Case 1				0.614
Case 2				0.749
Case 3				0.821
Case 4				0.728
Case 5				0.582
Case 6				0.539
Case 7		0.682
Case 8		0.692
Authenticity 1		0.815
Authenticity 2		0.828
Authenticity 3		0.609
Authenticity 4		0.749
Authenticity 5					0.553
Authenticity 6					0.628
Authenticity 7					0.367
Authenticity 8				0.386
Supervision 1	0.78
Supervision 2	0.791
Supervision 3	0.725
Supervision 4	0.804
Supervision 5	0.672
Supervision 6	0.611
Supervision 7					0.532
Supervision 8		0.321
Organization 1			0.548
Organization 2					0.667
Organization 3					0.645
Organization 4					0.607
Motivation 1			0.674
Motivation 2			0.79
Motivation 3			0.559
Motivation 4			0.621
Motivation 5			0.536
Self-awareness 1			0.506
Self-awareness 2	0.522
Self-awareness 3			0.614
Self-awareness 4			0.728
Extraction Method: Principal component analysis.
Rotation method: Varimax with Kaiser Normalization.

Descriptions of the Five Factors

Factor 1: Cases (7 items)

The item loading showed that item 16, which belonged to the factor authenticity of clinical experiences, loaded higher with cases. This showed that it was more related to case factor than to authenticity. Items 7 and 8 loaded higher in factor 2; therefore, they were moved there.

Factor 2: Authenticity of clinical experiences (7 items)

Items 7 and 8, from the cases factor loaded higher on this factor. These two items were originally constructed to remain in this factor. They were joined by item 24 from Supervision; this item loaded poorly and was retained in this factor based on its highest loading. Items 13, 14, and 15 from this factor loaded higher on the organization factor. Finally, item 16 from this factor was shifted to Cases.

Factor 3: Supervision (6 items)

Items from this factor loaded together except for items 23 and 24.

Factor 4: Organization of clinical encounters (7 items)

Items 13, 14, and 15 from the authenticity factor loaded higher in this factor. These three statements tied closely with organization factors and are interpreted to remain in this factor. In addition, the item 23 statement was more fitting to remain with the organization factor. Item 25 from this factor loaded with factor 5.

Factor 5: Motivation to learn (9 items)

This factor contains items 25 to 37 and item 25 from factor 4. This factor has the highest number of items, since it was merged with factor 6: self-awareness. The item 25 statement was found to fit better in this factor and complemented the other items of this factor.

Analysis was subsequently performed based on this arrangement of factors and items. Arrangements were not only considered based on the highest loadings but also by considering the theoretical implications for CLEQ.

Confirmatory Factor Analysis

SEM showed that the original six-factor CLEQ consisting of 37 items did not achieve model fit. Furthermore, during confirmatory factor analysis (CFA), many versions of the CLEQ were tested, including the re-arranged five-factor structure. None of the models showed model fit, since all the absolute and incremental indices did not suggest it.20 As a result, the first step was to covary items within factors based on the modification indices. The next step was the reduction of items based on the standardized residual covariance in order to select which items to remove as redundant (those items sharing the same meaning). The results are shown in Table 3. The four-factor model (ie, the shortened CLEQ) consisting of 18 items showed a good degree of model fit, as most of the indices indicated (X2=186.184, RMSEA=0.052, RMR=0.062, GFI=0.903, AGFI=0.865, CFI=0.951, NFI=0.871).21 The standardized factor loadings of items for the original six-factor structure ranged from 0.35 to 0.86, and item loadings for the four-factor structure ranged from 0.44 to 0.82, indicating that items from both structures contributed highly to the constructs being measured (Figure 1).20,21

Table 3

Results of Confirmatory Factor Analysis

Model Name	Chi-Square (χ2/df)	RMSEA (>0.07)	RMR (>0.08)	GFI (<0.95)	AGFI (<0.95)	CFI (<0.95)	NFI (<0.95)	No. Total Items Removed
Original six factors model (37 items)	1587.475	0.092	0.146	0.651	0.601	0.728	0.626	None
Model 1: seven factors (37 items)	1248.098	0.075	0.106	0.738	0.698	0.821	0.706	None
Model 2: six factors (37 items)	1255.219	0.074	0.101	0.727	0.687	0.821	0.679	None
Model 3-A: five factors (37 items)	1204.316	0.072	0.115	0.744	0.704	0.833	0.716	None
Model 3-B: five factors (28 items)	567.086	0.058	0.08	0.838	0.802	0.914	0.81	9
Model 4: four factors (18 items)	186.184	0.052	0.062	0.903	0.865	0.951	0.871	19

Figure 1

Standardized factor loadings for the best fit model of CLEQ.

The best fit was model 4, the four-factor structure after items (16, 24, 8, 25, 31, 11, 18, 28, 29, 9, 10, 12, 7, 28, 5, 30, 13, 2, 19, 34) were removed (Table 4).

Table 4

Modification Indices Until Model Fit Was Achieved

Name of Removed Items	Chi-Square	RMSEA (>0.07)	RMR (>0.08)	GFI (<0.9)	AGFI (<0.9)	CFI (<0.95)	NFI (<0.95)	No. Total Items Removed
Authenticity 8	1112.393	0.071	0.11	0.754	0.714	0.845	0.729	1
Supervision 8	1016.756	0.068	0.099	0.762	0.723	0.859	0.745	2
Case 8	919.378	0.065	0.099	0.779	0.74	0.876	0.761	3
Organization 1	861.398	0.063	0.098	0.794	0.754	0.886	0.775	4
Motivation 3	773.315	0.062	0.095	0.805	0.766	0.895	0.785	5
Authenticity 3	722.814	0.601	0.088	0.809	0.771	0.899	0.791	6
Supervision 2	652.393	0.058	0.084	0.825	0.788	0.91	0.801	7
Motivation 1	617.559	0.059	0.081	0.831	0.794	0.91	0.804	8
Authenticity 1	559.094	0.06	0.08	0.832	0.796	0.906	0.799	9
Authenticity 2	521.334	0.061	0.078	0.837	0.8	0.906	0.8	10
Factor-Authenticity (auth4-case7)	429.181	0.058	0.07	0.851	0.815	0.921	0.821	12
Organization 4	386.251	0.057	0.069	0.859	0.824	0.927	0.831	13
Case 5	347.453	0.056	0.68	0.866	0.83	0.933	0.842	14
Motivation 2	295.75	0.051	0.067	0.876	0.841	0.946	0.856	15
Authenticity 5	268.23	0.052	0.064	0.882	0.847	0.947	0.861	16
Case 2	245.126	0.054	0.063	0.888	0.852	0.945	0.862	17
Supervision 3	213.669	0.052	0.063	0.897	0.861	0.951	0.87	18
Self-awareness 1	186.184	0.052	0.062	0.903	0.865	0.951	0.871	19

Reliability analysis showed the four-factor model to be as reliable as the original six-factor model CLEQ (Cronbach alpha values of the original six-factor model ranged from 0.68 to 0.88, while model four ranged between 0.72 and 0.87).

Finally, correlations were computed between the four factors (Table 5). The table shows that all correlations are significant at the 0.01 level.21 The exploratory factor analysis in the original study showed a positive correlation between motivation and self-awareness, which became one factor in this study.13

Table 5

Correlation Matrix

Pearson Correlation
	Supervision	Motivation	Case	Organization	Overall
Supervision	1	0.543**	0.431**	0.612**	0.842**
Motivation	0.543**	1	0.301**	0.444**	0.740**
Case	0.431**	0.301**	1	0.401**	0.673**
Organization	0.612**	0.444**	0.401**	1	0.819**
Overall	0.842**	0.740**	0.673**	0.819**	1

Note: **Correlation is significant at the 0.01 level (2-tailed).

Discussion

This study was based on a previous study that focused on factor structure, validity, and reliability of the newly constructed CLEQ.13 The 37 items of the CLEQ were based on the previous study of the development of a clinical learning evaluation questionnaire for undergraduate clinical education. These items were placed according to six factors: cases, authenticity of the clinical learning experience, supervision, organization of the doctor–patient encounter, motivation to learn, and self-awareness. The first aim of this study was to investigate the reliability and validity of the factors on a different sample of students.13 The study was conducted with 185 undergraduate students from three medical schools in Riyadh, Saudi Arabia. During the analysis, the arrangement of items within factors differed from the previous study. Therefore, after establishing a five-factor structure of CLEQ, the items of this instrument were attributed to the factors on which they had the highest loading, and none of the items loaded in the sixth factor.22

The data from this study did not support the original six-factor structure consisting of 37 items or the proposed five-factor structure with 37 items measuring the clinical learning climate. The findings seem to be consistent with the previous study, which showed that construct validity was not well supported.13

The study then attempted to propose several versions of CLEQ that could meet the requirement of model fit. Even though it was not recommended by the previous study, it was found that the six-factor structure of the 37-item CLEQ (ie, the original CLEQ) failed to demonstrate model fit. This may suggest that CLEQ measures multiple constructs.20,21 The six-factor structure (see ) could not achieve model fit, suggesting the redundancy of a few items. The data indicated that the four-factor structure (shown in ) with 18 items, demonstrated model fit, since all fit indices and chi-square values were significant.19 Factor 2, Authenticity of Clinical Experience was removed because it poorly represented the construct being measured, and according to the findings of this study, the authentic clinical learning experience relied on the organization of patients encounters from the perspective of medical students.9 There is a significant correlation between organization and supervision factors, which indicates that those factors are closely related to the quality of the clinical learning experience from the perspective of medical students.23

The removal of 19 items during confirmatory factor analysis (CFA) may dramatically change the underlying factor structure of CLEQ, considering that they might hold useful and meaningful constructs in the clinical learning environment. However, these findings also suggest that there are repetitions of similar items assessing similar constructs that compromise the construct validity of the CLEQ.22

Most of the standardized correlation values of the four factors were more than 0.5, indicating overlap and indifference between constructs. This suggests that either the items need to be restructured to fit the proposed structure, or the model itself needs to be reconsidered and revised.18 Certainly, the four-factor model upon which it is built may need to be completely revised. These issues need to be addressed if CLEQ to be used. This might then be followed by a large-scale international sample being subjected to structural equation modeling analysis. Perhaps there are more constructs being measured by CLEQ, as was noted in the previous study.13,22

Considering all the adjustments, several assets have the potential to verify the authenticity of the data in this study. First, the sample was selected from students across different years of study that may represent students from different stages of the three medical schools. Second, the sample size was calculated based on the recommended ratio of subjects per item. Third, multiple alternative models were tested in this study to reach an ideal model fit. All modifications to the measurement models in this study were disclosed.24

Limitations

The findings of this study are based on the context of Saudi medical colleges. There is a small chance that language factors contributed to the potential shortcomings of this research, since the items in CLEQ were constructed in straightforward, comprehensible English sentences, there may have been a rough percentage of the students who are not proficient in English. This study was conducted in three medical colleges that were conveniently to sample, and may not represent medical students in Riyadh or the rest of the country.

Recommendations

Future research should compare the psychometric credentials of CLEQ in medical schools around the country and internationally. Further, other dimensions of validity should be explored, such as predictive validity, in order to obtain validity evidence on relationships between the survey scores and other variables.