Psychometric Properties of the Persian Questionnaire for Evaluation of Clinical Teaching at Outpatient Settings.

PURPOSE: To assess the validity and reliability of the Persian Outpatient Clinical Teaching Evaluation (OCTE) questionnaire, in order to evaluate clinical teaching for undergraduate medical students.
METHODS: This was a methodological study conducted in Tabriz University of Medical Sciences in 2018. To assess the validity of the questionnaire, ten academic staff who had received master's degrees in medical education were chosen by convenience sampling. The first draft of the Persian questionnaire was derived from the Ministry of Health's clinical teaching standards booklet. The questionnaire consisted of just 15 obligatory items in the first draft of the questionnaire. Subsequently, the questionnaire was modified to be used in teaching outpatient clinics. Content validity indices were calculated. Subsequently, the modified questionnaires were given to 92 academic staff in Imam-Reza Hospital's outpatient clinics. We applied principle component analysis (PCA) and Varimax rotation for exploratory factor analysis (EFA). In order to confirm the EFA, confirmatory factor analysis (CFA) was used. Cronbach's alpha method, the intra-class correlation coefficient (ICC) and test-retest reliability were used to assess the reliability.
RESULTS: The overall content validity ratio (CVR), content validity index (CVI), and impact score (IS) were 0.78, 0.79, and 3.26, respectively. Out of 92 academic staff, 85 participated. The Keizer-Meyer-Olkin (KMO) measure of sampling adequacy for data was 0.726 (p-value=0.0001). After the EFA, the 10-item questionnaire loaded on three factors. The internal consistency of the questionnaire was established by Cronbach's alpha of 0.89 and the ICC was 0.94. Furthermore, there was a strong correlation with a 2-week retest interval.
CONCLUSION: The study demonstrated that the Persian clinical teaching questionnaire had optimal psychometric properties. Thus, application of this questionnaire at outpatient settings can be helpful.

Introduction

Clinical teaching at outpatient settings is a critical part of medical students’ training and its evaluation is important.1 One of the common types of clinical teaching evaluations is using a questionnaire to grade teachers by students at the end of clinical rotations.2 Recently, medical educators have increasingly applied these questionnaires. Almost all of the questionnaires introduced in Western settings and their generalizability is limited for other countries and even universities.3 In order to solve these challenges, a study described the development and content validation of an instrument for evaluating clinical teaching addressed to an East Asian setting.4 An other study about the evaluation of clinical teaching pointed out that assessment tools should be compatible with the setting of organization, favorable to teachers, easy to conduct, and appropriate to all levels of educators.5 Accordingly, to improve the quality of clinical teaching, the first step is to use valid and reliable instruments in the clinical teaching environment. Then by implementing interventions during a dynamic process, more successful clinical teaching could be expected.6

In recent years, several questionnaires have been developed and validated for evaluation of clinical teaching, such as the Dundee Ready Education Environment Measure (DREEM) and the Service Quality Measurement Questionnaire (SERVQUAL). Evidence demonstrated that all these questionnaires were more appropriate for inpatient settings.7,8 Although the Persian version of the Ambulatory Care Learning Education Environment Measure (ACLEEM) questionnaire was developed by Parvizi et al9 in 2016 for evaluation of clinical teaching in outpatient settings, it was designed for residency medical clinics. To date, a valid and reliable Persian questionnaire to evaluate the clinical teaching for undergraduate medical students at the outpatient settings is lacking.

The Ministry of health in Iran developed the standards for clinical teaching because there was a need to improve the clinical teaching for undergraduate medical students.10 However, for the evaluation of clinical teaching, there were the basic necessities for the instrument to be valid, and reliable.11

A limited number of studies have validated the Persian instrument to evaluate clinical teaching for undergraduate medical students in outpatient settings. For example, a 10-item questionnaire was found to be valid for the evaluation of clinical teaching capacities in outpatient settings. Although this tool was reviewed for content validity by the authors, no other psychometric properties were investigated.12 Other studies validated the Persian questionnaire to evaluate the journal clubs and the morning reports in the teaching hospitals.13,14

Evaluation is the important factor in the performing and improving of educational programs.9 Teachers, students, and health care systems, along with patients, can be supposed to gain from improvement of clinical teaching. So, the aim of this study was to assess the validity and the reliability of the questionnaire derived from the clinical teaching standards booklet of Iran’s Ministry of Health to evaluate clinical teaching in outpatient settings.

Methods

Study Design

This was a methodological study conducted in Tabriz University of Medical Sciences in Iran, from October to December, 2018. We investigated the validity and the reliability of the questionnaire to evaluate the clinical teaching at the outpatient settings.

The Questionnaire

The first draft of the Persian questionnaire was derived from the Ministry of Health’s clinical teaching standards booklet.10 The Educational Deputy of the Ministry of Health has conducted a project called “Determination of Clinical Medical Education Standards” in 2015, in order to develop the clinical teaching standards booklet. The first step in the project was situation analysis of clinical education in Iranian medical universities. In the next step, medical education experts reviewed the literature and summarized the reports and the first draft of the clinical teaching standards booklet was developed. These standards were related to the learning environment in the medical education process including empowering the teacher on clinical teaching methods, supervising the students, location and equipment for clinical teaching, frequency and duration of clinical teaching, the combination and number of patients, object-based teaching, teaching content, effective educational practices, night shifts for medical interns, ethics, evidence-based practice, medical recording and documentation, and evaluation.15 The booklet consisted of outpatient clinics, teaching round, grand round, morning reports, journal reports, and essentials of clinical skills’ standards.10

The clinical teaching standards at the outpatient settings consisted of 27 items, 15 obligatory and 12 preferred items. This study was conducted based on only the obligatory items. As the original and validated questionnaire both were in Persian, no translation was performed in the present study. Subsequently, the questionnaire was modified to be used in the teaching outpatient clinics.

Validity and Reliability

Both qualitative and quantitative methods were performed to investigate the validity of the questionnaire. To do so, ten academic staff with master’s degrees in medical education were chosen by convenience sampling, hereafter referred to as our expert panel. In qualitative method, in terms of face validity, the experts were requested to write their opinions about the items (the location of the items, the structure, the ambiguity, and the difficulty of the items) and the scaling. In the quantitative method, the CVR and the CVI of the items and the whole questionnaire were calculated. Additionally, the impact score (IS) was applied to assess the face validity of the questionnaire. To find out the threshold point for CVR, Lawshe’s table was applied indicating a CVR of 0.62 as the lowest required value for each item for ten experts.16

According to the content validity, we asked experts to read the questionnaire and evaluateeach item based on a three-point grading including essential, beneficial but unessential, and unessential. In order to determine the CVI, the simplicity (absolutely simple, simple, fairly complex, complex), the relevance (irrelevant, fairly relevant, relevant, absolutely relevant), and the clarity (absolutely clear, clear, fairly clear, unclear) of the items were independently examined based on a four-point scale. The CVI for each item was calculated by dividing the number of experts who graded the items as scores three and four for each factor (simplicity, relevance, and clarity) divided by the total number of experts. Accordingly, a CVI more than 0.78 pointed out a sufficient content validity. In case of a CVI less than 0.78, the item was modified or omitted.17

The IS (frequency (%) × importance) was used to reveal the percentage of the experts, who identified the item as important or quite important. The IS scoring was based on a 5-point grading. The items were considered appropriate if they had an impact score of 1.5 or higher in terms of face validity.

In order to test the convergence and the discriminant validity, the questionnaire was delivered to the academic staff in Imam-Reza Hospital’s outpatient clinics, an educational and referral center and the largest outpatient clinics in Northwest of Iran. The advised sample size for factor analysis was supposed to be ten subjects per item.18 Therefore, we requested all academic staff who worked at outpatient clinics in Imam-Reza Hospital, to fill out the questionnaire consisting of 52 internal medicine specialists, nine infectious disease specialists, six neurologists, eight general surgeons, ten urologists, and seven otorhinolaryngologists. The checklist was based on a 5-point scale including always, often, sometimes, rarely, and never that scored 5 to 1, respectively. The demographic data of raters including age, sex, work experience and academic degree were also collected.

The discriminant validity was measured using the EFA. The KMO measure of sampling adequacy and Bartlett’s test of sphericity were applied to determine whether factor analysis could be appropriate for our data considering proportion of variances in the variables. The KMO value equivalent to or more than 0.70 was regarded as adequate for factor analysis. A significant Bartlett’s test of sphericity was regarded as acceptable. Various mathematical ways were applied to specify which items fitted together but the most ordinary one was the principle component analysis (PCA). This analysis allowed us to recognize the components, and defined the related variance accounted for each component and their agreement. Each component had a quality score (Eigen value). Exclusive components with high Eigenvalues were expected to suggest a real factor. Furthermore, if an item had more than one factor loading (cross loadings), the Varimax rotation was applied to rearrange the factor loadings such that each item measured absolutely one component. Additionally, the CFA was tested using Lisrel 8.80 software.

In terms of convergence validity, Cronbach’s alpha coefficient was calculated to examine the internal consistency of the questionnaire. Furthermore, in order to assess the repeatability of the questionnaire, the participants responded to the questionnaire with a 2-week interval and the ICC was calculated. The ICC values of 0.40 or more were considered satisfactory.19 For testing the reliability, the concordance rate of participant’s answers between the two times of filling out the questionnaire were compared by Kappa coefficient.20 The data were analyzed using SPSS software, Version 16.

Ethical Considerations

Our study was in accordance with the Declaration of Helsinki. We respected the autonomy of participants and protected their anonymity. Furthermore, all participants provided written informed consent to participate in this study. This research project was approved by the Ethics Committee of Tabriz University of Medical Sciences (IR.TBZMED.REC.810).

Results

The expert panel consisted of ten academic members of the Tabriz University of Medical Science with a mean age (SD) of 48.12 (4.13) years and all were men. To assess the construct validity and reliability, the questionnaire was sent to 92 academic staff, of which 85 had a mean (SD) age of 44.36 (5.41) years and agreed to participate (response rate was 92%), and 39 (45%) were women and 46 (54%) were men with an average of 16 years work experience.

In terms of face validity, the experts modified just one item to make it relevant, easy, and clear. The lowest and highest CVR were 0.51 and 0.95, respectively. Total CVR for whole questionnaire (mean CVRs of all items) was 0.78. The lowest and the highest CVI (mean of CVIs for the relevance, the clarity and the simplicity criteria) were 0.51 and 0.95, respectively. Total CVI (mean CVIs of all items) was 0.79. Additionally, total IS score of the questionnaire was 3.26. The final decision to modify, adopt or omit each item is shown in Table 1. The 10-item questionnaire was considered acceptable regarding the CVI, CVR, and IS.

Table 1

The Content and Face Validity of Each of the Final Items of the Questionnaire

Domain	Number	Item	CVR	CVI	IS	Decision
Preparation	1	The students visit the patients under the supervision of clinical teachers at the outpatient clinics	0.87	0.82	4.34	Adopted
	2	The maximum number of students who are under the supervision of a clinical teacher is five at the outpatient clinics	0.89	0.87	4.12	Adopted
	3	The teaching hospitals have a general outpatient clinic or training the students	0.59	0.71	1.32	Omitted
	4	Each office is equipped with the required tools for a physical examination	0.91	0.89	4.54	Adopted
	5	The outpatient clinics are equipped with enough chairs and tables for students	0.61	0.67	1.45	Omitted
Scheduling	6	The students have been trained at least 2 days per week in the outpatient clinics	0.88	0.91	4.67	Adopted
	7	The students are present in the clinic from the beginning to the end	0.94	0.95	4.78	Adopted
Implementation	8	Learning the goals and necessary experiences have already been determined and the students have been informed	0.86	0.94	4.29	Adopted
	9	The training methods are used such that the students will be able to visit prevalent patients independently at the end of their clinical course	0.51	0.63	1.14	Omitted
	10	For each outpatient visit, the effective educationalcommunication between the trainer and students takes place for a minimum of 3 minutes	0.95	0.85	3.78	Adopted
	11	The student actions have been documented in the students’ log books	0.81	0.89	3.89	Adopted
	12	Assessment of the students constitute their practice in the outpatient clinic	0.53	0.61	1.21	Omitted
	13	The students visit at least one new case at each session	0.58	0.51	1.35	Omitted
	14	The students have at least two night shifts and maximum of eight during each month of the course	0.93	0.88	4.97	Adopted
	15	Related references are in the clinic for the students' use	0.89	0.82	3.12	Modified

Abbreviations: CVR, content validity ratio; CVI, content validity index; IS, impact score.

The KMO measure of sampling adequacy for data was 0.726 (p-value= 0.0001). The results demonstrated that the data is appropriate for factor analysis and samples are adequate. The PCA extracted ten components and only three had an initial Eigen value higher than one. Rotated component matrix analysis for the questionnaire is shown in Table 2. Accordingly, these items were classified into three domains. The major components derived from three factors, had an initial Eigen value higher than one. Preparation (factor 1) with three items accounted for 34% of total variance. Scheduling (factor 2) with two items explained 23% of variance. Finally, implementation (factor 3) explained 14% of variance. As a result, exploratory factor analysis summed up the 10 questions of this questionnaire in three areas which explained 73% of the entire variance. We applied a confirmatory factor analysis model to establish the model of exploratory factor analysis (Figure 1).

Table 2

Exploratory Factors and Explained Variance After Rotation of the Questionnaire

Domain	Number of Items	Factor
		1	2	3
Preparation	2	0.949
	1	0.926
	4	0.831
Scheduling	7		0.905
	6		0.691
Implementation	10			0.784
	8			0.766
	11			0.684
	14			0.635
	15			0.611
Eigen value	–	3.45	1.51	1.13
Variance	–	32.05	15.23	11.25

Figure 1

Confirmatory factor model for outpatient clinical teaching.

To test the reliability, we examined the test–retest correlation coefficients between the first and the second time completing the questionnaire by Kappa coefficient. There was a strong correlation between 2-week tests (Table 3). The response rate for the ICC analysis was 100% (N= 85). The ICC and Cronbach’s alpha were 0.94 and 0.89, respectively. These showed high internal consistency for the questionnaire. Cronbach’s alpha was 0.88 (n = 3 items), 0.89 (n = 2 items), and 0.97 (n = 5 items) for the preparation, scheduling, and implementation domain, respectively.

Table 3

The Test–Retest Correlation Coefficients Between the First and the Second Time Completing the Questionnaire

Items	Kappa Value	p
Q1-1*Q1-2	0.84	<0.001
Q2-1*Q2-2	0.88	<0.001
Q3-1*Q3-2	0.98	<0.001
Q4-1*Q4-2	0.93	<0.001
Q5-1*Q5-2	1	<0.001
Q6-1*Q6-2	0.86	<0.001
Q7-1*Q7-2	1	<0.001
Q8-1*Q8-2	0.89	<0.001
Q9-1*Q9-2	0.96	<0.001
Q10-1*Q10-2	1	<0.001

Discussion

Based on a literature review there were a few relevant questionnaires available up to the time of performing the current study.4 Moreover, clinical teaching quality at the outpatient settings in Iranian medical universities has not been addressed sufficiently.12 Hence, in order to evaluate the clinical teaching quality at the outpatient settings, it was necessary to develop a valid and reliable questionnaire.

In this study, the Persian version of the questionnaire evaluating clinical teaching in outpatient settings was developed. According to the findings, the OCTE questionnaire had acceptable psychometric properties. Thus, medical faculties and departments in all Persian speaking countries can apply this tool in evaluating clinical teaching quality at outpatient clinics.

Our questionnaire was derived from Iran’s Ministry of Health clinical teaching standards booklet. This booklet was based on the tips on developing an ambulatory teaching setting in a new venue. It included preparation, delivery and evaluation domains.21 For this reason, we observed similar structure compared to our questionnaire.

In order to address possible biases in scoring the relevancy, clarity, simplicity, and necessity, we requested the experts of the medical education to provide their views on each item for the evaluation of the teaching in outpatient clinics. The experts had consensus on ten selected items, but their views on best format for the response scale differed from Yes/No answers to a 5-point Likert scale. They suggested an odd number of scoring on a response scale, and they recommended adding “sometimes” in response options to allow participants to be neutral and easily express their opinions. It was similar to some studies demonstrating that investigators may have preferred a 5-point Likert scale.18

We calculated the CVI and the CVR because the content validity is important in determining the concept of measurements. Although these are robust indices, they could not adequately support the construct validity of an instrument.19 So, we applied a construct validity test to solve this problem. The questionnaire was found to have favorable internal validity and reliability. After EFA, the 10-item questionnaire loaded on three factors. The percentage of total variance of 72.8 was acceptable. The derived three factors in the questionnaire appeared to be logical and rational. Five items loaded on the implementation of the clinical teaching at the outpatient setting domain, three items remained in the preparation domain, and finally, scheduling domain covered two other items.

The questionnaire had great internal consistency, as measured by Cronbach’s alpha. It was 0.89, which was consistent with the results of the study by Tayyebi et al12 in 2015. Favorable stability of the questionnaire might be a result of stability of academic staff’s views about the clinical teaching situation during the 2- week interval. Additionally, all the participants took part in the second test, leading to stable estimation. Accordingly, the questionnaire had strong test re-test reliability, which was assessed by the Kappa coefficient.

A study has assessed the validity and reliability of the Persian version of WATCH questionnaire according to the viewpoints of medical students.22 However, we used the experts’ opinions to validate the questionnaire, and it evaluated different domains of clinical teaching. Another study has evaluated the validity and the reliability of the Maastricht Clinical Teaching Questionnaire (MCTQ) in Bahrain. The MCTQ was a suitable tool to evaluate teachers’ skill during undergraduate clinical rotations. It was according to the cognitive apprenticeship model.23 Furthermore, Nation et al6 investigated the reliability and validity of a new clinical teaching assessment instrument (CTAI). Although MCTQ and the CTAI were valid and reliable instruments to evaluate clinical teachers’ skills, they were not designed to evaluate other aspects of educational environments in the outpatient settings such as policies and governance structures. The Persian version of ACLEEM questionnaire was a valid and reliable instrument for Iranian residents to evaluate residency outpatient settings. There was a large difference in structure compared to our questionnaire. Maybe the difference between residency and undergraduate ambulatory settings led to this mismatch.

Although this study had some strengths such as performing both qualitative and quantitative methods to evaluate validity and reliability of the questionnaire, there were also limitations. First, the medical education experts and academic staff were mainly considering the local priorities of teaching in outpatient settings, which may have impacted on content validity. Second, medical education experts and academic staff were selected by convenience sampling, so they may not be representative sample of all academic staff’s community. Third, age, gender and prior experience were some factors that could affect academic staff’s views toward clinical teaching in outpatient settings. We propose that future studies test the validity and reliability of the questionnaire in other universities and even other countries to assess generalization of this questionnaire.

Conclusion

The study findings demonstrated that the Persian OCTE questionnaire had optimal psychometric properties. Hence, application of this questionnaire in outpatient settings can be helpful in the evaluation of clinical teaching facilities and strengths by faculties and departments.