Psychometric evaluation of a questionnaire to evaluate organizational capacity development for faculty development programs.

BACKGROUND: Organizational capacity development is an important outcome of faculty development programs, but there is a lack of an appropriate instrument for its evaluation. AIMS AND
OBJECTIVES: The aim of this study was to develop a questionnaire to evaluate the organizational capacity development for faculty development programs and to test its psychometrics.
MATERIALS AND METHODS: The initial capacity development for faculty development questionnaire (CDQ-FD) of 26 items was developed based on a literature review and opinion of experts. Content validity ratio (CVR), content validity index (CVI), content validity index for items (I-CVI), and the content validity index for scales (S-CVI) were computed for content validity. Confirmatory factor analysis (CFA) and exploratory factor analysis (EFA) were performed for construct validation.
RESULTS: The score for CVR, CVI, I-CVI, and S-CVI was 0.71, 0.83, 0.87, and 0.90, respectively. EFA resulted in a three-factor model with total variance extraction of 64%. Cronbach's alpha and Spearman Brown coefficient were investigated for reliability assessment. The Cronbach's alpha of overall scale was 0.8 and the test-retest reliability of the overall scale was 0.78. The final CDQ-FD contained 21 items and three categories.
CONCLUSIONS: The CDQ-FD questionnaire appears to be a valid and reliable instrument for the evaluation of organizational capacity development for faculty development in the medical education. Copyright:

Introduction

Faculty development programs are an essential component of the academic success of individual faculty members as well as their institution.[12] One significant step in maintaining the effectiveness of the faculty development programs is the evaluation of their outcomes. However, most of the research has been focused on only measuring the short-term outcomes, especially at an individual level. The examples include the evaluations of participant satisfaction,[345] exploration of participant attitude, knowledge or skills,[678] and assessing changes in participant behaviors.[910] Despite increasing demands for the evaluation of faculty development programs at a much broader level beyond individual aspects, little has been published on the impact of such programs on the organizations in the medical education. One important impact of these programs may be on the promoting and developing the capacities of the organization in which teachers work.[111213]

Capacity development can be considered to be the changes in the behavior of both individuals and organizations, such as the growth of new knowledge, skills, attitudes, values, and relationships, that lead to improved organizational performance.[14] These new capabilities engage individual faculty members with the various members of the wider organizational system, including other educators and administrators, to empower changes in the organization, both at individual and collective levels.[151617] One of the difficulties in evaluating capacity development is that each educational program may use a unique set of approaches and strategies,[18] and therefore, requires the specific evaluation tools.[19]

There are few studies which have explored capacity development for faculty development in medical education. Capacity development was identified by Frantz et al. as one of the five key themes in the participant perceptions of a faculty development program in sub-Saharan Africa.[20] Another study by Frantz et al. investigated the contribution of a faculty development program to individual and collective capacity development in sub-Saharan Africa by using participant interviews.[21] To the best of our knowledge, there has been no previous study of validated questionnaires for evaluating organizational capacity development for faculty development programs. Exploring organizational capacity development for faculty development programs is essential since it helps policy-makers of faculty development programs to understand the strengths and limitations of the capacity development process, informing their future planning to reinforce or modify the subsequent programs.

Given the importance of faculty development programs having an impact at organizational level and because of the lack of a specific instrument for evaluating capacity development for these programs in medical education, this study aimed to develop and test the psychometric properties of a questionnaire to evaluate the organizational capacity development for faculty development programs.

Subjects and Methods

Setting

The research was conducted at Tehran University of Medical Sciences (TUMS) in Iran between 2017 and 2019. The TUMS's institutional review board approved the study (No.IR.TUMS.IKHC.REC.1396.4122). The participants did not receive any incentives, and participation was voluntary.

The “Basic Teaching Skills Course” is one of the faculty development programs implemented at TUMS to help new faculty members fulfill their teaching roles. The course has been running since 2003 and covers the essential subjects for teaching effectiveness such as instructional design, teaching methods, and student assessment. It is delivered in an interactive format with lectures, group works, and practice-based assignments.

Item development

The items in the capacity development for faculty development questionnaire (CDQ-FD) were developed based on a previous literature review[22] and also the opinion of experts to ensure that they were relevant to the specific context of medical education.[23]

A comprehensive literature review to identify a list of the indicators of organizational capacity development for faculty development was performed. Studies were included for the review if they met the following criteria: (1) focused on capacity development for faculty development programs in higher education and medical education, (2) published in English language, and (3) published between the years 1980 and 2017. The literature was searched using Medline, ERIC (EBSCO), Scopus, Embase, Web of Science, and Google Scholar using the key words: staff/faculty/teacher development, faculty/teacher/staff continuous professional development, organizational capacity development/building, and enhancement.

An expert panel session with nine key informants from faculty development program providers at TUMS was conducted in 2017 using nominal group technique to elicit the indicators of organizational capacity development for faculty development programs. The expert group was not provided with the items from the literature review. The group members suggested the indicators inductively through a brain-storming process. After these two steps, the researchers merged the common indicators of the literature review and expert group. The indicators from each source were similar conceptually, but used different terminology. The researchers chose to prefer the vocabulary of the experts to develop the items of the CDQ-FD to ensure greater potential content validity. Some indicators were included from one source but not the other, the researchers kept these.

Psychometric evaluation

Content validation

The content validity of the initial CDQ-FD was investigated both quantitatively and qualitatively by expert opinion. Ten experts were recruited based on their experience in the management and administration of faculty development programs and their expertise in organizational capacity development. They were selected within several universities of Medical Sciences in Iran. Experts were asked to consider each item of the CDQ-FD based upon the criteria of “essential,” “relevance,” “clarity,” and “simplicity.” Each item was assessed using Likert scales: A three-point scale for “essential” (1 – unessential, 2 – useful, but not essential, and 3 – essential,), and four-point scales for “relevance” (1 – not relevant, 2 – rather relevant, 3 – relevant, and 4 – completely relevant) and “clarity” (1 – not simple, 2 – rather simple, 3 – simple and 4 –completely simple) criteria. In addition, the experts were asked to provide comments about the “simplicity” of each item (fluency and using simple and understandable words) as well as the most appropriate placement and order of the items.

We examined content validity by computing content validity ratio (CVR) and content validity index (CVI) using ratings of item relevancy that were highlighted by the content experts.[24] Furthermore, some studies showed that the chosen method may influence the results of the item deletion.[25] Hence, we used further indexes for investigating CVI. These indexes include the content validity index for items (I-CVI) and the content validity index for scales (S-CVI).[26]

Given the ten experts who evaluated the items, the minimum acceptable amount of CVR was 0.62 based on Lawshe table. The formula for calculating CVI in Waltz and Bausell method is the number of all the respondents in “relevancy,” “clarity,” and “simplicity” criteria divided by the number of experts who have scored 3 or 4 in the relevant question in that criterion. In this formula, if an item has a score more than 0.79 that item is retained in the questionnaire. If CVI is between 0.70 and 0.79, the item is questionable and needs correction and revision. Furthermore, if it is less than 0.70, the item is unacceptable and it must be deleted. In Lynn's method, the formula for CVI of items is the number of experts who have scored 3 or 4 for the related items in the “relevancy” criterion divided by the total number of respondents. In I-CVI formula, if the score of each item is more than 0.78, that item remains in the questionnaire. If the calculated score is less than 0.78, the item is questionable and needs correction and revision. In order to calculate the S-CVI, the CVI for scales/average (S-CVI/Ave) was utilized. For computing the S-CVI/Ave, the average of I-CVI scores in relevancy criterion was calculated. The obtained score for S-CVI/Ave must be 0.90 or more.

Construct validation

The modified CDQ-FD based on content validity was sent to 311 faculty members of TUMS who had been participated in the Basic Teaching Skills Course. It was redistributed two more times at approximately 4-week intervals, via E-mail and also followed up through the social media.

For investigating the construct validity, first a confirmatory factor analysis (CFA) was performed to examine and verify the assumed five factors structure of the CDQ-FD with LISREL software (8.8 version. New Jersey). Several fit indices were carried out to assess the fit of the hypothesized model to the data: comparative fit index (CFI), goodness of fit index (GFI), and adjusted goodness of fit index (AGFI), with values of about 0.9 considered adequate; standardized mean square residual (SRMR) and root mean square error of approximation, which should approximately be equal or less than 0.08 to be indicative of adequate fit of the model to the data.[27]

In the next step, exploratory factor analysis (EFA) followed by a varimax rotation was applied to determine the factorial structure of the questionnaire. We applied the Kaiser-Meyer-Olkin (KMO) and Bartlett's test measure to assess the sample adequacy and sphericity of the CDQ-FD, respectively. A KMO value equal or above 0.70 and a significant Bartlett's test of sphericity were considered as acceptable criteria for sample adequacy and factorability of correlation matrix. The criteria for keeping the factor for this study were extraction values above 0.32 and Eigen values above 1.0.

Reliability assessment

The internal consistency of the CDQ-FD was investigated by Cronbach's alpha. Internal consistency of more than 0.7 was considered suitable. For determining instrument stability, test-retest method was utilized. The CDQ-FD was administered to 15 faculty members of TUMS, under similar conditions with a 7-day interval between the first assessment and the second one. This group was not included in the subsequent phase and were not the same as the construct validity participants. The two sets of obtained scores were compared with Spearman Brown coefficient and the minimum acceptable correlation coefficient was considered 0.7. The overall CDQ-FD development and validation process is shown in Figure 1.

Figure 1

Overall development and validation process of capacity development for faculty development questionnaire

Results

Demographic data

All 10 experts completed the content validation form. The majority of them (70%) were women, 50% were assistant professors, 30% were associate professors, one participant was a professor and one was an instructor. The final number of participants who completed the CDQ-FD for investigating construct validity was 203 of the 311 recruited, yielding a response rate of 64.9%. The sample size appeared to be sufficient given the recommendation for factor analysis of 5–10 person per item in the questionnaire.[28] Female participants (49.5%) were almost equal in number with the male participants. Most of the participants were assistant professors (88.8%) and 71.2% were affiliated to clinical science departments. Over half had 1–5 years’ experience of being faculty member (71.7%), with the majority were from the school of medicine of them (69.8%).

Content validity

The initial CDQ-FD consisted of 26 items divided in five categories [Online Supplemental Appendix 1, English version of the CDQ-FD]. The corrective comments of experts about the wording of items, such as fluency, using simple and understandable words, and the suitable placement of the words were used. Five items were revised to increase the ease of understanding the wording. For example, based on the experts’ comments, the item “enthusiasm and self-confidence in teaching” was separated into “enthusiasm in teaching” and “self-confidence in teaching.”

The overall CVR was 0.71, which was acceptable. The CVI for all items was 0.83 by using Waltz and Bausell method (In terms of relevance 0.80, clarity 0.81, and simplicity 0.88). Three items with CVI <0.70 were removed as they identified as being vague or similar to other items. Nine items were corrected and accepted.

By calculating the I-CVI, four items were removed that had a 0.5 score. One item was corrected and accepted, and the rest of the items were retained, all with a 0.87 score value. The scores of gained by each item are presented in Table 1. The S-CVI/Ave (average score) was 0.9, which are appropriate.

Table 1

Item’s content validity index for item score

Item	I-CVI score	Item	I-CVI score
1	1	14	0.9
2	1	15	1
3	0.9	16	0.9
4	1	17	1
5	0.9	18	0.9
6	1	19	1
7	0.9	20	0.5
8	1	21	0.5
9	1	22	1
10	0.5	23	1
11	0.5	24	1
12	1	25	1
13	1	26	1

I-CVI=Content validity index for item

Construct validity

The results of CFA showed an inappropriate fitness for the five factors structure of the questionnaire (RSMEA: 0.13, GFI: 0.70, AGFI: 0.63, CFI: 0.89, NNFI: 0.87, and SRMR: 0.073).

EFA and sample size adequacy were examined using the SPSS software. The results showed that the item D5Q3 was a barrier for the positive definition of the correlation matrix, and after deleting this item, the results of the KMO and Bartlett's test indicated ample adequacy of the sample size and factorability of correlation matrix for conducting EFA (KMO index = 0.923, P < 0.001, Bartlett's test = 3645.222 and df = 210).

Viewing of the scree plot revealed three factors with eigen values greater than 1 and these factors explained 61.4% of the total variance. The item D1Q1 did not have any loading on any of the extracted factors. Therefore, this item was also deleted, and the remaining items were again examined by EFA. After deletion of the item D1Q1, the scree plot, the total variances, and the rotated factor matrix, two factors with eigenvalues > 1 explained a total of 51.65% of the variance. The next factor with eigenvalues equal to 0.98 was then analyzed and following the inclusion of this factor, the analyses and the total variance increased to 64%. The first factor included 13 items, and the second and third factor each included 4 items. In summary, the EFA identified a three factor structure. The first factor named “development and innovation in teaching and learning process and communications,” the second named “development and sustaining faculty development programs” and the third named “development of educational leadership and management.” The results of the EFA are presented in Table 2.

Table 2

Results of exploratory factor analysis of the capacity development for faculty development questionnaire

Number	Item	Factor 1	Factor 2	Factor 3	Name of factors
1	I have obtained the competencies to apply interactive teaching methods aligned with educational conditions	0.552			Development and Innovation in Teaching and Learning Process
2	My competencies to transfer concepts and skills to learners have been enhanced	0.737
3	My competencies to manage the classroom have improved	0.783
4	I have obtained the competencies to motivate students for learning	0.768
5	My enthusiasm in teaching have been enhanced	0.705
6	My self-confidence in teaching have been enhanced	0.832
7	I have obtained the competencies to apply novel methods for assessing learners	0.574
8	I have obtained the competencies to provide feedback to learners	0.533
9	I motivate to receive feedback on my own teaching performance	0.514
10	My teaching quality has improved	0.848
11	I have obtained the competencies to communicate with learners, colleagues and patients appropriately	0.771
12	I have obtained the competencies to do teamwork	0.746
13	I have obtained the competencies to use medical education evidences in my educational activities	0.664
14	I motivate for more request in new faculty development programs in medical education		0.858		Development and Sustaining Faculty
15	I encourage and provide guidance to other colleagues to participate in faculty development programs		0.776		Development Programs
16	I motivate more to become familiar with various fields of medical education		0.678
17	I efforts to be up-to-date in the field of medical education		0.48
18	I am motivated to analysis the university/school policies regarding educational activities			0.692	Development of Educational
19	I cooperate in the implementation of educational development processes at university/school			0.697	Leadership and Management
20	I help new colleagues for career progression			0.482
21	I motivate to identify educational problems, and, design and implement the appropriate interventions			0.565
Eigen value		11.57	1.84	0.98
Percent total variance		34.61	17.03	12.34

Reliability assessment

Cronbach alpha coefficient for all items of the CDQ-FD was 0.80. The Cronbach alpha coefficient for “development and innovation in teaching and learning process,” “development and sustaining faculty development programs,” and “development of educational leadership and management” were 0.80, 0.82, and 0.78, respectively, which was suitable. The Spearman Brown coefficient was 0.78 indicating that the instrument stability was acceptable.

Production of the final questionnaire

After investigating reliability and validity, the CDQ-FD with 21 items in three categories was finalized. These three categories included “development and innovation in teaching and learning process” with 13 items, “development and sustaining faculty development programs” with 4 items, and “development of educational leadership and management” with 4 items, and English version of final CDQ-FD].

Discussion

This study described the development and psychometric testing of the first instrument to evaluate organizational capacity development for faculty development at TUMS. The initial CDQ-FD included 26 items, and after content validation through two methods of Waltz and Bausell and Lynn, 23 items were retained. All CVIs were appropriate. The results of the EFA indicated that the three-factor model fits the data reasonably well. These categories included “development and innovation in teaching and learning process,” “development and sustaining faculty development programs,” and “development of educational leadership and management.” Two items were deleted through EFA and the final questionnaire consisted of 21 items. Even though there are no studies reporting the development and validity evidence of a questionnaire for capacity development of faculty development in medical education in any language, our results are closely aligned with the previous published work on the conceptualization of capacity development. The indicators highlight the importance of individual and collective development, with the evolution of professional identity as an educator and the empowerment of faculty members, to enable the organization to change and effectively cope with the complexity of factors in the wider organizational system.[29] About 65% of the participants answered all items for construct validation and this may indicate the future potential usefulness and functionality of the CDQ-FD.

The “development and innovation in teaching and learning process” category had a focus on developing competencies in the teaching and learning process, including various teaching and student assessment methods. The category of “development and sustaining faculty development programs” represented the interest of teachers in medical education and their support and collaboration with colleagues, which is essential to sustain and develop the programs. “Development of educational leadership and management” category referred to involvement in the development, implementation, and evaluation of the medical education institution. Further analyses showed acceptable internal consistency and reliability for CDQ-FD.

In the present study, the factor “development and innovation in teaching and learning process,” explained 34.6% of the total variance. These findings are consistent with the results of previous studies, with most faculty development initiatives having an emphasis on teaching and learning aspects[3031] and improving communication within the organizational systems.[32]

Lee et al. found that faculty development programs were also effective for improving faculty's teaching and learning competencies,[10] and some studies have claimed that these programs enhance humanistic capabilities such as professionalism, communications skills, group networking, and teamwork.[3334]

The factor “development and sustaining faculty development programs,” as the second factor with 17.0% of the total variance, has had little discussion in previous studies of faculty development programs. This new understanding of organizational capacity development is important for the future evaluation of the effectiveness of faculty development programs. The factor “development of educational leadership and management,” with 12.3% of the total variance is consistent with prior studies. Some researchers have reported that participation in faculty development programs produced more positive attitudes towards teaching, as well as greater involvement in organizational roles, such as leader and manager.[3536]

The item “I have obtained the competencies to design a course plan based on educational principles,” did not have any loading on any of the extracted factors and was deleted. A reason for this might be that course planning was embedded in other topics of the “Basic Teaching Skills Course” and not specifically taught on the course. Another deleted item was “I motivate to attend seminars and conferences related to medical education.” Its elimination might be that the seminar formats are unfamiliar to the faculty and that the faculty development course may not provide participants with sufficient information about this important educational approach.

Examining content validity through different indices provided a variety of evidence for CDQ-FD content validity.[28] Analyzing the content validity with two different methods showed no difference between indices and deleted items, which further assured us of the CDQ-FD content validity. The results of internal consistency with alpha's Cronbach coefficient of 0.80 for all items and 0.80, 0.82, and 0.78 for categories demonstrated acceptable levels. Our findings are consistent with prior studies. The Cronbach's alpha for the questionnaire which Jacobs et al. used to explore the effects of the efforts to improve evidence-based decision-making capacity ranged from 0.67 to 0.94.[37] The results of test-retest method and calculating Spearman Brown coefficient indicated that tool stability was acceptable. Therefore, considering that alpha's Cronbach coefficient was more than 0.7, the reliability of the CDQ-FD was considered suitable and verified the results of EFA.

There are some limitations to the study. First, all evaluations are based on faculty development participant's perceptions, which is a potential source of bias about the capacity development results. We therefore recommend using other insights such as policy makers, funding agencies, and student's perceptions.

Second, when using test-retests of questionnaires, there is always a risk that respondents may be influenced by answering the first questionnaire, and the answers to the second questionnaire will include differences due to an intervention effect. It is possible that some participants after the first questionnaire became more familiar with the impact of faculty development programs on capacity development and therefore changed their answers before the retest questionnaire; however, this was not evident in our results. Third, is generalizability of our findings. To be used in another context the CDQ-FD needs further validation in groups speaking other languages, different cultures and in other universities. Finally, no other capacity development for faculty development questionnaire is available. Therefore, it was not possible to validate the new questionnaire against a gold standard and testing criterion validity. Future research could examine how institutes experience the benefit of the questionnaire in faculty development interventions and development in the organization.

Conclusions

This is the first questionnaire for evaluating organizational capacity development for faculty development programs and it appears to be a valid and reliable instrument for the evaluation of organizational capacity development for faculty development in medical education. The questionnaire was developed and evaluated psychometrically by a variety of methods. All CVIs and test-retest reliability were appropriate. The results of the EFA indicated that the three-factor model fits the data reasonably well. Overall, the three categories of indicators in the final questionnaire are closely aligned with previous published work on the conceptualization of capacity development. The indicators highlight the importance of individual and collective development of faculty members to enable the organization to change and effectively cope with the complexity of factors in the wider organizational system.

Financial support and sponsorship

This work was a part of a PhD thesis at the Tehran University of Medical Sciences and was funded by the National Agency for Strategic Research in Medical Education, Tehran, Iran. Grant No. 970080.

Conflicts of interest

There are no conflicts of interest.

After participating in “Basic Teaching Skills Course”

Number	Item	Very much	Much	Average	Low	Very Low
Category 1: Development and innovation in teaching and learning process
1	I have obtained the competencies to design a course plan based on educational principles
2	I have obtained the competencies to apply interactive teaching methods aligned with educational conditions
3	My motivation to transfer concepts and skills to learners have been enhanced
4	My competencies to manage the classroom have improved
5	I have obtained the competencies to motivate students for lifelong learning
6	My enthusiasm and self-confidence in teaching have been enhanced
7	I have obtained the competencies to apply new methods for assessing learners
8	I have obtained the competencies to provide feedback
9	I am motivated to receive feedback on my own teaching performance
10	I meet the principles of professional behavior in education and clinical practice
11	I ensure fairness in teaching and assessment of learners
12	My teaching quality has improved
Category 2: Development and innovation in communications and collaborations at different levels
13	I have obtained the competencies to communicate with learners, colleagues and patients appropriately
14	I help new colleagues for career progression
15	I have obtained the competencies to do teamwork
Category 3: Development and sustaining faculty development programs s
16	I am motivated for more request in new faculty development programs in medical education
17	I encourage and provide guidance to other colleagues to participate in faculty development programs
18	I am motivated more to become familiar with various fields of medical education
19	I make greater efforts to be up-to-date in the field of medical education
20	I have obtained the ability to refer to specialized evidence or consult with experts when answering a question or to inform decision making in the field of medical education
Category 4: Development of educational leadership and management
21	I am motivated to evaluate the quality of education in my own department
22	I am motivated to evaluate the university/school policies regarding educational activities
23	I cooperate in the implementation of development processes at university/school
Category 5: Development in scholarship
24	I have obtained the competencies to use medical education evidences in my educational activities
25	I am motivated to identify educational problems, and, design and implement the appropriate interventions
26	I am motivated to attend seminars and conferences related to medical education