Reliability and validity test of the Korean version of Noe's evaluation.

PURPOSE: This study aims to demonstrate whether the functionalities between the widely used questionnaire in other countries and the Korean-translated version show similarity. Also, it intends to verify their reliability and validity.
METHODS: The original questionnaire was first developed by professor Noe at the University of Minnesota with 29 items named as "mentoring functions" to identify the participants' psychological and career-development functions. Using the Korean-translated version of the original questionnaire, the study was conducted on total 288 Kyung Hee University medical students, ranging from the first-year students to the third-year students on December 2015. In order to investigate if the survey form could be applied to the Korean participants, exploratory factor analysis and confirmatory factor analysis were conducted, using varimax rotation. Cronbach α statistics and the figures of standardized regression weights were analyzed respectively to indicate construct, convergent, and discriminant validities.
RESULTS: The result of exploratory factor analysis shows there are a total of three functions, including the additional "friendship function" (Eigen value, 1.152; significant level if higher than 1.0) with its high emphasis. The result of confirmatory factor analysis also demonstrates the similarity (p=0.000). As the whole reliability scale of the three functions is significantly high (Cronbach α , 0.971), each reliability scales of the three functions are shown to be high (0.814-0.955) as well.
CONCLUSION: The model, with its three functions, proved the significant statistics regarding the reliability and validity. According to this ground, the conclusion is that the adapted questionnaire used in this study could be applied to Korean medical students.

Introduction

Medical students are under significant pressure because they not only have to master their major studies and practical techniques, but also set their career paths and look for jobs at the same time. According to the study [1], although medical students have various problems with their academics and career paths, it is shown that 67% of them do not find any colleagues or professors who can provide sincere advice and feedback. Thus, their burdens in the academics and futures are cumulative.

According to a study [2] conducted in the United States, most medical schools have mentoring programs (79%) or advising programs (100%). To follow this trend, Korean medical schools are adopting similar mentoring programs.

In mentoring programs, mentors are defined as doctors and medical professors who are relatively experts in their fields, and mentees are medical students who are less experienced. Mentoring is a series of activities [3] where mentors pass down their tips and know-how to their mentees and encourage them. Therefore, mentoring includes interactions between mentors and mentees, and it eventually brings positive effects for both of them. Concretely, it is expected that mentors [4] can help their mentees develop teamwork, communication, and personal-relationship skills through mentoring programs. Also, mentees can not only decide their career paths more wisely, but also reduce their pressure by sharing their personal issues with their mentors. Particularly, medical students can develop a deep understanding of a doctor as a professional job and strengthen their network through mentoring systems, because in large part, the mentees follow the same career directions as their doctor mentors did. With mentoring programs, both mentors and mentees can maintain their networks and develop their career even after graduation. Due to these benefits, accepting mentoring programs in many Korean medical schools is a rising trend.

However, there is a lack of development for valid measurements in order to prove its effectiveness and its domestic application. Therefore, developing appropriate measuring systems is significant and necessary to accumulate empirical research for the mentoring field. For these reasons, this study intends to develop an adapted Korean-version questionnaire, which [5] was originally created by professor Noe at the University of Minnesota in 1988. Finally, the purpose of this study is to establish a foothold to afford better mentoring programs to Korean medical students.

Subjects and methods

1. Subjects

The participants of this study were current Kyung Hee University medical students in their first, second, and third years. Of 288 total students, except those who responded falsely (four students) and did not respond, there were 174 male students and 114 females.

2. Measurement tool

The questionnaire that professor Noe created in 1988 at the University of Minnesota is composed of 29 items. However, this study was conducted with the Korean version (Appendix 1). All the items were measured using a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). The original questionnaire was used after receiving prior consent from Professor Noe by email. Each item corresponds to an individual function, but the overall functions are largely divided into psychosocial (15 items) and career development functions (14 items). This classification of the functions is cited from the classification of mentoring functions [6], by Kathy Kram at Boston University. According to the classification of the original questionnaire, the psychosocial functions include acceptance and confirmation, role model, counseling and friendship, whereas the career development functions include coaching, protection, exposure and visibility, sponsorship, and challenging assignments. The survey questionnaire was provided as supplementary material to this article.

3. Methods

The sample estimation process should be conducted robustly in order to enhance the quality of studies. Larger samples are superior to smaller samples in many ways. Researchers can minimize the probability of committing errors, increase the generalizability of their findings, and maximize the accuracy of their conclusion through employing larger samples. According to researchers, 10 cases for each item should be sufficient to perform an exploratory factor analysis [7]. This study has a sufficient number of subjects with a total of 288 valid subjects.

This study used the Kaiser-Meyer-Olkin (KMO) test and Bartlett’s test of sphericity to measure the adequacy of samples in terms of the distribution of values for the execution of factor analysis. Exploratory factor analysis was used to examine whether the application of the adapted questionnaire to the Korean medical students can correspond to the same categorization of these functions. In the exploratory factor analysis, the principal factor method and the varimax method were used. This study conducted the confirmatory factor analysis to examine the model validity of the factors that are tied to the same group from the exploratory factor analysis. Also, since it is important to retain the reliability and validity in the confirmatory factor analysis, Cronbach α statistic was calculated and construct, convergent, and discriminant validity were conducted. The confirmatory factor analysis was performed with the use of chi-square, root mean square error of approximation (RMSEA) and comparative fit index to examine the fitness and correspondence of the model as a whole. Standard regression weights were used for the construct validity. The convergent validity was examined by using the construct reliability and variance extracted index. The discriminant validity was examined by using coefficient of determination and variance extracted index. This study used SPSS version 21.0 (IBM Corp., Armonk, USA) and Amos 20.0 (IBM Corp.), respectively, and the significance level was set for 0.05.

Results

1. General characteristics of the participants

The participants of this study were composed of 60.4% male students and 39.6% of female students. The number of people in each year is as follows (Table 1).

Table 1.

Demographic Characteristics of the Samples (n=288)

Characteristic	No. (%)
Gender
Male	174 (60.4)
Female	114 (39.6)
Grade
1	90 (31.2)
2	101 (35.1)
3	97 (33.7)

2. Result of exploratory factor analysis

In this study, the KMO test and Bartlett’s test of sphericity were used to measure the adequacy of the collected samples prior to the execution of exploratory for its suitability in the factor analysis. The outcome for the KMO test was 0.963, so the conformity was confirmed to perform the factor analysis. Also, as the stage to figure out common factors and to indicate the suitability of the factor analysis model, Bartlett’s test of sphericity generated 7,490.33 (0.000), showing the statistical significance. Since the variables are correlated with each other, this study can use the principal factor method. According to these grounds, the principal factor method, which uses varimax rotation, was performed as the factor extraction method. The table of the rotational component matrix is shown in the Table 2. The factor analysis was conducted with the principal component analysis using of varimax rotation to group common factors within 29 items. The result of the exploratory factor analysis within the 29 items of the original questionnaire was different from the result of this Korean version research form. The original questionnaire was classified into two parts: psychosocial and career development functions.

Table 2.

The Result of Factor Structure

Item	Factor 1 (psychosocial function)	Factor 2 (career development function)	Factor 3 (friendship function)	Cronbach α
n 8	0.779			0.955
n 5	0.769
n 6	0.738
n 1	0.717
n 12	0.715
n 7	0.712
n 2	0.712
n 10	0.674
n 9	0.666
n 3	0.661
n 14	0.650
n 4	0.644
n 13	0.620
n 15	0.582
n 18		0.824		0.954
n 19		0.812
n 21		0.792
n 17		0.753
n 16		0.723
n 24		0.704
n 20		0.698
n 23		0.670
n 25		0.643
n 11		0.629
n 26		0.610
n 22		0.540
n 29			0.744	0.814
n 27			0.680
n 28			0.510
Eigen value	16.357	2.159	1.152
Kaiser-Meyer-Olkin				0.963
Bartlett’ test of sphericity
Chi-square				7,490.33
df (p-value)				406 (0.000)

df: Degree of freedom.

However, this new study showed different outcomes; “friendship functions” was added. As a result, the researchers of this study decided to group the 12 items except two items in “psychosocial functions,” because the items were originally in the same category. With regard to the factor 2, the 12 items except one item were overlapped with the original questionnaire, so we decided to call them “career development functions” as the original ones are named. The last three items were classified as “friendship functions” in the original questionnaire as sub-functions. Therefore, we emphasize “friendship functions” as new functions of this study. The total explanation power of the three factors was 67.819% and each Eigen values were higher than 1.0 (Table 2).

3. Result of confirmatory factor analysis

Previously, through the exploratory factor analysis, we determined that the two categories the original research form proposed were inappropriate in South Korea. Accordingly, the confirmatory factor analysis was performed to examine the suitability of the three factor structures of the mentoring program functions that were discovered in the exploratory factor analysis. The fit index obtained in the confirmatory factor analysis is shown in the Table 3. The result of the structure equation analysis showed chi-square statistics, the degree of freedom (df), and p-value. Generally, goodness of fit is interpreted as statistically significant if p-value is higher than 0.05 and p-value gets higher when the statistics of chi-square gets smaller. The probability of the statistics of chi-square also increases when the size of samples gets higher. In other words, since the p-value is influenced by the chi-square and the size of samples, it cannot be said that the goodness of fit is lower in a situation where the size of samples is high and the data with many variables [8]. Therefore, it is important to make a decision regarding the goodness of fit with other various indexes.

Table 3.

Structure Model Fit Indices

	df	χ2	p-value	CFI	RMSEA	RMSEA (90% CI)
						LO	HI
Factor model	406	7,490.33	0.000	0.853	0.101	0.095	0.106

df: Degree of freedom, CFI: Comparative fit index, RMSEA: Root mean square error of approximation, LO: Lower limit, HI: Upper limit.

There is RMSEA that is not affected by the size of samples. RMSEA is statistics that equals chi-square adjusted by using df and size of sample. Mostly, if RMSEA is lower than 0.08, it demonstrates that the goodness of fit is considerably high. Especially, within a 90% confidence interval, it demonstrates higher goodness of fit [9] when the upper value is lower than 0.08. The confirmatory factor analysis in this research demonstrated a favorable RMSEA (0.101), showing in 0.095 to 0.106 range in the 90% confidence interval. Also the confirmatory factor analysis demonstrated a reasonable comparative fit index as 0.853. A standard coefficient estimation shown in Fig. 1 demonstrates statistical significant in the level of significance 0.05. Standard coefficient estimation of the sub-function of three factors indicated 0.80 to 0.83.

Fig. 1.

Three-Factor Model of Mentoring Function

4. Result of reliability analysis

Cronbach α (Table 4) regarding the 29 items and the three factors were calculated to examine the reliability that measured the mentoring programs. As a result of the internal consistency of the scales, Cronbach α statistics for the factor 1, 2, and 3 were 0.955, 0.954, and 0.814, respectively. The overall Cronbach α statistic including all the 29 items is 0.971, demonstrated competent level of the reliability.

Table 4.

Reliability of Factors

Factor	Cronbach α
Total	0.971
Factor 1 (psychosocial function)	0.955
Factor 2 (career development function)	0.954
Factor 3 (friendship function)	0.814

5. Result of validity analysis

Construct validity was secured because standardized regression weights (Table 5) in all items were higher than 0.5. Construct reliability (Table 6) was confirmed because the internal consistency of the observed variables was higher than 0.7. Convergent validity was retained since all variance extracted indices was higher than 0.5. Regarding discriminant validity, a comparison of variance extracted indices (Table 6) and coefficients of determination (Table 7) shows that three of them are higher and three of them are lower, therefore it secured partial discriminant validity.

Table 5.

Standardized Regression Weights Regarding the Three Factors

Item	Factor	Estimate
n 8	Factor 1	0.790
n 5	Factor 1	0.794
n 6	Factor 1	0.763
n 1	Factor 1	0.712
n 12	Factor 1	0.829
n 7	Factor 1	0.753
n 2	Factor 1	0.799
n 10	Factor 1	0.800
n 9	Factor 1	0.765
n 3	Factor 1	0.783
n 14	Factor 1	0.738
n 4	Factor 1	0.744
n 13	Factor 1	0.801
n 15	Factor 1	0.821
n 18	Factor 2	0.779
n 19	Factor 2	0.805
n 21	Factor 2	0.804
n 17	Factor 2	0.790
n 16	Factor 2	0.729
n 24	Factor 2	0.850
n 20	Factor 2	0.834
n 23	Factor 2	0.821
n 25	Factor 2	0.824
n 11	Factor 2	0.722
n 26	Factor 2	0.815
n 22	Factor 2	0.787
n 29	Factor 3	0.677
n 27	Factor 3	0.746
n 28	Factor 3	0.869

Table 6.

Construct Reliability and Variance Extracted Index

Factor	Construct reliability	Variance extracted index
Psychosocial function	0.952	0.657
Career development function	0.971	0.702
Friendship function	0.812	0.592

Table 7.

Coefficient of Determination among the Factors

Factor	Coefficient of determination
Factor 1 & Factor 1	0.689
Factor 1 & Factor 3	0.640
Factor 1 & Factor 3	0.640

Discussion

The original Noe’s questionnaire “Mentoring Functions Item” consisted of 29 total items. Although the questionnaire was created in 1988, a lot of research [10,11] has been performed continuously based on this survey form. In various countries, other studies [12,13,14,15] investigating the function of mentoring programs have been published as well.

The questionnaire used in this study has been used widely in abroad and its validity has been proved. Also, a lot of research conducted in companies or schools in Korea have proven its validity. However, that was not the case for Korean medical students. Therefore, it was necessary to prove if the questionnaire can be applied to medical students. According to the results of factor analysis, the mentoring functions of Kram [6], which Noe quoted, were originally classified into two functions. However, we discovered a total of three functions with an emphasis on the friendship functions through our research conducted on Korean medical students. The order of Cronbach α statistics is as follows from the highest to the lowest: psychosocial functions (0.955), career development functions (0.954), and the friendship functions (0.814). This indicates that the overall reliability is high.

Thus, this adapted questionnaire used in this study is suitable for Korean students to measure the satisfaction of the mentoring programs. According to the study conducted in the United States [16], Noe’s questionnaire is the most frequently used questionnaire for evaluating mentoring programs, and it examines two functions–psychosocial functions and career development functions.

Kram [6] classified psychosocial and career development functions as the main functions of the mentoring programs. However, since then, many non-Korean studies propose that highly emphasized “role model functions” should be the third functions. After Kram classified the mentoring functions into two parts, many follow-up researchers [5] have maintained the same point of views. However, the study conducted by Scandura [17] added the “role model functions,” resulting in total three functions. This result shows some differences, because this study emphasizes “friendship functions,” not the role model functions.

The reliability and validity of each tri-functional questionnaires reach beyond satisfactory level. We have come to the conclusion that the research form can be applied to Korean medical students for the reason mentioned above. It is widely known that medical students are under so much pressure [18] because of the heavy workload, long period of study, and the pass/fail system. The fact that mentoring programs can be helpful for the medical students [19,20] has been proved through research. Considering the specific circumstances of medical school, it is important to develop measurement tools for Korean medical students to implement qualified mentoring programs.

This study confronts two limitations as follows. First, bilingual participants in Korean and English are ideal for comparing the original and the adapted questionnaire, but this study is far from that ideal process. The reason is that the researchers think it will give extra burden to medical students, who struggle with heavy workload although the students were bilingual. Second, the participants of the study are chosen only from Kyung Hee University, not from other universities. Due to the specific characteristics of mentoring programs that only Kyung Hee University possess, it is necessary to collect extra samples from other universities. In spite of these limitations, this study holds a significant meaning in that it provides an opportunity to adopt Korean version questionnaire about mentoring programs which are widely used in various countries. The purpose of the study is achieved sufficiently, since it proves its high reliability and validity in different aspects.

Through this study, we want to make two proposals. First, to secure the criterion-related validity of the study, it is necessary to use other studies with different measurement tools as criteria. This study cannot prove the criterion-related validity, due to the fact that there are no Korean measurement tools in which the reliability and validity have been qualified. Hence, the development of the measurement tools for mentoring programs should be implemented in follow-up studies. Second, this study is focused only on medical students. However, considering that interest in mentoring programs for residents and faculty members is increasing [21], measurement tools for mentoring programs for these people in follow-up studies should be developed.

In conclusion, the questionnaire used in this study proves its reliability and validity. Therefore, we are looking forward to applying this adapted questionnaire to other Korean medical schools regarding the research of mentoring programs.