Validating Self-Reflection and Insight Scale to Measure readiness for Self-Regulated Learning.

BACKGROUND: Professional behavior of physicians is under scrutiny by medical associations, media, and patients; therefore, medical students are expected to be self-directed learners rather than the passive ones. One of the useful strategies for professional development and life-long learning of students is self-regulated learning. Self-regulation concept and lifelong learning commitment are in the heart of medical practice. Therefore, this study aimed to evaluate the validity of Self-Reflection and Insight Scale (SRIS) to inspect the medical students' readiness for self-regulation.
MATERIALS AND METHODS: SRIS was translated according to the Sousa and Rojjanasrirat guideline. To examine the reliability and validity evidence of the scale, 136 medical students from Tehran University of Medical Sciences completed the questionnaire. Internal consistency and intraclass correlation were used to examine the reliability evidence, as well as qualitative content validity, and confirmatory factor analysis and exploratory factor analysis (EFA) were used to examine the construct validity of the scale.
RESULTS: The content validity of the scale was verified. Cronbach's alpha and the Interclass Correlation Coefficient value for the four-factor model was 0.87 and 0.79, respectively. Goodness-of-fit indices displayed acceptable and poor values (P = 0.0001, χ2 = 373.51, df = 167, Root Mean Square Error Of Approximation = 0.096, standardized root mean square residual = 0.12). EFA was conducted; a well-structured model was achieved through the EFA. The new four-factor model was extracted as the best model by performing EFA.
CONCLUSION: SRIS Persian version is saturated with four factors and has desirable content validity and constructs reliability.

Introduction

In the constantly changing world, medical doctors must continuously obtain the highest standards of patient care.[1] Professional behavior of physicians is under scrutiny by medical associations, media, and patients. Problematic behavior demonstrated by clinicians is a serious challenge[2] that affects the quality of patient care,[3] has a significant economic impact on hospital costs,[4] is a potential threat to the educational environment,[5] and could be harmful to well-being of the health-care team.[6]

Given the evidence suggesting that disruptive behavior of medical doctors is associated with their previous performance as students,[7] the crucial role of medical schools in helping medical students is evident. Medical students should be able to undertake responsibility in terms of identifying their own learning needs and learning activities. One of the useful strategies for professional development and lifelong learning of students is Self-Regulated Learning (SRL).[89]

In the field of educational research, SRL has been defined in different ways and many models have been developed to describe its process.[10111213] Moreover, a number of researchers have designed measurement instruments, in light of increasing interest in the identification and assessment of SRL and its related attributes.[14151617] Yet, one should notice that most of the widely used questionnaire measures other constructs, rather than SRL.[1819202122] Even the most verified instrument in SRL fails to assess post-action strategies such as self-reflection. However, Grant et al. designed the Self-Reflection and Insight Scale (SRIS) to measure two underlying constructs of self-regulation, namely self-reflection and insight.[23] They developed this instrument based on their generic model of self-regulation which was conceptualized in 2001[13] [Figure 1]. Self-reflection refers to inspection and evaluation of the thoughts, feelings, and behavior, while insight refers to the clarity of perceptions, emotions, and behaviors.[24] Both are considered as metacognitive features as the heart of self-regulation.[23] The reflecting ability on thoughts, feelings, and emotions lies on the basis of self-assessment and self-critical.[25] Reflective learning can improve professionalism, and reflective performance can stimulate self-regulation learning that leads to improved continuous performance and better management of the complex health system and patient improvement.[26] Furthermore, students should have insight into their knowledge and performance to be effective in self-regulation.[27]

Figure 1

Generic model of self-regulation and goal attainment showing role of self-reflection and insight. [Published with permission from A.M. Grant]

The SRIS in its original format had two subscales and 20 items. This scale has been validated in a couple of studies, and its reliability has been reported to be between 0.70 and 0.90.[2023282930] However, Roberts and Stark, who used this scale to measure students’ readiness for self-regulating of professional behavior, have identified one additional domain in their factor analysis. They named the factors as engagement in reflection, need for reflection, and insight. They claimed that this finding is consistent with Grant's basic conceptualization.[24] Since validity and reliability of a scale depend on the interpretation of the scores,[31] these controversial findings on the validity of SRIS led us conduct a study in another setting to validate this instrument. Moreover, considering the curricular reform of undergraduate medical programs, it seems necessary to have an authentic tool to examine the self-regulatory behavior of medical students. Hence, to have a tool that can measure the self-reflection and insight of medical students which are central to the development of SRL in medical students, this study aimed to investigate the validity of the SRIS.

Materials and Methods

Study setting

The participants included medical students of Tehran University of Medical Sciences (TUMS) who were in the clerkship training and volunteered to be enrolled in the study. The study was approved by the Ethics Committee of the TUMS (IR.TUMS.MEDICINE.REC.1395.1722). We collected informed consent before the data collection and assured the students about anonymity and confidentiality of data.

Instrument

According to the study conducted by Roberts and Stark, the SRIS, a self-report 20-item tool, consists of three subscales including insight (8 items), need for reflection (6 items), and engagement in reflection (6 items).[24] A six-point Likert scale (1 = strongly disagree, 2 = slightly disagree, 3 = disagree, 4 = slightly agree, 5 = agree, 6 = strongly agree) is used to score responses. Higher scores reflect more inspection and evaluation as well as the perception of thoughts, beliefs, feelings, and behaviors.[29] The scale score is obtained from the sum of scores of items.

Translation of the scale

The English version of SRIS was translated into Persian, according to the guideline developed by Sousa and Rojjanasrirat.[32] Two Iranian English professors individually translated the tool. We compared and combined the two versions to present the best wording and terms. The translated questionnaire, then, was revised by three faculty members, two PhD in medical education and one clinical specialist. They corrected vague terms and accommodated the questionnaire for the usage of medical students. Next, the Persian version was retranslated into English by two experts in the English language to ensure the semantic equivalence of the scale. We compared the retranslated version to the original tool, so a series of ambiguous word and complicated sentences were identified. As a result, we made some changes in items 2, 6, 7, and 13 in the Persian version.

Evaluating content validity of the tool

We tested the content validity of the questionnaire using a qualitative approach. We invited five faculty members (three PhDs and two clinical specialists) to discuss and express their opinion about the relevance, importance, and clarity of items in the questionnaire. The panelists decided whether each item should be retained, modified, or removed from the tool. We also asked the panel members to indicate any other items that they think should be added to the questionnaire. This effort led to modifying and restructuring several items to avoid using negative verb in the sentence. Moreover, the ordering of questions 2 and 4–6 was displaced in the Persian version, because the negative sentences were one after another which might have caused confusion for students.

In addition, we asked 10 medical students to complete the questionnaire and to highlight difficult and vague items. Based on their feedback, some minor modifications were made, and the words “self-reflection” and “insight” were defined in footnotes. In this way, the Persian version of SRIS was finalized.

Evaluating construct validity of the tool

To evaluate the construct validity of the tool, we conducted confirmatory factor analysis (CFA) and exploratory factor analysis (EFA). For this purpose, the Persian SRIS was distributed among TUMS medical students who were in the clerkship rotations during the summer 2017. For CFA, we considered maximum likelihood estimation method using LISREL8/83 software (Scientific Software International, Skokie, IL, USA). Moreover, to determine the fitness of the proposed model for the data, we computed a series of indices including Chi-square ratio, normal fit index (NFI), nonnormed fit index (NNFI), comparative fit index (CFI), goodness-of-fit index (GFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). EFA was conducted due to acceptable and weak values of goodness-of-fit indices. A well-structured model was achieved through the EFA. EFA was examined by varimax rotation method using SPSS-19 software (IBM, Armonk, NY, USA).

Evaluating reliability of the tool

Reliability of the tool was calculated by internal consistency and test–retest reliability. Internal consistency of the tool was measured by computing Cronbach's alpha coefficient. Moreover, “alpha if item deleted” for each item was calculated. To inspect test–retest reliability, a subgroup of the same medical students completed the questionnaire twice, separated by 2-week interval, and interclass correlation coefficient (ICC) was calculated. The satisfactory value of Cronbach's alpha and ICC was considered (≥0.70).[33]

Results

One hundred and thirty-six students completed the questionnaire. The students’ average age was 23.6 years (standard deviation = 1.24), and 66 students (48.5%) were female. There were no missing data. The range of students’ scores was between 60 and 117 (out of 120). Table 1 shows detailed descriptive statistics for each item, each subdomain, and the total score.

Table 1

Statistics of ratings by clerkship students to the 20-item self-reflection and insight scale questionnaire (Persian version) on a six-point Likert scale (strongly disagree: 1 to strongly agree: 6) (n=136)

Items	Subscales	Mean (SD)	Cronbach’s alpha, if item deleted	ICC of items	Factor loading
Insight
3	I am usually aware of my thoughts	4.91 (0.65)	0.83	0.55	0.37
4	I am often confused about the way that I really feel about things	3.98 (1.27)	0.80	0.65	0.74
6	I usually have a very clear idea about why I have behaved in a certain way	4.31 (1.09)	0.82	0.73	0.53
9	I am often aware that I am having a feeling, but I often do not quite know what it is	3.83 (1.32)	0.80	0.19	0.80
11	My behavior often puzzles me	4.41 (1.20)	0.80	0.65	0.70
14	Thinking about my thoughts makes me more confused	4.76 (1.20)	0.82	0.72	0.54
17	Often I find it difficult to make sense of the way I feel about things	4.19 (1.19)	0.81	0.74	0.72
20	I usually know why I feel the way I do	4.38 (1.07)	0.82	0.74	0.66
Need for reflection
12	It is important to me to try to understand what my feelings mean	4.88 (1.00)	0.76	0.88	0.77
15	I have a definite need to understand the way that my mind works	4.3 (1.05)	0.56	0.37	0.73
18	It is important to me to be able to understand how my thoughts arise	4.64 (1.02)	0.52	0.28	0.70
Intention for reflection
1	I do not often think about my thoughts	4.97 (0.95)	0.64	0.79	0.62
2	I am not really interested in analyzing my behavior	4.81 (1.07)	0.63	0.29	0.64
5	It is important for me to evaluate the things that I do	4.99 (0.82)	0.69	0.30	0.48
7	I am very interested in examining what I think about	4.91 (0.86)	0.65	0.30	0.55
13	I do not really think about why I behave in the way that I do	4.81 (0.92)	0.62	0.52	0.60
Engagement in reflection
8	I rarely spend time in self-reflection	4.38 (1.09)	0.74	0.66	0.72
10	I frequently examine my feelings	4.41 (1.02)	0.71	0.66	0.47
16	I frequently take time to reflect on my thoughts	4.58 (0.96)	0.67	0.03	0.62
19	I often think about the way I feel about things	4.75 (0.83)	0.77	0.36	0.74

SD=Standard deviation, ICC=Interclass correlation coefficient

Confirmatory factor analysis

The t-values obtained for this tool indicate that all paths are significant at the 0.001 level. Goodness-of-fit indices displayed acceptable and weak values (P = 0.0001, χ2 = 373.51, df = 167, RSMEA = 0.096, SRMR = 0.12, GFI = 0.78, CFI = 0.93, NFI = 0.85, NNFI = 0.92). Thus, the most effective software routes were developed to modify the model. Two covariance relationships were plotted between questions 8 and 12 and questions 15 and 18 [Figure 2]. However, drawing these two paths did not significantly improve the fit indices (P = 0.0001, χ2 = 323.21, df = 165, RSMEA = 0.084, SRMR = 0.11). The NFI, NNFI, and GFI should be ≥0.95 while it was 0.87, 0.81, and 0.93, respectively, in the given paper. The CFI should be ≥0/09, while it was 0.94 in the study. Both SRMR and RMSEA should be <0.08.[3435] The significance level of χ2 would be >0.05 as well as χ2/df value between 2[36] and 5,[37] which appeared to be χ2/df = 1.95 in the current study.

Figure 2

Three-factor confirmatory factor analysis model of the self-reflection and insight scale showing standardized factor loadings

Exploratory factor analysis

The EFA was conducted with the principal component analysis method and the varimax rotation. Before the factor analysis, sampling adequacy using the Kaiser-Meyer-Oklin ( KMO) test and rejecting the null hypothesis based on the correctness of the matrix of homogeneity in the population were confirmed to be meaningful at level 0.001 by Bartlett's test of sphericity that the matrix is homogeneous, respectively (KMO = 0.844, χ2 = 1016/72, P < 0.001). In other words, implementing the factor analysis is justifiable. According to Andersson et al., the KMO values >0.6 were acceptable for factor analysis.[38] The extracting criterion of the factors was the scree plot curve and the eigenvalue was above 1. Four factors namely insight, need for reflection, intention for reflection, and engagement in reflection were extracted [Table 1]. These four factors explained the 56.8% of the variance of the total variables. The factor matrix demonstrated that the first factor has more factor loading (18.23) and contribution than other factors.

Based on the results of EFA and cutpoint of eigenvalue, none of the items were omitted. Items which were jointly associated with a same factor and made a subscale were extracted [Table 2]. We identified four factors as follows:

Table 2

Loadings from exploratory factor analyses: Orthogonal rotation (all factor loadings) (n=136)

Items	Component
	1	2	3	4
q9	0.796
q4	0.742
q17	0.716
q11	0.690
q20	0.656
q6	0.535
q2		0.642
q1		0.618
q13		0.597
q7		0.554
q14		0.538
q10		0.528	0.466
q5		0.480
q19			0.745
q8		0.439	0.717
q16		0.438	0.623
q3	0.374		0.384
q12				0.774
q15				0.734
q18				0.709

Insight: Items 3, 4, 6, 9, 11, 14, 17, 20

Need for reflection: Items 12, 15, 18

Intention for reflection: Items 1. 2, 5, 7, 13

Engagement in reflection: Items 8, 10, 16, 19.

The new four-factor model, as the best one, accounted for 56.8% of the total variance of the variables. Table 3 considers the eigenvalues, the variance of each factor, and cumulative percentage of the factors. Varimax rotation was used to better differentiate the factors.

Table 3

Eigenvalue, percentage of explanation of variance, and cumulative percentage of four factors, n=136 of self-reflection and insight scale

Factors	Before rotation			After rotation
	Eigenvalue	Percentage value	Cumulative percentage	Eigenvalue	Percentage value	Cumulative percentage
1	6.149	30.746	30.746	3.646	18.232	18.232
2	2.629	13.146	43.892	3.230	16.148	34.380
3	1.520	7.599	51.491	2.286	11.429	45.809
4	1.061	5.306	56.797	2.198	10.988	56.797

Reliability of the tool

Twenty medical students completed the questionnaire twice. Forty questioners were collected in the first and second administrations. The Cronbach's alpha of the new four-factor scale for the whole scale was 0.87 and the subscales of insight, need for reflection, intention for reflection, and engagement for reflection had an alpha of 0.83, 0.71, 0.70, and 0.78, respectively. None of the “alpha if item deleted” values were greater than overall alpha.

The ICC value was 0.80 for the whole scale, and 0.84, 0.51, 0.58, and 0.65 the above-mentioned subscales, respectively. The related values for each subscale are presented in Table 1.

Discussion

The current study aimed to examine the validity evidence of SRIS to assess the readiness of medical students for SRL in an Iranian university.

The content validity of SRIS was confirmed by the experts. The reliability of the tool was supported by good-to-excellent internal consistency and test–retest reliability. In Grant et al.'s investigation, the Cronbach's alpha for subscales of self-reflection and insight along with the reliability of test–retest was reported 0.91, 0.87, 0.77, and 0.78, respectively.[23] The Cronbach's alpha reliability in the study by Roberts and Stark was reported as 0.83, 0.87, and 0.85 for the subscales, namely, engagement in reflection, need for reflection, and insight, and 0.88 for the total scale.[24] In the study of Aşkun and Çetin in Turkey, the Cronbach's alpha for the subscales of insight and reflection and the total items was 0.65, 0.80, and 0.70, respectively.[30] Chen et al. investigated the psychometric properties of the SRIS in China. The Cronbach's alpha was 0.83 for insight, 0.87 for self-reflection, and 0.79 for the total scale.[29] All these differences in the coefficients could be interpreted that reliability is not feature of the tool-like validity. A tool used in different subjects can indicate wide variations in reliability.[31]

The results of this study were congruent with that of Aşkun and Çetin. In their study, factor analysis of SRIS ver. 2008 indicated inadequate fit indices. Yet, the results were inconsistent with Chen et al.'s results done on ver. 2002 of SRIS two-factor scale. Further, their CFA displayed that two-factor model possesses goodness-of-fit indices (RSMEA = 0.057, χ2/df = 3.29, CFI = 0.98, GFI = 0.96, NFI = 0.97, and SRMR = 0.64).

In Grant et al.'s study, an EFA was conducted by using varimax rotation, generally indicated two factors, including 20 items. Subscales of self-reflection (12 items) and insight (8 items) explained 56% of the total variance, which is in line with the present study. At the present study, the four factors explained the 56.8% of the variance of the total variables.

Application of results

This study had provided robust evidence about the psychometric characteristics of an instrument, which had been developed to measure self-regulation among medical students. In addition, we have introduced the Persian version of the instrument to the Iranian community of educational health professions. We believe that students, faculty members, and program directors in different settings would benefit from this scale in different ways. These applications include facilitating SRL among students, preparing them for better management of the complicated problems, planning for curricular revisions, and monitoring curriculum changes.

Limitations of the study

One of the limitations of the present study is that it has conducted in just one university, which could have an impact on the generalizability of results. It is suggested to validate the tool in other settings and universities. In addition, this study failed to evaluate the relation of this tool with other instruments which measures self-regulation construct. Investigating this correlation as well as working more on confirming four factors related to this instrument in different fields is recommended.

Conclusion

The validity evidence of SRIS was assessed to evaluate the medical students’ readiness for SRL. The results indicated that the scale is saturated with four factors in the Iranian community with acceptable reliability.

Financial support and sponsorship

This article has been derived from a thesis research project and received financial support from TUMS (Grant No. 35253).

Conflicts of interest

There are no conflicts of interest.