Validity testing of classroom community scale in virtual environment learning: A cross sectional study.

BACKGROUND: Fostering a Sense of Classroom Community is considered to be associated with the reduction of student's dropout rate in an online environment. Many scales have been developed to measure sense of community in online and traditional learning, and Rovai's Classroom Community Scale has been widely used. This study was designed to examine the psychometric properties and the theoretical structure of the Classroom Community Scale.
MATERIALS AND METHODS: A total of 215 postgraduate virtual students responded to the Classroom Community Scale. A measurement model was evaluated using confirmatory factor analysis to determine the adequacy of goodness-of-fit to sample data.
RESULTS: The confirmatory factor analysis provided valid data that the Classroom Community Scale with a two-factor structure is a valid scale with adequate model fit. Connectedness and learning subscales were also valid and reliable. Overall, the results supported the high reliability, face and content validity of all items of the scale.
CONCLUSIONS: The 20 item Classroom Community Scale provides a valid and reliable scale to measure sense of community among postgraduate medical education students.

Introduction

During the past few decades, online education has exponentially grown in higher education [1]. Recent findings indicate that one third of all higher education students take at least one online course and almost 70% of higher education organizations consider online education to be essential in their long-term approach [2]. The fast growing online education programs supported by emergent technologies has created serious concerns about the quality of online learning experiences to teachers and educational leaders [3], meaning that the accessibility to online learning, does not necessarily increase the quality of learning experiences. Indeed, teachers should use internet to meet learners' needs [4]. The higher rate of student dropout in online education compared with face-to-face learning, is mainly attributed to student isolation [[5], [6], [7]]. Patterson and McFadden in 2009 indicated that dropout rates in virtual learning is six to seven times more than face-to-face learning [8]; therefore, student retention in online programs has been certainly considered an important issue [9].

Several studies have been conducted on the reasons of dropout rate in online programs [[10], [11], [12]]; the results indicated that fostering a sense of classroom community (SCC), would reduce students dropout rate in the online environment [13,14], however there is still a considerable amount of uncertainty regarding the measurement of sense of community concept in online learning. This study was therefore conducted to explore the validity and reliability of the Classroom Community Scale (CCS) developed by Rovai (2002) for measuring SCC in online learning in Iran.

Theoretical framework

The term ‘community’ in the Dictionary of Sociology, is defined as a “place-oriented concept” [15]. In today's world, the meaning of ‘community’ is changing from geographical feature to communicative approach and it is important to define this term [16]. According to McMillan and Chavis in 1986 sense of community is “a feeling that members have of belonging, a feeling that members matter to one another and to the group, and a shared faith that members' needs will be met through their commitment to be together” [17]. In a virtual learning environment, the sense of community has a specific definition and is mostly known as “a sense of virtual community”. Blanchard in 2007 defined sense of virtual community as “members” feelings of membership, identity, belonging, and attachment to a group that interacts primarily through electronic communication [18]. there are several instruments available in the literature for measuring SCC in online environment, such as Social Presence and Privacy Questionnaire (SPPQ) [19], Social Presence Questionnaire of Online Collaborative Learning (CSCW) [20], Sense of Community Survey [21], Online Student Connectedness Survey [22], Rovai's CCS is the most commonly used instrument [23]. Rovai in 2002, suggested the following seven factors for measuring SCC in the virtual environment: transactional distance, social presence, social equality, small group activities, group facilitation, teaching style and learning stage and community size, the two factors Connectedness and Learning were identified as latent factors in factor analysis. Rovai in 2002, believed that strong feelings of SCC would decrease feelings of isolation and alienation from the institution in virtual environment [24]. Research results suggest that student burnout and feelings of isolation are related to low SCC [25,26].

Rovai's CCS has been used in several settings to measure sense of community among virtual students [27], it assists educators and policy makers to assess the success of educational interventions aimed at promoting a sense of virtual community. Hill in 1996, recognized the necessity of research in a different context to understand SCC concept and assumed that the construct of SCC varied from setting to setting. Since it is context specific [28].

Although extensive research has been conducted on SCC, there is no valid and reliable measure of SCC in face to face and particularly in online learning environment in Iran. Consequently, the validation of a concise and reliable scale, such as Rovai's CCS, could contribute to the knowledge of SCC in virtual environment and provide a good opportunity to compare this issue among different countries. Therefore, the purpose of this study was to investigate the validity and reliability of a translated version of Rovai's CCS.

Methods

Design

This quantitative study was conducted between March 2016 and February 2017 in Iran.

Participants

The sample consisted of 215 postgraduate volunteer students who had spent their first semester. The participants were recruited from five medical universities. No reward was provided for virtual students participating in the study.

Instrument

The Rovai's Classroom Community Scale (CCS) was used to assess psychometric properties. The scale consists of 20 five-point Likert-type items with choices ranging from 0 = strongly agree to 4 = strongly disagree. The odd-numbered items were related to feelings of connectedness subscale and the even-numbered items were related to learning subscale. Total score on CCS is expressed as the sum of all answers weighted. According to Rovai in 2002, the connectedness scale “represents the feelings of the community of students regarding their connectedness, cohesion, spirit, trust, and interdependence [29], and the learning scale “represents the feelings of community members regarding interaction with each other as they pursue the construction of understanding and the degree to which members share values and beliefs concerning the extent to which their educational goals and expectations are being satisfied” [29].

Procedures

The World Health Organization protocol was carried out using a forward-backward translation technique [30]. The Persian version of the scale which contained the purpose of the study was disseminated to virtual students through email.

Statistical analyses

Validity assessment

Face validity: In this part, both quantitative and qualitative methods were applied. For quantitative part, 10 virtual students were asked to evaluate the CCS questionnaire and score the importance of each item on a five-point Likert scale in order to calculate ‘Item Impact Score’ (Impact Score = Frequency (%) × Importance). The impact score of 1.5 or above was considered satisfactory as recommended [31]. For the qualitative part, the same students were asked about the ‘relevancy’, ‘ambiguity’, and ‘difficulty’ of the items; and some minor changes were made to the preliminary questionnaire.

Content validity: A panel of content expert consisting of 12 specialists in medical and virtual education was given the 20 CCS items to evaluate for content validity. Qualitative content validity was determined based on ‘grammar’, ‘wording’, ‘item allocation’, and ‘scaling’ indices. All items were checked and modified according to the recommendations of the panel. Content Validity Ratio (CVR) and Content Validity Index (CVI) were calculated for quantitative content validity. The CVR is an item statistic useful in rejection or retention of individual items and is internationally recognized as the method for establishing content validity [32]. The CVI is the mean CVR for all the items included in the final instrument [33].

In order to calculate CVR, the expert panel was asked to evaluate each item using a three-point Likert scale: 1 = essential, 2 = useful but not essential, and 3 = unessential. Then, according to Lawshe's table [31]. Items with CVR score of 0.56 or above were selected. For the CVI, based on Waltz and Bausell [34], the same panel was asked to evaluate the items according to a four-point Likert scale on ‘relevancy’, ‘clarity’, and ‘simplicity’; Polit and Beck in 2006 stated CVI score of 0.80 or above was considered satisfactory [35].

Factor analysis

There are two types of factor analysis: Confirmatory Factor Analysis (CFA) and Exploratory Factor Analysis (EFA) [36]. EFA is used to find the latent factors of a scale [37].

CFA

The appropriateness of the factor structure of the CCS was evaluated through CFA using LISREL software. The following six common measures were used to assess the model's overall goodness of fit: chi-square/degree of freedom ratio, Normed Fit Index (NFI), Non- Normed Fit Index (NNFI), Comparative Fit Index (CFI), and Root Mean Square Error of Approximation (RMSEA) and Standardized Root Mean Square Residual (SRMSR). Accepted level of these measures are shown in Table 3.

Table 3

The results of intra-class correlation reliability (N = 30).

	Intra-class Correlation	Confidence Interval (95%)	PV
Total tool	.939	0.808–0.975	P < 0.001
Dimensions of scale
Connectedness	.935	0.843–0.071	P < 0.001
Learning	.944	0.865–0.975	P < 0.001

In order to evaluate the reliability of the Persian version of the CCS, internal consistency, test–retest analyses, and construct reliability were performed. Reliability Cronbach's coefficient alpha was obtained for the scale as a whole and for each subscale. Furthermore; intra-class correlation coefficients (ICCs) were calculated to establish the test–retest reliability of the CCS over an interval of 2 weeks [38]. Cronbach's alpha value of 0.7 or greater [26] and ICC of 0.4 or greater [25] were considered acceptable [38].

Sampling method

The minimum sample size to conduct the factor analysis is a ratio of 5/10:1, that is, 5–10 times more than the number of the items of the intended instrument [39]; consequently, a sample of 215 virtual students was recruited through convenience sampling method.

Results

The participants consisted of 215 (131 women, 84 men) postgraduate virtual students with an average age of 42 ± 4; the details related to other demographic characteristics of the students are presented in Table 1.

Table 1

The demographic characteristics of the virtual students.

Variable	Category	Number of participant	%
Course	e-learning	17	7.9%
	Health education	19	8.8%
	medical education	146	67.9%
	Medical librarianship	14	6.5%
	Drug supervision	19	8.9%
Education level	MA/MS	192	89.1%
	PhD	10	4.7%
	MPH	12	5.6%
Setting	Tehran University	73	34%
	Shahid Beheshti University	24	11.2%
	Iran University	59	27.4%
	Shiraz University	29	13.5%
	Isfahan University	20	4.8%

The CVR and CVI values of items of CCS were greater than 0.55, and 0.80, respectively (Table 2); therefore, none of the items were excluded in these steps of psychometric evaluation.

Table 2

The CVR and CVI values for all items of CCS.

Domain	Item Number	Items	CVI			CVREssential [[1], [2], [3]]
			Simplicity [[1], [2], [3], [4]]	Relevancy [[1], [2], [3], [4]]	Clarity [[1], [2], [3], [4]]
Connectedness items (n = 10)	1	I feel that students in this course care about each other	0.86	0.8	0.93	0.78
	3	I feel connected to others in this course	0.86	1	0.8	1
	5	I do not feel a spirit of community	0.86	0.93	0.8	0.86
	7	I feel that this course is like a family	0.86	1	0.93	1
	9	I feel isolated in this course	1	1	1	1
	11	I trust others in this course	0.86	0.93	0.8	1
	13	I feel that I can rely on others in this course	0.86	0.8	0.8	0.78
	15	I feel that members of this course depend on me	0.93	1	0.93	1
	17	I feel uncertain about others in this course	0.93	1	1	0.86
	19	I feel confident that others will support me	0.86	1	0.8	1
Learning items (n = 10)	2	I feel that I am encouraged to ask questions	0.86	1	0.8	0.86
	4	I feel that it is hard to get help when I have a question	1	1	0.93	1
	6	I feel that I receive timely feedback	0.86	1	0.8	1
	8	I feel uneasy exposing gaps in my understanding	0.86	093	0.93	1
	10	I feel reluctant to speak openly	0.86	1	0.8	1
	12	I feel that this course results in only modest learning	0.93	1	1	1
	14	I feel that other students do not help me learn	1	1	0.86	1
	16	I feel that I am given ample opportunities to learn	1	0.93	0.86	1
	18	I feel that my educational needs are not being met	1	1	0.8	1
	20	I feel that this course does not promote a desire to learn	1	1	1	0.86

The intra-class class correlation coefficient (ICC) was measured using test-retest reliability. The ICC for the total scale, and for the connectedness and learning subscales were 0.939, 0.935 and 0.944, respectively (Table 3). Also, the internal consistency (Cronbach's alpha) for the total scale and for the connectedness and learning subscales were 0.87, 0.86 and 0.85, respectively (Table 4).

Table 4

The results of reliability with the internal consistency method (Cronbach's alpha) (N = 215).

	Number of Item	Cronbach's Alpha Coefficient
Total tool	20	.87
Dimensions of scale
Connectedness	10	.86
Learning	10	.85

Confirmatory factor analysis was performed to confirm the relationship between 20 items to the 2 factors of connectedness and learning determined. There has been used the confirmatory factor analysis to confirm the relation of 20 items to 2 factors determined. As shown in Table 5, all the indices exceeded their commonly accepted levels, demonstrating that the model exhibited a good fit with the collected data.

Table 5

Fit index for measurement model.

Index	Measurement Model	Recommended Value	References
X2/d.f	2.71	≤3	[40]
Normed Fit Index (NFI)	.96	≥0.95
Non- Normed Fit Index (NNFI)	.97	≥0.95
Comparative Fit Index (CFI)	.98	≥0.95
Standardized Root Mean Square Residual (SRMSR)	.041	≤0.08
Root Mean Squire Error of Approximation (RMSEA)	.073	0.06–0.08

Discussion

Given the worldwide use of CCS within the context of virtual education, it was important to investigate the factor structure and psychometric Properties of the Persian version of the CCS distributed to a sample of 215 Iranian virtual students. Overall, the results supported the scale's underlying factor structure and provided evidence of high reliability. The face and content validity, established during the process of developing the instrument were in line with the findings of Rovai in 2002, and Shackelford and Maxwell in 2002, the use of expert opinion facilitates the development and validation of scales because it is a simple, and valuable procedure for gathering content-related information about a concept [40].

The same method was adopted in the current study; the content of the original CCS scale was reviewed and necessary modifications were made to ensure that the scale was culturally applicable in Iran. The results also indicated that internal consistency and ICC were outstanding for the total scale and the subscales. Adequate internal consistency for full CCS and subscales shown in Table 4, nearly concurred with those of the original English CCS scale which reported values of 0.93 for total scale and 0.92 for connectedness and 0.87 for learning subscales.

The CCS scale also showed excellent test-retest reliability with an ICC value of 0.93 and with values of connectedness learning subscales which were 0.93 and 0.94, respectively (See Table 3).

The Rovai's Classroom Community Scale (CCS), also equal-length split-half coefficient was 0.8 and indicating good reliability [29]. In this study, assessment of fit between the model and the observed variables (items) was presented through the CFA approach and fit index showed that the model concurred with the data and provided the best fit with observed variables. In other words, CFA provided valid data that the CCS scale with two-factor structure was a valid scale with adequate model fit. The two subscales of connectedness and learning were also valid and reliable. The findings of this study are subject to at least two limitations. First, the data for CFA were obtained from a convenience sample of 215 virtual students, and, therefore, the generalizability of these results to other students could be limited. Second, the present study was mainly focused on postgraduate students; therefore, further research might explore whether the two-factor CCS structure is consistent across undergraduate and graduate students. Notwithstanding these limitations, the study provided a valuable means of assessing the psychometric properties of the CCS especially by a CFA in Iran. Also, the number of participants (N = 215) was relatively large to conduct the psychometric evaluation and the factor analysis. Therefore, further research is recommended considering different populations, places, and time periods in Iranian context.

Recommendations for future research

Future research should examine the psychometric properties of CCS scale in face to face learning environment and compare results with virtual environment.

Conclusions

The Persian version of CCS is a psychometrically sound scale to measure SCC in virtual environment; it is a valuable assessment tool that can be used for various purposes [1]: to measure SCC in online learning [2]; to measure the efficacy of courses designed to promote online classroom community [3]; to reduce feelings of isolation experienced in online learning [4], to create classroom and school environments that promote community learning, and [5] to enhance student satisfaction, learning, and persistence in virtual environments.

Ethical considerations

The study was approved by the Ethics Committee of Tehran University of Medical Sciences (Code: IR.TUMS.MEDICINE.REC.1395.713). The students were informed of the objectives of the study and the participation in the study was voluntary.

Sources of funding

No source of funding for research.

Author contribution

Dr Soleiman Ahmady: writing the paper, study concept or design, data collection, data analysis and data interpretation.

Dr Noushin Kohan:writing the paper, data analysis or interpretation.

Dr Rafaat Bagherzadeh: writing the paper, data analysis or interpretation.

Dr Tayebeh Rakhshani: writing the paper, data analysis.

Maryam Shahabi: writing the paper, study concept or design, data collection, data analysis and data interpretation.

Conflicts of interest

No conflicts of interest.

Research registration number

Research not involving human participant.

Guarantor

Dr Noushin Kohan.

Maryam Shahabi.

Consent

Research studies don't require consent because not involving patient.

Provenance and peer review

Not commissioned, externally peer reviewed.