School and Teacher Information, Communication and Technology (ICT) readiness across 57 countries: The alignment optimization method.

This study investigated the measurement invariance of school and teacher Information, Communication and Technology (ICT) readiness among 57 countries that participated in the Program in International Student Assessment (PISA) 2018 assessment. School and teacher ICT readiness scale is 11-item scale with two subfactors: school ICT readiness and teacher ICT readiness subscales (Bozkus, International Online Journal of Education and Teaching, 8(3), 1560-1579, 2021). With the novel alignment optimization method, we revealed that the school ICT readiness subscale was invariant for unbiased country comparisons but overall noninvariance was identified for the teacher ICT readiness subscale. Additionally, the rank of the school ICT readiness factor means indicated that Singapore, Sweden, B-S-J-Z (regions of China), United Arab Emirates and United States were among the top league, while countries like Indonesia, Poland, Ireland in between, and Japan, Mexico, Colombia, Argentina and Brazil ranked comparatively the lowest. Measures of school location, school type and class size further confirmed the validity of the school ICT readiness subscale. It was expected that the study would enhance our understanding of school and teacher ICT readiness across countries with the application of an alternative alignment optimization approach in examining ICT related scales.

Introduction

The digital age in the last decade has witnessed a rapid development of Information and Communication Technology (ICT) in the education field across the world (Cradler et al., 2002; OECD, 2016; Sezer, 2017). A growing body of literature has examined the ICT-related constructs as well as how it reflects educational quality and equity (Gumus & Atalmis, 2011; Lowther et al., 2008; Novak et al., 2018). Currently, under the ongoing circumstances of the worldwide COVID-19 pandemic, digital technology has especially gained much more attention for schools worldwide since online teaching becomes a necessary alternative to in-person classroom teaching (Kong et al., 2022). Nonetheless, facing the global emergency, schools have met unpredictable complex challenges, such as lack of infrastructure of digital device (Kim et al., 2021) and teachers’ limited capacity using digital device (Bozkus, 2021). All of these have unraveled the important supporting role of a good school and teacher ICT readiness environment to students with online learning (Morse, 2004; Norris, 2001; Van Dijk, 2020).

Throughout the years, the international large-scale assessments such as Program for International Student Assessment (PISA) have recognized the significance of ICT and measured ICT use across the countries through various aspects, such as school ICT resources, teacher ICT self-efficacy, and student ICT familiarity, skills, self-efficacy and engagement (OECD, 2005, 2016; 2019a, b). In PISA 2018, a developing scale consisting of school infrastructure of digital device and teachers’ capacity using digital devices was used to measure school and teacher ICT readiness across the countries. To make valid cross-country comparisons, an assessment to ensure the scales function the same way across the countries is a prerequisite. However, it remains an open question as to whether cross-cultural invariance of many ICT-related scales is supported. Therefore, establishing measurement invariance is an indispensable pre-procedure to ensure the scale’ validity when researchers aim to conduct multiple country comparisons and latent mean comparisons (Tracey & Xu, 2017; Vandenberg & Lance, 2000). A novel measurement invariance alignment method, proposed by Asparouhov and Muthén (2014), has been gradually recognized by researchers in conducting invariance tests across multiple groups.

To date, there has been no in-depth measurement invariance analysis of the scale measuring PISA 2018 school and teacher ICT readiness. There is a great need to examine whether this feature holds before it is widely adopted into use for researchers to conduct comparisons directly. Therefore, the first purpose of this study is to examine whether the measurement invariance of the scale across the countries is supported at an acceptable level with alignment method, using a set of items provided by PISA 2018. Second, if measurement invariance of the scale holds, the countries’ ICT readiness factor scores are comparable. Lastly, we would like to select a few relevant variables to examine group mean differences as validation measures. Specifically, this study starts by outlining the conceptual background of PISA 2018 school and teacher ICT readiness scale, and elaborating on the logic of alignment method, literature of application of alignment method on ICT-related scale to test for measurement invariance; then, followed by the detailed data selection, analysis procedure and result; lastly ended with a discussion of the methodological and practical significance of this approach to measure invariance of ICT readiness scale, with attention to its implications for future cross-country research using large-scale surveys.

Conceptual background

A large body of existing studies have been exploring the reasons of successful ICT implementation in schools, from the individual student and teacher level to the wider school context (e.g., Davies & West, 2014; Eickelmann, 2011; Inan & Lowther, 2010; Lim et al., 2013; Petko, 2012). Petko (2012) proposed a “skill, will, tool” theoretical model emphasizing the teacher skills, self-efficacy in technology can achieve the technology integration together with the infrastructure of digital devices within schools. School infrastructure of digital devices is usually considered as school ICT readiness, a prerequisite for supporting teachers’ capacity using digital devices (Liu et al., 2016; Petko, 2012; Petko et al., 2018). Liu et al. (2016) have identified a few factors, such as school technology support, and school access to technology in classroom as impacts affecting classroom technology integration. Teachers’ capacity to utilize the digital devices is often considered as teacher ICT readiness, more successful to implement when having the school support (Daly et al., 2009; Richardson & Placier, 2001).

School ICT readiness: School infrastructure of digital devices

Infrastructure of digital devices that schools provide, regarded as school ICT readiness, affects both the way that teachers use for teaching, and students’ ICT-related learning quality, engagement and familiarity in use (Fraillon et al., 2014; Lau & Sim, 2008; Liu et al., 2016; Ma & Qin, 2021; Murillo & Román, 2011; Woessmann & Fuchs, 2004; Zhang & Liu, 2016). Digital infrastructure is identified as the first barrier in technology integration even among the current “digital native” generation who grew up with digital technologies (Li et al., 2015; Sang et al., 2011). Students in a technology-rich school environment are more motivated, more confident in their digital abilities and tend to perform better in ICT-related performance (Sun et al., 2018; Wastiau et al., 2013). With a multi-level data analysis, Liu et al. (2016) found out that school digital access and support greatly affected teacher use of technology and classroom technology integration in primary schools. Saal et al. (2021) reported that computer availability and frequency of use in the mathematical classes were positively associated with the students’ mathematics achievement. An inverted U-shape relationship was identified between school internet use and student performance in mathematics and reading (Woessmann & Fuchs, 2004). Factors that affect the frequency of digital use and technology integration in schools are such as, colleagues’ support and principals’ discretion over technology spending (Karaca et al., 2013; Miranda & Russell, 2011).

Teachers ICT readiness: Teachers’ capacity using digital devices

Teachers’ capacity using digital devices, another aspect of digital readiness, has often been considered as teacher ICT readiness. It refers to the teachers’ confidence in their abilities to use digital technologies and willingness to utilize them in education (Fraillon et al., 2014; Petko et al., 2018; Wastiau et al., 2013), which directly associates with successful technology implementation and integration in schools (Petko et al., 2015; Petko et al., 2018). Many factors impede the realization of teachers’ capacity utilizing the digital devices. It is claimed that teachers’ belief and lack of sufficient skills were the main obstacles impeding the digital technology integration (Petko et al., 2018). First, teachers who get used with the traditional approach of teaching has a pedagogical prejudice and negative belief against the usage of digital technologies in the classroom (Ertmer et al., 2015). Teachers’ self-efficacy affects their confidence in effectively using digital technology for education as well (Fraillon et al., 2014). Second, if teachers do not feel competent and lack of sufficient skills in utilizing the technologies, it also impacts the effective application of digital devices in classroom teaching. Speaking of teachers’ skills, technological pedagogical content knowledge (TPACK) currently serves as an important model in this domain (Blackwell et al., 2016), which differentiates seven different types of knowledge about pedagogy, technology and content as well as their combinations (Voogt et al., 2013). Necessary training is required for effective digital technology integration together with administrative, and peer support.

Measurement invariance

Measurement invariance is a prerequisite for valid scale comparison research (Vandenberg & Lance, 2000). Invariance of the measure represents the scale functions the same across the groups. Specifically, it means the psychometric properties (e.g., factor loadings, item intercepts) relating the observed variables to the latent factor(s) should be similar across groups. Alignment method, proposed as an alternative to the traditional multi-group confirmatory factor analysis (MGCFA) in recent years, can conveniently estimate the means and intercepts of two or more groups. It overcomes the limitations of MGCFA such as labor-intensive and error-prone when the number of items and groups increase (Byrne & Vijver, 2017; Magraw-Mickelson et al., 2020; Muthen & Asparouhov, 2014) and allows for approximate rather than exact measurement invariance. Through automating invariance testing among groups with expected non-invariance, alignment method can estimate the factor loadings, item intercepts and factor means across groups in the presence of partial invariance, which greatly simplifies the testing procedures and has been considered as an optimal pattern of measurement invariance (Asparouhov & Muthén, 2014; Flake & Luong, 2021; Flake & McCoach, 2018; Muthen & Asparouhov, 2014).

There are usually two steps involved when conducting alignment analysis. The first step is called FREE alignment, through which a configural model is established and factor loadings and indicator intercepts are freely estimated for each group. The factor means are fixed at 0 and factor variance are fixed at 1. The second step is FIXED alignment optimization, in which the factor means and variances are freely estimated, and for every group factor mean and variance parameter, there are factor loading and intercept parameters that yield the same likelihood estimation as the configural model, therefore, model fit of the M0 is unaffected by alignment optimization and should be equal to the model fit of M1 (Asparouhov & Muthén, 2014). Based on the item-level significance tests for good performance, the cutoff point 25% of non-invariant parameters is recommended as a “rough rule of thumb” (Muthen & Asparouhov, 2014). FIXED alignment is required when there are only two groups compared and FREE alignment is recommended to work better when there are three and more groups involved. Moreover, researchers can assess the invariance effect size measure, which quantifies how much variability in the item parameter estimates can be explained by the groups’ factor means and variances. An R near 1 indicates complete invariance because the variability in item parameters is completely explained by group mean differences, whereas an R near 0 indicates that group mean differences explain none of the variability in the item parameter (Byrne & Vijver, 2017; Asparouhov & Muthén, 2014). Collectively, the information can serve as a guide for the follow-up decisions regarding item functioning and development.

Nonetheless, though the importance and necessity of measurement invariance before conducting the group mean comparisons has been recognized since the inception of large-scale national and cross-national assessments such as PISA, it is still rarely used partly due to difficulty in interpretation and implementation when more groups are involved with the traditional MGCFA, and partly due to unfamiliarity with alignment method. Meng et al. (2019) established the measurement invariance at the scalar level from PISA 2015 ICT student engagement scale with MGCFA but only limited to the comparison between just two countries German and China. Alignment method has still rarely been known and it is even less used in testing the measurement invariance of ICT-related scales. There was only one measurement invariance study on students’ mathematics, science and ICT familiarity scale across PISA 2015 participating countries with the alignment method (Odell et al., 2021).

The present study

School and teacher ICT readiness scale has been studied as a multidimensional construct, in which the two aspects are interrelated with each other (Bozkus, 2021). To our knowledge, measurement invariance of this newly introduced school and teacher ICT readiness scale in PISA 2018 has not been found to be addressed (OECD, 2020). Recognizing its importance, this study would like to initially assess the measurement invariance of the school and teacher ICT readiness scale before applying it into robust group comparisons. If measurement invariance of the ICT-related questionnaires holds at an acceptable level, then it will be valuable to compare its factor score means across the groups. Moreover, to have a better understanding of the scale with other relevant variables, a few school level indicators would also be use as validation measures.

Method

Data source and sample description

The data source for this study was the international large-scale assessment PISA 2018, a two-stage stratified assessment which mainly focused on 15-year-old students’ reading, science, and mathematics literacy. PISA 2018 is the latest seventh cycle, which focuses on reading in a digital context (OECD, 2019b). The questionnaires are designed by the PISA Governing Board, in which the content specialists and measurement experts work together to test its applicability in measuring students’ performance (OECD, 2019a). For the detailed sampling procedure, please refer to the specific PISA technical report (Kastberg et al., 2021).

The dataset for the current analysis was from the school-level questionnaires administered to school principals who participated in PISA 2018. Since the questionnaire was optional for the participating countries (OECD, 2019a), we have removed the countries that chose not to take the surveys (e.g., Cyprus and Moscow City (RUS)), and those that had very limited responses from the schools (< 100) (e.g., Malta, Brunei Darussalam, Montenegro). The final 57 countries (37 were OECD participating countries) for analysis were selected from America, Europe, Africa, the Middle East, Asia and Oceania, which was a good representation of different geographical regions and distinctive education systems. The country ID variable CNTRYID was used to define the 57 countries as latent classes for further measurement invariance analysis. The total schools from these countries were 18,041 and the average of the school numbers was 316, ranging from 142 in Iceland to 1089 in Spain. The country ID, Country name and number of schools that participated in the survey for each country was provided in Table 1.

Table 1

Country ID, Country Code and Number of schools for participating 57 countries

Country ID	Country	Country Code	n	Country ID	Country	Country Code	n	Country ID	Country	Country Code	n
8	Albania	ALB	327	360	Indonesia	IDN	397	124	Canada	CAN	821
56	Belgium	BEL	288	392	Japan	JPN	183	484	Mexico	MEX	286
112	Belarus	BLR	234	608	Philippines	PHL	187	152	Chile	CHL	254
276	Germany	DEU	223	643	Russian Federation	RUS	263	36	Australia	AUS	763
428	Latvia	LVA	308	704	Vietnam	VNM	151	170	Colombia	COL	247
528	Netherlands	NLD	156	764	Thailand	THA	290	380	Italy	ITA	542
826	United Kingdom	GBR	471	398	Kazakhstan	KAZ	616	752	Sweden	SWE	223
191	Croatia	HRV	183	410	Korea	KOR	188	348	Hungary	HUN	238
233	Estonia	EST	230	702	Singapore	SGP	166	40	Austria	AUT	291
300	Greece	GRC	242	804	Ukraine	UKR	250	250	France	FRA	252
372	Ireland	IRL	157	975	B-S-J-Z (regions of China)	QCI	361	208	Denmark	DNK	348
440	Lithuania	LTU	362	400	Jordan	JOR	313	554	New Zealand	NZL	192
616	Poland	POL	240	376	Israel	ISR	174	840	United States	USA	164
705	Slovenia	SVN	345	784	United Arab Emirates	ARE	755	32	Argentina	ARG	455
983	Tatarstan (RUS)	QRT	239	682	Saudi Arabia	SAU	234	76	Brazil	BRA	597
246	Finland	FIN	214	792	Turkey	TUR	186	203	Czech Republic	CZE	333
498	Moldova	MDA	236	703	Slovak Republic	SVK	376	352	Iceland	ISL	142
578	Norway	NOR	251	756	Switzerland	CHE	228	724	Spain	ESP	1089
620	Portugal	PRT	276	458	Malaysia	MYS	191	422	Lebanon	LBN	313

Variables

School and teacher ICT readiness scale is a self-reported 11-item four-point Likert-type scale. As evidenced by Bozkus (2021), this scale is a two-factor construct: one is school infrastructure of digital devices, which is measured with five items SC155Q01HA to SC155Q05HA (e.g., The number of digital devices connected to the Internet is sufficient) and the other is teachers’ capacity using digital devices, which is measured with six items SC155Q06HA to SC155Q11HA (e.g., Teachers have the necessary technical and pedagogical skills to integrate digital devices in instruction). School principals were asked to rate their agreement with each statement by selecting from four response options (“Strongly disagree”; “Disagree”; “Agree”; “Strongly agree”). The scale had appropriate reliability across the participating countries (Omega ω = .90). A total of 18,041 school principals answered to the list of questionnaires, in which 17,305 principals fully responded to all items. For the details regarding how the scale was administered, please refer to specific PISA manual (Kastberg et al., 2021).

To explore whether and how the factor may relate with some school and teacher level variables, a few relevant measures such as school location (Looker & Thiessen, 2003; Zhao & Frank, 2003), school type (Besley & Ghatak, 2001) and class size (Hislop & Ellis, 2004; Van de Vord & Pogue, 2012) were used for validity. For instance, school location variable SC001Q01TA, which included five ordinal categories from 1 = “A village, hamlet or rural area (fewer than 3 000 people)”, 2 = “A small town (3 000 to about 15 000 people)”, 3 = “A town (15 000 to about 100 000 people)”, 4 = “A city (100 000 to about 1 000 000 people)” and 5 = “A large city (with over 1 000 000 people)”; school type variable SC013Q01TA, which describes whether the school is managed by a public education authority, government agency or a non-government org; and class size variable CLSIZE, which includes nine categories from “15 students or fewer” to the largest size “More than 50 students”. The descriptive statistics of each item in the scale and the relevant validity measures were provided in Table 2.

Table 2

Descriptive statistics of the school and teacher ICT readiness scale and validity measures

Items	Item wording	N	M	SD
	School ICT readiness: School Infrastructure of digital devices
SC155Q01HA	The number of digital devices connected to the Internet is sufficient	17,256	2.79	0.91
SC155Q02HA	The school’s Internet bandwidth or speed in sufficient	17,245	2.73	0.93
SC155Q03HA	The number of digital devices for instruction is sufficient	17,221	2.65	0.90
SC155Q04HA	Digital devices at the school are sufficient powerful in terms of computing capacity	17,244	2.68	0.87
SC155Q05HA	The availability of adequate software is sufficient	17,245	2.73	0.83
	Teacher ICT readiness: Teachers’ capacity using digital devices
SC155Q06HA	Teachers have the necessary technical and pedagogical skills to integrate digital devices in instruction	17,260	2.78	0.67
SC155Q07HA	Teachers have sufficient time to prepare lessons integrating digital devices	17,258	2.70	0.76
SC155Q08HA	Effective professional resources for teachers to learning how to use digital devices are available	17,248	2.75	0.75
SC155Q09HA	An effective online learning support platform is available	17,228	2.56	0.85
SC155Q10HA	Teachers are provided with incentives to integrate digital devices in their teaching	17,238	2.57	0.88
SC155Q11HA	The school has sufficient qualified technical assistant staff	17,254	2.59	0.91
	School location	N	%
SC001Q01TA	A village, hamlet or rural area (fewer than 3000 people)	2665	0.16
	A small town (3000 to about 15,000 people)	3174	0.19
	A town (15,000 to about 100,000 people)	4480	0.27
	A city (100,000 to about 1,000,000 people)	3951	0.24
	A large city (with over 1,000,000 people)	2495	0.15
	School type	N	%
SC013Q01TA	A public school (Managed by a public education authority, government agency, or governing board)	12,990	0.81
	A private school (Managed by a non-government org; e.g. a church, trade union, business, or other private institution.)	3041	0.19
	Class size	N	%
CLSIZE	15 students or fewer	1696	0.11
	16–20 students	2308	0.15
	21–25 students	4141	0.27
	26–30 students	3448	0.22
	31–35 students	1290	0.08
	36–40 students	978	0.06
	41–45 students	540	0.03
	46–50 students	295	0.02
	More than 50 students	913	0.06

Analytical approach

First, before testing the measurement invariance of school and teacher ICT readiness scale, a conceptually consistent and cross-country measurement model needed to be tested. Therefore, a single-group confirmatory factor analysis (CFA) using robust weighted least squares mean and variance (WLSMV; Bowen & Masa, 2015) would be conducted to examine the factor structure of the ICT scale for the selected countries (regions). The model fit indices include comparative fit index (CFI; Bentler, 1990) and Tucker-Lewis index (TLI; Tucker & Lewis, 1973) with acceptable fit ≥ .90 and good fit ≥ .95, root mean square error of approximation (RMSEA; Steiger & Lind, 1980) with acceptable fit < .06 and standardized root mean residual (SRMR; Joreskog & Sorbom, 1981) with acceptable fit ≤ .08 (Hu & Bentler, 1999). Chi-square statistics is also reported but only for model comparisons not for accessing model fit since usually a statistically significant chi-square will be produced due to a large sample size (Chen, 2007). To account for the uneven probability of selection of schools within each country, school-level weighting variable W_FSTUWT_SCH_SUM was incorporated into the analysis.

After testing the conceptually consistent and cross-country measurement model, the next step is to test the measurement invariance with alignment method. Due to non-implementation of cross-loading in alignment method, five-item subscale of school infrastructure of digital devices and six-item subscale of teachers’ capacity of using digital devices were conducted separated with the goal of comparing mean scores in each subscale across the selected countries. Since FREE alignment is more recommended than FIXED alignment in more than two group comparison, we would first adopt the FREE alignment. If it produces any warning message, we would switch it the FIXED alignment. In the FREE alignment, all factor means are allowed to be estimated, but requires greater factor loading non-invariance (Muthen & Muthen, 2019). The reference group used was the country with the factor mean closest to 0, either positive or negative. In the FIXED alignment, the factor mean was constrained to zero for a particular group, similar to typical identification methods in CFA. Given that small number of valid missing responses on individual items existed for some responses, the full maximum likelihood estimation with robust standard errors (MLR; Yuan & Bentler, 2000) estimator was adopted. Both the CFA and alignment procedure were conducted in Mplus 8.4 (Muthen & Muthen, 1989–2019) and all other data cleaning and analysis was conducted using R version 4.0.3 (R Core Team, 2020).

Result

Evidence of factor structure

As mentioned in the analytical procedure, school and teacher ICT readiness scale was evidenced as a two-factor construct (Bozkus, 2021). We conducted a CFA across all the groups and found that the factor structure was supported with good model fit, χ2 (43, N = 17, 305) = 1275.996 (p < .001), CFI = .981 > .95, TLI = .976 > .95, RMSEA = .041[.039, .043] < .06, SRMR = .059 < .08. As stated previously, the chi-square test is possibly to be rejected with large sample size. Therefore, based on the model evaluation criteria, we concluded that there was adequate evidence of factor structure of the school and teacher ICT readiness scale to conduct measurement invariance test.

Alignment method analysis of school and teacher ICT readiness scale

Measurement invariance with FREE alignment approach was initially performed separately on the two subscales of the construct. No warning of untrustworthy standard errors was produced; therefore, we would adopt the FREE alignment approach for the optimization analysis. Table 3 demonstrated an invariance pattern with the two alignment fit indices: (a) fit function contribution value and (b) R value. First, high fit function contribution value is an indication of possible noninvariant item. For school readiness subscale, the intercept of SC155Q02HA showed higher absolute fit function than those of the other items indicating a higher noninvariance feature of SC155Q02HA than that for the other items. For teacher readiness subscale, SC155Q10HA showed much higher absolute fit function than those of the other items, indicating a higher noninvariance feature of SC155Q10HA than that for the other items. Second, the higher R is, the more likely the item is invariant. Contrarily, the lower R is, the more likely the item is non-invariant. In Table 3, SC155Q01HA and SC155Q03HA were close to 0, which was an indication of high noninvariance of these two items. Comparing the fit values between school ICT readiness subscale and teacher ICT readiness subscale, teacher ICT readiness showed higher values in fit function contribution and much lower R values (all <.06), which partly showed that the items in the subscale teacher ICT readiness subscale was more noninvariant than that for those in the subscale school ICT readiness subscale. Additionally, the overall fit function contribution values for the factor loadings were lower than those for the intercepts for both subscales, indicating a higher degree of invariance among the loadings than that for the intercepts.

Table 3

Alignment fit statistics

		Intercepts		Loadings
	Item	Fit function contribution	R2	Fit function contribution	R2
	FREE approach
School ICT readiness	SC155Q01HA	−655.617	.931	−671.087	.000
	SC155Q02HA	−921.670	.816	−652.502	.704
	SC155Q03HA	−632.303	.904	−658.898	.000
	SC155Q04HA	−746.881	.892	−570.352	.909
	SC155Q05HA	−790.490	.845	−679.292	.767
	Sum	−3746.961		−3232.131
Teacher ICT readiness	SC155Q06HA	−898.626	.521	−727.518	.226
	SC155Q07HA	−810.820	.685	−714.759	.115
	SC155Q08HA	−741.913	.799	−638.788	.348
	SC155Q09HA	−1125.620	.518	−652.018	.514
	SC155Q10HA	−1585.309	.336	−777.876	.519
	SC155Q11HA	−1157.385	.518	−658.074	.389
	Sum	−6319.673		−4169.033

The noninvariance of items and intercepts across countries was shown in Appendix Table 6, where a parenthesized group is an indication of noninvariance. For instance, Country ID 191, 440, 616 and 203 that have been parenthesized for SC155Q01HA intercept indicated that these four countries Croatia, Lithuania, Poland and Czech Republic had noninvariant factor intercepts for SC155Q01HA. Overall, the total number of parentheses in intercepts was much larger than the total number of parentheses in loadings, which suggests that the intercepts of the items were more noninvariant than the loadings of the items. Therefore, metric invariance might hold but not possible scalar invariance. In terms of school ICT readiness, given 5 items and 57 countries, 8 noninvariant parameters (of a total 285 (57*5) parameters) revealed evidence of factor loading noninvariance to be exceedingly low at 2.8%. Turning to the intercepts, though 53 noninvariant parameters were found, their overall percentage of 10.7% was still substantially lower than the recommended 25% cut-off point noted above. When it came to teacher ICT readiness, 103 noninvariant parameters of a total of 342 (57*6) parameters revealed that 30.1% of parameters were noninvariant, which was far higher than the 25% cut-off point. Therefore, overall, we felt confident in the trustworthiness of the latent mean estimates and comparison for the school ICT readiness subscale but not for teacher ICT readiness subscale across the 57 countries. Figure 1 also provided a comparison of proportions of invariant parameters between ICT readiness subscales and threshold visually.

Table 6

Noninvariance of items and intercepts across 57 Countries

Items
Intercepts
SC155Q01HA	8 56 112 276 428 528 826 (191) 233 300 372 (440) (616) 705 983 246 498
	578 620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703
	756 458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 (203) 352 724 422
SC155Q02HA	(8) (56) (112) (276) (428) (528) (826) (191) 233 (300) (372) (440) 616
	(705) (983) 246 498 578 620 (360) 392 608 (643) (704) (764) (398) (410) 702
	(804) (975) (400) 376 784 682 792 703 756 458 (124) 484 152 36 170 380 752
	348 40 250 (208) 554 840 32 76 (203) 352 724 422
SC155Q03HA	8 56 112 276 (428) 528 826 191 233 300 372 440 616 705 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q04HA	(8) 56 (112) 276 (428) 528 826 (191) 233 300 372 440 616 (705) (983) 246
	498 578 620 360 392 608 (643) 704 764 (398) 410 702 (804) 975 400 376 784 682
	792 (703) (756) 458 124 484 152 36 170 380 752 (348) 40 250 208 554 840 32
	76 (203) 352 724 422
SC155Q05HA	8 56 112 276 428 528 826 (191) 233 300 372 (440) 616 705 (983) 246 (498)
	578 620 360 392 608 643 704 764 398 410 702 (804) (975) 400 376 784 682 (792)
	703 (756) 458 124 484 (152) (36) 170 380 752 348 (40) 250 208 554 840 32 76
	203 352 724 422
SC155Q06HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 246 498 578 620
	360 392 608 643 704 764 398 (410) (702) 804 975 400 376 784 (682) 792 703 (756)
	458 (124) 484 152 (36) 170 380 (752) 348 40 (250) 208 554 840 32 76 203 352 724 422
SC155Q07HA	8 56 112 276 428 528 826 191 (233) 300 372 (440) 616 705 983 246 498 578
	620 360 392 608 643 704 764 398 410 (702) 804 975 400 376 784 682 792 703 756
	458 124 484 152 (36) 170 380 752 348 40 250 208 (554) 840 32 76 203 352 724 422
SC155Q08HA	8 56 (112) 276 428 528 826 191 233 300 372 440 616 705 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q09HA	(8) 56 (112) 276 428 528 826 191 233 300 372 440 616 705 (983) (246) 498
	(578) 620 360 392 (608) (643) 704 (764) 398 410 (702) 804 (975) 400 376 784
	682 792 (703) 756 (458) 124 484 152 (36) 170 380 (752) 348 40 (250) (208) (554)
	(840) 32 76 203 352 724 (422)
SC155Q10HA	(8) (56) (112) (276) (428) (528) 826 (191) (233) (300) (372) (440) (616)
	(705) (983) 246 (498) (578) (620) (360) (392) 608 (643) 704 (764) (398) 410
	702 (804) (975) (400) 376 (784) (682) (792) (703) (756) 458 124 484 152 36
	170 380 (752) (348) (40) (250) (208) 554 840 32 (76) (203) (352) 724 (422)
SC155Q11HA	(8) 56 112 276 428 528 826 191 233 (300) (372) 440 616 705 (983) 246 498
	(578) 620 360 392 608 (643) 704 764 (398) 410 (702) (804) (975) 400 376 (784)
	682 (792) (703) (756) (458) 124 484 (152) (36) 170 380 (752) (348) 40 250 (208)
	(554) 840 32 (76) 203 (352) 724 422
Loadings
SC155Q01HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 (246) 498 (578)
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q02HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 246 498 578 620
	360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756 458
	124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q03HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 (246) 498 (578)
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 (170) 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q04HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 246 498 578 620
	360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756 458
	124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q05HA	8 56 112 276 428 528 826 191 233 300 372 440 616 705 983 246 498 578 620
	(360) 392 (608) 643 704 764 398 410 702 804 975 400 376 784 (682) 792 703 756
	458 124 (484) 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q06HA	8 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578 620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756 458 124 (484) 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q07HA	8 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 (170) 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q08HA	8 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q09HA	8 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q10HA	(8) 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 (682) 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422
SC155Q11HA	8 56 112 276 428 528 826 191 233 300 372 440 616 (705) 983 246 498 578
	620 360 392 608 643 704 764 398 410 702 804 975 400 376 784 682 792 703 756
	458 124 484 152 36 170 380 752 348 40 250 208 554 840 32 76 203 352 724 422

8 = Albania; 56 = Belgium; 112 = Belarus; 276 = Germany; 428 = Latvia; 528 = Netherlands; 826 = United Kingdom; 191 = Croatia; 233 = Estonia; 300 = Greece; 372 = Ireland; 440 = Lithuania; 616 = Poland; 705 = Slovenia; 983 = Tatarstan (RUS); 246 = Finland; 498 = Moldova; 578 = Norway; 620 = Portugal; 360 = Indonesia; 392 = Japan; 608 = Philippines; 643 = Russian Federation; 704 = Vietnam; 764 = Thailand; 398 = Kazakhstan; 410 = Korea; 702 = Singapore; 804 = Ukraine; 975 = B-S-J-Z (regions of China); 400 = Jordan; 376 = Israel; 784 = United Arab Emirates; 682 = Saudi Arabia; 792 = Turkey; 703 = Slovak Republic; 756 = Switzerland; 458 = Malaysia; 124 = Canada; 484 = Mexico; 152 = Chile; 36 = Australia; 170 = Colombia; 380 = Italy; 752 = Sweden; 348 = Hungary; 40 = Austria; 250 = France; 208 = Denmark; 554 = New Zealand; 840 = United States; 32 = Argentina; 76 = Brazil; 203 = Czech Republic; 352 = Iceland; 724 = Spain; 422 = Lebanon

Fig. 1

Comparisons of proportion of invariant parameters between ICT readiness subscale and threshold

Factor mean values for school ICT readiness subscale across countries

The factor mean values for school ICT readiness subscale by country were shown in Table 4, which was arranged in an ordered list ranging from high to low. As showed in the table, New Zealand (country ID 554) was selected as the reference group with a factor mean closed to 0 (M = −.002). The rank order of factor means demonstrated that Singapore (702) had the highest factor mean in school infrastructure of digital devices, whereas Brazil (76) showed the lowest. The five countries with the highest school ICT readiness factor means were Singapore (702), Sweden (752), B-S-J-Z (regions of China) (975), United Arab Emirates (784) and United States (840). The lowest five countries in school ICT readiness factor score were Japan (392), Mexico (484), Colombia (170), Argentina (32) and Brazil (76).

Table 4

Mean comparison and ranking of the school ICT readiness scale across 57 countries in PISA 2018

School ICT readiness: school infrastructure of digital devices
Rank	Country	Factor Mean	Rank	Country	Factor Means	Rank	Country	Factor Means
1	702	0.438	20	428	−0.311	39	348	−0.720
2	752	0.248	21	360	−0.318	40	422	−0.757
3	975	0.165	22	56	−0.349	41	682	−0.766
4	784	0.104	23	250	−0.362	42	498	−0.792
5	840	0.101	24	826	−0.377	43	724	−0.842
6	705	0.087	25	643	−0.392	44	300	−0.857
7	208	0.057	26	246	−0.409	45	608	−0.917
8	36	0.045	27	704	−0.439	46	376	−0.925
9	554	−0.002	28	191	−0.442	47	804	−0.931
10	792	−0.014	29	112	−0.442	48	276	−0.946
11	40	−0.032	30	983	−0.442	49	400	−0.965
12	578	−0.057	31	703	−0.444	50	8	−0.986
13	756	−0.094	32	380	−0.448	51	620	−1.033
15	124	−0.094	33	764	−0.474	52	458	−1.045
15	528	−0.131	34	203	−0.486	53	392	−1.047
16	440	−0.158	35	372	−0.550	54	484	−1.243
17	352	−0.186	36	398	−0.568	55	170	−1.394
18	233	−0.224	37	152	−0.586	56	32	−1.502
19	410	−0.229	38	616	−0.633	57	76	−1.533

8 = Albania; 56 = Belgium; 112 = Belarus; 276 = Germany; 428 = Latvia; 528 = Netherlands; 826 = United Kingdom; 191 = Croatia; 233 = Estonia; 300 = Greece; 372 = Ireland; 440 = Lithuania; 616 = Poland; 705 = Slovenia; 983 = Tatarstan (RUS); 246 = Finland; 498 = Moldova; 578 = Norway; 620 = Portugal; 360 = Indonesia; 392 = Japan; 608 = Philippines; 643 = Russian Federation; 704 = Vietnam; 764 = Thailand; 398 = Kazakhstan; 410 = Korea; 702 = Singapore; 804 = Ukraine; 975 = B-S-J-Z (regions of China); 400 = Jordan; 376 = Israel; 784 = United Arab Emirates; 682 = Saudi Arabia; 792 = Turkey; 703 = Slovak Republic; 756 = Switzerland; 458 = Malaysia; 124 = Canada; 484 = Mexico; 152 = Chile; 36 = Australia; 170 = Colombia; 380 = Italy; 752 = Sweden; 348 = Hungary; 40 = Austria; 250 = France; 208 = Denmark; 554 = New Zealand; 840 = United States; 32 = Argentina; 76 = Brazil; 203 = Czech Republic; 352 = Iceland; 724 = Spain; 422 = Lebanon

School ICT readiness factor scores across the school location, type and class size

The result showed that schools from large cities had much higher school ICT readiness (M = 0.24) than those from village (M = −0.24, d = 0.47), those from small town (M = −0.06, d = 0.30), and those from town (M = −0.05, d = 0.29). Though school ICT readiness scores were positive for schools both from cities and large cities, there still existed difference (M = 0.24, Mcities = 0.04, d = 0.19). Private schools (M = 0.42) had a statistically significant higher school ICT readiness than those from public schools (M = −0.13, d = 0.57). Class sizes that were between 16 to 30 students had much higher school ICT readiness factor scores (M = 0.04 to 0.07) than the ones that were “15 students or fewer” or “31 to more than 50 students” (M = −0.27 to −0.08). Table 5 and Fig. 2 provided both statistics and visual picture of how these groups performed in school ICT readiness.

Table 5

School ICT readiness factor scores across the school location, type and class size

Variable	N	M	SD	SE	95% CI (M)
Location
Village	2665	−0.24	0.98	0.02	[−0.28, −0.20]
Small town	3174	−0.06	0.94	0.02	[−0.09, −0.02]
Town	4480	−0.05	0.97	0.02	[−0.08, −0.02]
City	3951	0.04	0.98	0.02	[0.01, 0.08]
Large city	2495	0.24	1.07	0.02	[0.20, 0.28]
School type
Public school	12,990	−0.13	0.98	0.01	[−0.15, −0.11]
Private school	3041	0.42	0.96	0.02	[0.38, 0.45]
Class size
15 students or fewer	1696	−0.08	0.96	0.02	[−0.13, −0.04]
16–20 students	2308	0.07	0.96	0.02	[0.03, 0.11]
21–25 students	4141	0.08	0.95	0.02	[0.05, 0.11]
26–30 students	3448	0.04	0.97	0.02	[0.01, 0.07]
31–35 students	1290	−0.10	1.03	0.03	[−0.15, −0.04]
36–40 students	978	−0.27	1.10	0.04	[−0.34, −0.20]
41–45 students	540	−0.18	1.12	0.05	[−0.27, −0.08]
46–50 students	295	−0.19	1.09	0.06	[−0.31, −0.06]
More than 50 students	913	−0.15	1.07	0.04	[−0.22, −0.08]

CI Confidence Interval

Fig. 2

School ICT readiness factor score and 95% confidence intervals across validation measures

Discussion

With the alignment method, this study examined the measurement invariance of school and teacher ICT readiness scale using PISA 2018 dataset. Using dataset from America, Europe, Africa, the Middle East, Asia and Oceania, it revealed that approximate level of measurement invariance existed for school ICT readiness subscale across all 57 countries, but overall non-invariance existed for teacher ICT readiness subscale. It provided a practical example of how to apply the newly developed alignment method into ICT-related scale with an aim of measurement invariance testing across multiple groups, which was a novel approach in ICT-related area that has not been widely known, implemented and accepted by researchers. It overcomes the tendinous numerous modification indices and error-prone procedures that occurs in traditional MGCFA and should be widely implemented in measurement invariance test for multiple groups.

Measurement invariance testing are recommended to be conducted before any cross-group ICT-related mean score comparisons for researchers and practitioners. Through alignment method, researchers and practitioners could gain a large amount of nuanced knowledge on the fit index and significance testing of the intercepts and loadings of a specific ICT construct, either the scale is on the student, teacher, or school level. Moreover, researchers and practitioners may broaden their understanding of scale’s cross-country differences by focusing on only noninvariant ICT items, and further identifying the sources of noninvariance. In our study, teacher ICT readiness subscale was identified to be noninvariant across the countries overall. It would be of high value to investigate the sources of noninvariance, especially when distinct cultural factors might affect the item responses. Understanding the existence of noninvariance and what contribute to the noninvariance in ICT readiness scale will assist researchers and practitioners with developing more culturally invariant items of scales in the further item development process.

The result from alignment method indicated that factor mean scores can be compared for the school ICT readiness subscale across the countries. By comparing the factor mean scores across the countries (regions) together, it was found that there was a big difference in the mean scores. For instance, Singapore showed the highest factor mean in school infrastructure of digital devices. This could be related with the Singapore’s long-term governmental support of technology use in schools. Early back to 2008, the ministry of education (MOE) in Singapore established five “Schools of the Future”, which served as a model in not only the curriculum design, teaching and learning but also the material resources (Lim, 2015). The independencies of the constituent elements among the self-organizing capacity, coevolution with other systems and fitness development and policymaking in ICT reform have brought Singapore’s stable leading position of school ICT readiness (Toh & So, 2011). Other countries such as Sweden, B-S-J-Z (regions of China), United Arab Emirates and United States all ranked among the top league of the assessment. Though these countries reside in different continents, they shared similar characteristics regarding digital resources as reported (Ikeda, 2020). Regardless of the socio-economic background of their students, a higher proportion of schools from these countries had an effective online learning support platform and computers with high-speed Internet connectivity and broad bandwidth; provided guidance on the use of digital devices and had specific programs to prepare students for disciplined Internet behavior. However, Japan, Mexico, Colombia, Argentina and Brazil were in the lowest rank. It is reported that less than 30% of students in these schools in these countries had similar platform as those in top ranked countries and it is mainly due to the large socio-economic disparity these counties have (Ikeda, 2020). The variance of the factor scores directly reflected that investment and support of school infrastructure of digital devices were treated quite differently with similar or distinct cultures and socio-economic disparities across the globe. Though development in technology infrastructure in schools is a worldwide investment, education equity is still a central issue of the education system across the world.

Notably, technology integration is a complex process, which requires cooperation from various aspects, such as teachers and school administrators work together with the classroom environment, curriculum, and everyday routines (Yang et al., 2021). It is a pity that teacher ICT readiness subscale could not be directly compared across the countries in our study. However, for teacher ICT readiness subscale, there was an alternative way to examine the cross-country differences in further studies. If we group the countries based on their similar characteristics or cultural background, then measurement invariant might hold. It also reflected the complexity involved in the attempt to attain cross-group invariance of both the factor loadings and item intercepts related to psychological assessment scales when multiple groups are involved and with a cross-cultural nature.

School ICT readiness factor scores were also found to be closely related with a few school background factors, such as school location, school type and class size. School locations have an impact on pattern of use and attitudes to technology (Looker & Thiessen, 2003). Schools from city and large cities usually have relatively sufficient financial support to develop its school digital infrastructure, therefore schools from these areas have much higher ICT readiness score. However, schools from rural areas such as village, small town or town are often lack of support from government or funding department, and their school ICT readiness scores accordingly are lower than those in the other areas. In terms of school type, there has been hot discussions of the division of responsibility between the private and public schools (Besley & Ghatak, 2001). A big contrast of the ICT readiness scores was found between public and private schools in our study. Another interesting finding was the differences of school ICT readiness scores across various class size. Hislop and Ellis (2004) reported that class size for on the online versions was on average 19.3 and 26 for the in-person class. In our analysis, class sizes ranged from 16 to 30 students had much higher factor means in school ICT readiness than those that were 15 students or fewer. Class size between 16 to 30 students can achieve optimal effect even in the school ICT readiness score, which was consistent with the previous research. During the COVID-19 pandemic period, most schools switched to the online teaching format, which could bring more pressure to schools and teachers in village, small town or town areas. It was also worth investigating the differences of school ICT readiness between the in-person class size and online class size.

Limitation

Several limitations must be acknowledged in the current study. Practically, PISA 2018 was conducted far before the eruption of COVID-19 across the world. The scale we estimated might not be able to accurately reflect the current global situation in the school and teacher ICT readiness. It is worth examining the same issue with the PISA 2021 dataset, which will better reflect the reality of global ICT readiness when facing significant change of teaching format. Methodologically, cross loadings still cannot be accommodated with the alignment method (Asparouhov & Muthén, 2014). Therefore, the invariance analysis was conducted separately for each subscale in our study. Lastly, although we are not able to explore possible mechanisms for non-invariance, future research should consider how external variables might explain non-invariance across cultures by using the alignment and/or alignment-within-CFA frameworks (Marsh et al., 2018).

Conclusion

With the novel alignment optimization approach in measurement invariance, this study was expected to provide researchers and other stakeholders with more nuanced knowledge of the school and teachers ICT readiness. The invariance of school ICT readiness subscale across the globe allowed the researchers to have a better understanding of how school ICT readiness performs in the participating countries. The non-invariance of teacher ICT readiness subscale encouraged the researchers to explore the substantive and methodological sources that cause the root source of non-invariance.