PISA data clusters reveal student and school inequality that affects results.

The data from the PISA survey show that student performance correlates with socio-economic background, that private schools have higher results and more privileged students, and that this varies between countries. We explore this further and analyze the PISA data using methods from network theory and find clusters of countries whose students have similar performance and socio-economic background. Interestingly, we find a cluster of countries, including China, Spain and Portugal, characterized by less privileged students performing well. When considering private schools only, some countries, such as Portugal and Brazil, are in a cluster with mostly wealthy countries characterized by privileged students. Swedish grades are compared to PISA results, and we see that the higher grades in private schools are in line with the PISA results, suggesting that there is no grade inflation in this case, but differences in socio-economic background suggest that this is due to school segregation.

Introduction

PISA (Programme for International Student Assessment) [1], organized by the OECD, is a 3-year recurring knowledge test used to assess the performance in science, mathematics and reading of 15-year-olds. PISA draws considerable attention worldwide and affects educational policies [2-6]. The 2015 PISA survey, which we focus on here, included 519334 students in 73 countries. The corresponding data set with student responses to the PISA questionnaire also includes a set of items to assess the socio-economic background of the students [7, 8]. This allows for studying the connection between socio-economic background and student performance.

There has been some criticism against PISA, for example regarding the effect of cultural biases in student responses, the lack of adaptations to local curricula and language, and the deceiving simplicity in ranking countries [9, 10], and it has been argued that this criticism is largely ignored [10]. But it has also been shown that the PISA results are indicative of the educational outcome of students [11].

While some studies focus on for example gender differences [12, 13], we here focus on the connection between the socio-economic background of the students and their performance in PISA, since there is growing evidence that this is an important determinant for student performance [14-17], while at the same time the educational system in most cases is designed to compensate for the variations in the students’ socio-economic background [18-20]. We therefore explore any possible connections between performance and socio-economic status.

Some alternative analyses to reveal patterns in the PISA data have been done, focusing on the raw data in PISA—the student responses to items (questions)—to study for example longitudinal trends based on item characteristics [21] or country-specific differences using cluster analysis [12, 22]. We believe that the data can be further explored and any possible hidden patterns be discovered using modern methods, mainly from the field of network theory that has developed tremendously over the last decades [23].

In this work we therefore apply tools from this field, previously applied mostly in theoretical ecology on species distribution data [24-28], to find patterns in the PISA data that can tell us something about the structure in performance and socio-economic background of the students and further nuance the PISA results.

We also study the performance of students at private and public schools respectively, since this aspect of school organization also has been shown to affect student performance [29, 30], and grade inflation has been reported at private schools [31-33], in this way reinforcing social inequality and undermining the compensatory effect of the educational system. We compare PISA data to grading in Swedish schools, where we have access to grading data in these two sectors respectively, to investigate possible grading inflation in Sweden.

Results

PISA consists of a set of items, or questions, that the student should answer in order to assess the student’s knowledge in different areas, such as science and mathematics. These items are ranked according to their overall difficulty d which is derived from student responses worldwide [8]. Here we normalize d such that the normalized difficulty of item i is , where j ∈ [0, N] if N is the number of items. This means that , which is necessary for further calculations if we want to sum difficulty-weighted responses. The student response r is here coded as either 1 (pass) or 0 (fail). We can then assign a number P to a student i, giving the student’s number of points in science and mathematics, by calculating

This measure then gives a number in a transparent way for each student’s achievement in science and mathematics, weighted by the item difficulty.

PISA also includes an index called ESCS (Economic, Social and Cultural Status) derived from the student responses about home possessions and the parents’ education and occupation [8]. This index quantifies the students’ economic, social and cultural background. We are here interested in the relation between the student’s socio-economic background and performance in mathematics and science. It is therefore interesting to study how the students are distributed over P and ESCS, and how this varies between countries and between public and private schools.

Fig 1(a) shows the distribution of all participating students (without student weights and resampling) over points and ESCS. We see that the distribution is not uniform, but skewed so that students with higher points tend to have high ESCS. Highly performing students thus tend to be more socio-economically privileged. Using the student weights [8] to resample the data (details in Methods) we can calculate the correlation (Spearman’s ρ) between Points and ESCS to be ρ = 0.2836±0.0002, 95% CI. If we consider public and private schools separately this correlation is ρ = 0.2777±0.0002 for public schools and ρ = 0.2246±0.0005 for private schools. The tendency that highly performing students are also socio-economically privileged is thus stronger in public schools than in private schools, on a world-wide level. Another aspect of the differences between public and private schools is shown in Fig 1(b), where we see that the average number of points in public schools is lower than in private schools (4.366±0.001, 95% CI vs. 5.290±0.002), and that the mean ESCS in public schools is lower than in private schools (−0.3899±0.0003 vs. 0.0516±0.0005).

Fig 1

The distribution (number of students, as isolines) of student results (Points) over economic, social and cultural status (ESCS) is skewed so that highly performing students tend to be more privileged (a). Fig 1(b) shows that private schools tend to have somewhat better results than public schools (top), and that students at private schools are more privileged (bottom). The 57464 students with zero points (visible in (a)) have been omitted in the top figure to better display the distribution of the other students.

As a further step in understanding the differences between countries Fig 2 shows the correlation (Spearman’s ρ) between points P and ESCS for the participating countries. This ranges from ρ = 0.313±0.002, 95% CI, for Peru, to ρ = 0.052±0.002 for Algeria. This shows that there are variations between countries, and that the tendency that highly performing students are also socio-economically privileged is not equally strong in all countries.

Fig 2

The correlation (Spearman’s ρ) between student points in mathematics and science, and student ESCS for the participating countries (ISO 3166-1 alpha-3 codes).

The error bars indicate a 95% confidence interval obtained by bootstrapping the data using the student weights (details in Methods). It is evident that privileged students (high ESCS) tend to perform better in PISA in many countries, but this tendency is weaker in countries to the right in the figure.

This type of data where different groups (here countries) are distributed differently in a possibly multidimensional space can be difficult to interpret based on simple correlations, and the large-scale patterns in the data can be difficult to find. A growing body of research is therefore developing, mainly in the field of theoretical ecology [24-28], to employ methods from network theory to find these patterns by, for example, representing species distributions in geographic space [24-26] or in a space of climatic variables [27] as networks in order to study their structure. We here apply this framework to the PISA data to identify clusters of countries that are connected together due to their students performing similarly in PISA while at the same time having a similar socio-economic background, meaning that their students are distributed similarly in the ESCS-Points space.

To start with we discretize the space of socio-economic background and performance in PISA, i.e. the ESCS-P space, by selecting the number of quantiles to divide each dimension in. This is done by computing the Jensen-Shannon divergence of the distributions of the participating countries in this space, in order to see when it is no longer meaningful to continue to divide the space. Details on this are provided in Methods, where we show that this space should be divided in nine quantiles in each dimension. Each bin in this distretized space is a node in a bipartite network where these space nodes are one category of nodes and the countries are the other category. A country node is connected to these space nodes based on how the country’s students are distributed in the ESCS-P space. If the fraction of students in a space node (ESCS-P bin) for a country exceeds the fraction of students in this bin averaged over all countries, the link weight between this country and the corresponding space node is the difference between these fractions. Details on this are given in Methods, and Fig 3 shows an overview of this whole procedure.

Fig 3

An illustration of the method used here to find large scale patterns in the PISA data using network theory.

The PISA data (A) are mapped onto a two dimensional space divided in quantiles (B) where each quantile corresponds to a node in a bipartite network with links weights calculated from the data (C). A community detection algorithm (Infomap) is used to find highly interconnected nodes forming clusters (modules) in this network (D). These clusters contain countries with similar patterns in the PISA data.

Once the network is constructed we employ the Infomap algorithm [34] to find sets of highly interconnected nodes, called modules in network science but commonly referred to as network clusters. Infomap is widely used and has been shown to be highly reliable for clustering networks [26, 35, 36].

Fig 4 shows the resulting clusters of countries and ESCS-Points nodes. The blue cluster is characterized by privileged students on the whole spectrum of performance, and includes many of the world’s wealthy countries. The orange cluster has the same range of privileged students but with zero or few points. The green and purple clusters are characterized by less privileged students that perform poorly, with the green having somewhat better performance than the purple. Interestingly, the red cluster is characterized by less privileged students that perform well in PISA. This red cluster includes both Spain and Portugal, China, Latvia and Poland, but the significance analysis (details in Methods) however shows that Poland is less significantly clustered in this red cluster. The significance analysis also shows that both the blue and the red together with the orange cluster have a high support (0.97, 0.72 and 0.89) in the bootstrapped networks (inset in Fig 4).

Fig 4

The clusters resulting from the network representation of the PISA data.

The blue cluster is characterized by privileged students over the whole scale of performance. The green and purple clusters have less privileged students performing poorly. The red cluster has under-privileged students that perform well in PISA. The bootstrap support for the clusters is indicated by the numbers in the inset, and the node significance by the transparency of the colors.

These clusters of countries can contribute to our understanding of inter-country differences and can nuance the PISA overall ranking [37]. We can see, for example, if the top ranking countries tend to be clustered together and if they share some common feature. We see however that this is not the case. Some top ranking countries like China, including Hong Kong, are in the red cluster, while most of the others are in the blue cluster. In a top ranking country like China the underprivileged students perform well, which the OECD point out in their report [37], stating that they have greater equity than other countries. They also mention Canada in this context but the results shown here indicate that Canada is better characterized by privileged students, together with many other top ranking countries. The OECD does not mention the interesting features of the countries on the Iberian peninsula, Latvia and Poland that they share with China, in that they have high-performing under-privileged students. Countries like Canada and Estonia are ranked high, while many other countries in the same blue cluster are not, such as Sweden and The United States.

If we consider data from all schools, public schools and private schools separately, we obtain three different networks that can be clustered and compared using an alluvial diagram, as show in Fig 5. We see that the clustering of students at private schools is fragmented, resulting in many clusters with low bootstrap support. This is due to the scarce data with relatively few students in private schools worldwide. We can however see some interesting features that have support in the bootstrapped data, such as Portugal and Brazil both being clustered in the blue cluster, with privileged students over the whole scale of performance, when considering students from private schools only. Also, Latvia is clustered in the blue cluster when considering only public schools. The other countries in the red cluster remain in this cluster both for public and private schools.

Fig 5

An alluvial diagram showing how the clustering of countries changes when considering all schools, private schools or public schools, and the bootstrap support of the clusters.

The clustering of countries using data only from private schools is fragmented and some of the clusters have low bootstrap support. We can however see, for example, that Portugal and Brazil both change cluster to the blue cluster, which is characterized by privileged students over the whole scale of performance, when considering private schools only.

To further study the differences between public and private schools we calculate the median value of points P in each interval of ESCS ranging from -8 to 4 in steps of one. We use bootstrapped data sets to obtain a confidence interval for this median value. Fig 6 shows these data for all countries (a), China (b), Portugal (c) and Brazil (d). In all cases the private schools do not include the tail of socio-economically weak students, but this is most apparent in Brazil. Fig 6 shows that both Portugal and Brazil have significantly higher results in private schools, which was indicated by them switching clusters in the cluster analysis. China has only small differences between public and private schools, but a clear tendency for high-performing students to be also socio-economically privileged. This aspect of China’s results goes against OECD’s statement of greater equity in China [37]. Portugal and China are clustered together in the red cluster, with high-performing less privileged students, and this is also seen here where students with low ESCS have relatively high points, but for Portugal this is due to the higher performance of these students in public schools. The data for Spain show that students at public and private schools perform similarly.

Fig 6

The bootstrapped median values and confidence intervals for the number of points in different intervals of ESCS.

The network analysis revealed significant differences between public and private schools in Brazil and Portugal, which here shows as higher results (Points) for private schools. Both Portugal and China have relatively high results for less privileged students (low ESCS), which is why they are clustered together when considering all schools.

The world-wide data (Fig 6(a)) shows interesting differences between public and private schools. For example, less privileged students achieve better results in public schools than they do in private schools. At the same time more privileged students score higher in private schools than they do in public schools.

School inequality and grade inflation in Sweden

Sweden has seen a highly debated rapid change in educational policy towards market competition in the school system [38-41]. An increase of the inter-school variations in grading of students has been observed [42], and others report that the right to freely choose your school has led to an increase in school segregation [43]. Students at private schools in Sweden are clearly more privileged (ESCS = 0.575±0.005, 95% CI) than students at public schools (ESCS = 0.283±0.002), and they also perform better in PISA (Points = 5.79±0.02 vs. Points = 5.26±0.01). We can compare the points in PISA to the corresponding grades in Swedish schools [44], as shown in Table 1 where we include all points, and points in mathematics and science separately. We see that the grading is in line with the PISA results, and these data thus suggest that there is no grade inflation at private schools, contrary to what has been suggested previously [31-33]. In fact, based on the PISA results even higher grades in mathematics can be justified at private schools.

Table 1

PISA points and the average grades in mathematics, physics and chemistry of students in private and public schools in Sweden.

Students in private schools have better results in PISA and also higher grades than students in public schools. These data thus suggest that there is no grade inflation at private schools in Sweden.

	Public Schools	Private Shools	Difference (%)
Points	5.26±0.01	5.79±0.02	10.1
Points, Mathematics	1.450±0.006	1.62±0.01	11.7
Points, Science	3.814±0.006	4.17±0.01	9.3
Grade, Mathematics	12.0	13.3	10.8
Grade, Physics	12.7	14.0	10.2
Grade, Chemistry	12.8	14.0	9.4

However, by considering how the students are distributed over ESCS in public and private schools respectively, as shown in Fig 7, we see that the public schools have a large part of their students in the lower ESCS quantile, while the opposite is true for private schools. This can explain why private schools have the higher PISA scores (and also grades) that we have seen, since the less privileged students do not attend private schools to the same extent that they attend public schools. This suggests that there is a school segregation in Sweden, where the less privileged students predominantly attend public schools, and that this is responsible for the differences in PISA results and also grades between public and private schools.

Fig 7

Histograms of PISA points divided in three quantiles (low-high in (a)-(c)) of public school ESCS in Sweden show that public schools have a large part of their students in the lowest quantile (a), while the opposite is true for private schools (c). This can explain part of the higher PISA results and grades in private schools in Sweden.

Discussion

We have seen that students from a privileged socio-economic background tend to perform better in PISA and that this tendency varies between countries and private and public schools. These tendencies are known from previous studies, but the network based approach that we propose here sheds new light on this and shows, for example, that some countries go against this general trend since their less privileged students have relatively high results. This group includes China, Spain and Portugal, and further analysis showed that privileged students tend to perform better than less privileged also in these countries, but that less privileged students achieve higher results compared to the world-wide average. In Portugal however, students in private schools perform significantly better than students in public schools. This is not the case in China and Spain. Some countries, such as Brazil, show large differences between public and private schools.

This type of results can nuance the PISA overall ranking and inform policy makers, for example about which student groups to target in order to invest resources in the most efficient way, if the goal is to improve the PISA results in a country. For example, Portugal’s public schools perform well but their private schools are significantly better, which should raise questions about the causes of this. Also, Poland and Latvia are clustered weakly in the cluster characterized by socio-economically weak students performing well, and one could ask why this is the case and what can be done if one should desire to be more connected to this cluster.

This way of analyzing difficulty-weighted item responses opens up new ways of doing research with the PISA data. A possible next step in this line of research is to study time resolved (longitudinal) data gathered from several rounds of PISA to see how trends develop over time. It would be interesting to see if there is a detectable signal over time of increasing school and student differences, i.e. segregation, and how this affects the overall results country-wise and world-wide.

A wider contribution of this paper is to advance the understanding of network theory as a tool for data analysis, by contributing with another application of these methods.

The comparison between PISA results in public and private schools in Sweden and their respective grades, showed that the higher grades in private schools are justified given the PISA results. This is evidence that there is no grade inflation in private schools in Sweden since the PISA tests are marked centrally in a standardized manner, thus excluding the role of the school and the teacher which can otherwise affect the results in for example nation-wide but locally marked tests. We however also saw that these differences can be explained by socio-economically weak students predominantly attending public schools.

Methods

Data

The PISA 2015 data have been obtained from the OECD website [45] in SPSS format and read using Pandas to do further analysis. Data on item difficulties have been obtained from the PISA 2015 Technical Report [8].

Data on whether a school is private or public is primarily taken from the item “SC013Q01TA”, which reads “Is your school a public or a private school?”. The exception is Sweden where this information is coded in the field “STRATUM”, and values “SWE0001”, “SWE0002”, “SWE0003”, “SWE0004” correspond to public schools. These data are missing for approximately 1/10 of the students worldwide.

Significance analysis

To estimate summary statistics we perform a bootstrap analysis, as prescribed by the OECD, by resampling the data with replacement to obtain a set of 100 bootstrap samples. Using this set we can calculate for example mean values and confidence intervals in a standard manner. The resampling of students is done with the recommended PISA weights (W_FSTUWT) that compensate for possible biases in the selection process [7]. We have however done extensive testing with and without these weights and the results are in general robust with respect to these weights being used or not.

The significance analysis of modules and module node assignments follows the procedure outlined by Calatayud et al [26]. We measure the distance between two modules C and C as the Jaccard distance J(C, C), given by which is one minus the fraction of the number of common nodes and total number of nodes. The significance of a module i in a reference partition R is the fraction of partitions that have a module with a smaller Jaccard distance to i than a threshold τ, that is where we sum over all p − 1 partitions P that are not the reference partition R, and Θ is the Heaviside step function. We have here used τ = 0.8. Due to the bootstrapping procedure with the PISA weights there is no reference partition based on raw or unsampled data. Instead we use as reference partition the bootstrapped (with PISA weights) partition with the shortest average Jaccard distance to all other bootstrapped partitions.

The node significance of node v in reference partition R is calculated as the fraction of partitions in which v appears in the module that is most similar to v’s module in the reference partition. Using the Kronecker delta function δ this can be written where is the module index of node v in partition P, and is the module index of the module in partition P that is most similar to the module of v in partition R.

Dividing the space of socio-economic background and PISA performance

It is important to choose the best possible division of the ESCS-P space to define the nodes in this space. If the division is to coarse grained we loose information, with the extreme case being no division at all and all countries would then be connected to the same node in this space, giving no structure whatsoever. In the other extreme we divide the space in to many quantiles so that only one or very few countries are connected to a node. This also leads to a loss of information.

The best division is somewhere in between these extremes and to find it we calculate the Jensen-Shannon divergence between the countries’ distributions in the ESCS-P space, where is the Jensen-Shannon divergence. Here H is the entropy, meaning that where X is some variable with possible values x and p(x) is the relative frequency of x, or the probability of x if X is a stochastic variable. The unit of entropy is normally called bits and it measures the information content in a distribution of X, such that a completely uniform distribution has maximum entropy (since it is completely unpredictable if X is a stochastic variable).

The Jensen-Shannon divergence is thus the difference between the entropy of the average distribution of countries, and the average of the entropy of each country’s distribution. If this is zero all countries are equally distributed in the ESCS-P space, as is the case if we have only one ESCS-P node. When the number of quantiles is increased JSD also increases if there is a difference between how the countries are distributed, but only up to a some point when we gain no or little more information by dividing the space further.

We calculate JSD when varying the number of quantiles that we divide the ESCS-P space in and calculate the JSD for each division. Fig 8 shows JSD for quantiles in the range 2-16, and we see that there is a steep increase in JSD up until some point between 6 and 8 quantiles. To further illustrate this we show in Fig 8(b) the increment of JSD as we increase the number of quantiles. We see here that after approximately nine quantiles the increment flattens out, and we therefore choose to use nine quantiles in our analysis.

Fig 8

The Jensen-Shannon divergence (a) and its increments (b) when varying the number of quantiles in the ESCS-P space.

After approximately six quantiles we gain little information by dividing further.

Building the network of countries and space nodes

Once the space of ESCS and points is divided in quantiles we construct a bipartite network by linking countries to the ESCS-points nodes (space nodes) where the country has a larger fraction of its students than the average of all countries.

The bipartite network is , where is the set of N country nodes and is the set of N space nodes. Here N = q2 if q denotes the number of quantiles in each dimension. The set of potential links thus connects country nodes to space nodes. A link e is given by where S denotes the number of students in country c′ that are in node ep′, the total number of students in country c′, S the total number of students, in all countries, in node ep′, and S the total number of students in all countries. This means that there is a link between a country and a space node if the fraction of students of that country in that node is larger than the fraction of students of all countries in that node. A country is thus connected to a space node if the country has a higher proportion of the students in a part of the ESCS-P space than the average, thus capturing the notable features of that country.

13 Jan 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Jacob Freeman

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex.

3.  Thank you for stating the following in the Acknowledgments Section of your manuscript:

(N. Eliasson and M. Oskarsson are acknowledged for advice on data retrieval. The computations were enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at HPC2N partially funded by the Swedish Research Council through grant agreement no. 2018-05973.)

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

(The author received no specific funding for this work.)

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. We note that Figure 4 in your submission contain map images which may be copyrighted. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For these reasons, we cannot publish previously copyrighted maps or satellite images created using proprietary data, such as Google software (Google Maps, Street View, and Earth). For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure 4 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

The following resources for replacing copyrighted map figures may be helpful:

USGS National Map Viewer (public domain): http://viewer.nationalmap.gov/viewer/

The Gateway to Astronaut Photography of Earth (public domain): http://eol.jsc.nasa.gov/sseop/clickmap/

Maps at the CIA (public domain): https://www.cia.gov/library/publications/the-world-factbook/index.html and https://www.cia.gov/library/publications/cia-maps-publications/index.html

NASA Earth Observatory (public domain): http://earthobservatory.nasa.gov/

Landsat: http://landsat.visibleearth.nasa.gov/

USGS EROS (Earth Resources Observatory and Science (EROS) Center) (public domain): http://eros.usgs.gov/#

Natural Earth (public domain): http://www.naturalearthdata.com/

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I have the following comments on the paper "The PISA data reveal student and school inequality that affect results."

1. The Methods section should be presented before the Results and Discussion sections.

2. The authors calculated students' "normalized points" Pi by dividing points scored per question by the difficulty of each question. The nominator is the difference between the max difficulty and the min difficulty of each student (page 8). First, however, the max and min difficulties of questions answered vary across students, and the gap between the max and min difficulty also varies across students. Therefore, using the nominator as the gap of max and min difficulty of individual students does not normalize PISA scores across the student distribution. Second, PISA test scores have been weighted to take into account the difficulty of different questions and allow meaningful comparisons across students/cohorts/countries. PISA test score is a better measure of student performance than Pi calculated by the authors. Also, the construction of Pi should be placed in the Methods section.

3. How do the authors define "privileged students" using the ESCS score? What is the cut-off point, and why is that point chosen as a cut-off?

4. Why did the authors use bootstraps (page 7) to calculate summary statistics? The PISA sample is supposed to be representative at the country level. They can use any statistical software to calculate the presented results. From what I read here, the results are the mean of test scores and the correlation between test scores and ESCS for different groups of students.

5. "Significance analysis" (page 7): It's not clear what the purpose of the significance analysis presented in this section is. If I understand correctly, the authors use significance analysis to test for the difference between mean scores across students from different economic backgrounds and/or whether the correlation coefficient between test scores and ESCS is statistically significant. If so, t-test and regressions are methods commonly used in social science for these purposes. If the authors decide to use modules and module node assignments, please cite a peer-reviewed published paper that has used this method to investigate similar issues in education or, in addition, present t-tests and regressions to show that the two approaches are similar.

6. "Building the network of countries and space nodes" (page 9): What is the purpose of using this method? What do the authors mean by "where the country has a larger fraction of its students than the average of all countries"? Do they mean a larger fraction of "privileged students" in the whole student population? If so, the statistic they need to calculate here is the deviation from the mean of the fraction of "privileged students", and that can be calculated using any statistical software.

7. Page 9: What is "Figure ??"

8. The authors need to provide the take-away point from each figure presented in the results section.

9. Figure 1 looks like the authors map PISA's deciles with ESCS scores, but I'm unsure.

10. The authors also conclude that students from privileged backgrounds performing better in Sweden results from school segregation. However, at least in the US, private schools are likely to select students with involved parents, and families with involved parents are likely to have higher test results regardless of ESCS status. Also, to attract students, private schools may provide more educational resources to students and have better teachers than public schools. Can you control for these different effects?

11. Cultures, such as China, where acceptance into top universities is dependent on doing well on exams, means that Chinese students in Public schools are also likely to attend after-school private schools. Is there a way that you can control for students who attend both public schools and private schools?

12. The authors' main conclusions are 1) students with a higher socio-economic status (SES) perform better than students with a lower SES, and 2) the extent of the performance gap varies across countries. The authors should reference more of the literature on higher SES students performing better than low SES students, which is well established in the social science literature. They should also reference the literature on human capital theories that provide explanations for the difference in academic achievement between students from different economic backgrounds.

Reviewer #2: The author analysed PISA data to explore the relationship between students' performance and economic status, searching for differences between public and private schools across countries. He applied network-based tools to construct and cluster a bipartite network formed by countries and portions of a bidimensional space representing students’ performances and economic status. Moreover, the author conducted complementary analysis to further the understanding of network clustering. He found a cluster of countries where less privileged students perform well, being this pattern an exception to the rule.

I think the manuscript is well written, well conducted and state-of-the-art in methodology. The results are robust and the interpretations seem appropriate to the methods and results. My complete inexperience in the topic prevent me to evaluate the relevance of the results, still from a complete layman’s point of view I found the results very interesting. I did not find any point that could be further improved but this two minor suggestions:

I had difficulty following Figure 1a. I think the lines represent isolines, but I am not sure. I would color or number the lines and provide a legend to facilitate understanding.

Please check “Figure ??” in lines 286 and 288

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

2 Mar 2022

Rebuttal letter

Response to the Academic Editor

Thank you very much for taking the time to consider my manuscript. It combines seemingly disparate fields and I appreciate the effort to seek out reviewers from different areas of research, and I support your intention to seek out also a third reviewer.

Below follow my comments and corresponding changes. I must mention though that it is my impression that most of the remarks made by reviewer 1 are a result of misunderstandings and/or lack of insight into statistical methods beyond those that are standard in the social sciences. I have however tried to improve the manuscript as requested.

Kind regards,

Magnus Neuman

ACADEMIC EDITOR: […] To meet Plos criteria for publication, the author must, at minimum, address the concerns of reviewer one related to the measurement of PISA scores (Reviewer 1’s point 2) and better ground the study in the relevant literature and theory. The statistical approach is novel in this field of study; the author just needs to make clear what new knowledge this approach affords over standard statistical approaches used in previous studies or how this study replicates, in a conceptual, previous results.

This relates to point 12 of reviewer 1 and is partly addressed in my response to that. I have added another two references [16, 17] that contain results on the relation between educational outcome and socio-economic status. However, previous work offer few nuances beyond stating that academic results improve with increasing socio-economic status (as stated in the introduction), and I believe that the present work can fill that gap.

The main results of the present study (as stated in the abstract, results and discussion) are the identification of a cluster of countries characterized by high performance and less privileged students, together with variations between public and private schools, and the novel approach to studying these data. These results are novel and should be of general interest.

The second sentence of the discussion has been changed in order to further emphasize the contrast between previous work and the present work.

Furthermore, I suggest changing the title to ”PISA data clusters reveal student and school inequality that affects results”, in this way narrowing down the scope of the title and pointing more explicitly in the direction of the novel results.

Response to Reviewers

Reviewer #1

Thank you very much for your valuable comments. I have placed my clarifications and declared the corresponding changes to the manuscript after each of your points below.

I have the following comments on the paper "The PISA data reveal student and school inequality that affect results."

1. The Methods section should be presented before the Results and Discussion sections.

It is commonplace in many journals today, including PLOS ONE, to place the methods section last. This improves readability.

2. The authors calculated students' "normalized points" Pi by dividing points scored per question by the difficulty of each question. The nominator is the difference between the max difficulty and the min difficulty of each student (page 8). First, however, the max and min difficulties of questions answered vary across students, and the gap between the max and min difficulty also varies across students. Therefore, using the nominator as the gap of max and min difficulty of individual students does not normalize PISA scores across the student distribution. Second, PISA test scores have been weighted to take into account the difficulty of different questions and allow meaningful comparisons across students/cohorts/countries. PISA test score is a better measure of student performance than Pi calculated by the authors. Also, the construction of Pi should be placed in the Methods section.

I found two typos in this section and apparently I have also expressed myself unclear. I would like to thank the reviewer for drawing my attention to this. The first remark made by the reviewer is a result of a misunderstanding. The max and min difficulties refer to globally calculated difficulties, including all items/questions for all students, as reported in the OECD technical report. This is therefore a proper normalization of the item difficulties. I have rewritten this sentence and hope that this will avoid similar misunderstandings.

I also found a typo in Eq. 1, where one of the indices was missing, which might have caused confusion. This is now corrected.

In the second remark, the reviewer claims that ”PISA test scores” are a better measure of student performance than the one proposed in the article. Firstly, there is no such thing as ”PISA test scores” on a student level. Secondly, assuming that the reviewer alludes to ”plausible values”, which are reported at student level, there are two arguments against using these: 1. They should not be used to calculate student level results, and 2. This would be significantly less transparent than using the raw data as in the present work. I changed the sentence following Eq. 1 to emphasize this more.

Important parts of the methods are often placed in the Results section. Once again, this is to improve readability.

3. How do the authors define "privileged students" using the ESCS score? What is the cut-off point, and why is that point chosen as a cut-off?

The division of students into groups/clusters is a result of the network representation of the PISA data and the clustering of the network with state-of-the-art methods including Infomap. No claim has been made about the existence of a strict cut-off, and I see no reason to clarify this further. Continuing the analogy with climate regions, what is the cut-off (in terms of eg. precipitation) between tropical and sub-tropical regions?

4. Why did the authors use bootstraps (page 7) to calculate summary statistics? The PISA sample is supposed to be representative at the country level. They can use any statistical software to calculate the presented results. From what I read here, the results are the mean of test scores and the correlation between test scores and ESCS for different groups of students.

The bootstrapping is prescribed by the OECD, as described in the methods section. The first sentence in ”Significance analysis” is now changed in order to make this clearer.

5. "Significance analysis" (page 7): It's not clear what the purpose of the significance analysis presented in this section is. If I understand correctly, the authors use significance analysis to test for the difference between mean scores across students from different economic backgrounds and/or whether the correlation coefficient between test scores and ESCS is statistically significant. If so, t-test and regressions are methods commonly used in social science for these purposes. If the authors decide to use modules and module node assignments, please cite a peer-reviewed published paper that has used this method to investigate similar issues in education or, in addition, present t-tests and regressions to show that the two approaches are similar.

T-tests and regressions are not applicable when working with networks and module node assignments. Instead, significance analyses involving bootstrapping are often used. This is straightforward in fields such as network science, and the reference provided (24, in Phys Rev. E) should suffice. The method we use has the advantage of using the relatively transparent Jaccard index (based on set theory) instead of eg. mutual information (based on entropy).

6. "Building the network of countries and space nodes" (page 9): What is the purpose of using this method? What do the authors mean by "where the country has a larger fraction of its students than the average of all countries"? Do they mean a larger fraction of "privileged students" in the whole student population? If so, the statistic they need to calculate here is the deviation from the mean of the fraction of "privileged students", and that can be calculated using any statistical software.

The purpose of using this method, as stated in the introduction, is that we can benefit from the development of methods in network theory, which have shown useful in eg. theoretical ecology to find large scale patterns in data. The other remarks made by the reviewer are somewhat unclear, but I do believe that the method is sufficiently explained in one figure, the introduction and in the results and methods sections.

7. Page 9: What is "Figure ??”

This is a LaTeX related typo that I now have corrected.

8. The authors need to provide the take-away point from each figure presented in the results section.

I believe that this is already in place.

9. Figure 1 looks like the authors map PISA's deciles with ESCS scores, but I'm unsure.

I do not understand this remark, but I have added a short explanation of Fig. 1a to the figure caption.

10. The authors also conclude that students from privileged backgrounds performing better in Sweden results from school segregation. However, at least in the US, private schools are likely to select students with involved parents, and families with involved parents are likely to have higher test results regardless of ESCS status. Also, to attract students, private schools may provide more educational resources to students and have better teachers than public schools. Can you control for these different effects?

This is a very interesting topic, but unfortunately, I believe the present dataset does not contain enough information to answer these questions.

11. Cultures, such as China, where acceptance into top universities is dependent on doing well on exams, means that Chinese students in Public schools are also likely to attend after-school private schools. Is there a way that you can control for students who attend both public schools and private schools?

I believe this information is not included in the present dataset.

12. The authors' main conclusions are 1) students with a higher socio-economic status (SES) perform better than students with a lower SES, and 2) the extent of the performance gap varies across countries. The authors should reference more of the literature on higher SES students performing better than low SES students, which is well established in the social science literature. They should also reference the literature on human capital theories that provide explanations for the difference in academic achievement between students from different economic backgrounds.

I already reference previous research on this issue, but I have now added further references to previous research [16, 17]. The main conclusion is however the identification of a cluster of countries characterized by high performance and less privileged students, together with variations between public and private schools, and the novel approach to studying these data. This is clearly stated at several locations in the manuscript, including the abstract, but to further emphasize this I suggest changing the title to ”PISA data clusters reveal student and school inequality that affects results”.

Being a multidisciplinary study, the list of referenced literature cannot be exhaustive from the point of view of every related field. One example of this is that I do not relate the findings to human capital theories, and neither do most of the other papers that deal with connections between educational outcome and socio-economical status.

Reviewer #2:

Thank you very much for the time invested and for the valuable comments.

The author analysed PISA data to explore the relationship between students' performance and economic status, searching for differences between public and private schools across countries. He applied network-based tools to construct and cluster a bipartite network formed by countries and portions of a bidimensional space representing students’ performances and economic status. Moreover, the author conducted complementary analysis to further the understanding of network clustering. He found a cluster of countries where less privileged students perform well, being this pattern an exception to the rule.

I think the manuscript is well written, well conducted and state-of-the-art in methodology. The results are robust and the interpretations seem appropriate to the methods and results. My complete inexperience in the topic prevent me to evaluate the relevance of the results, still from a complete layman’s point of view I found the results very interesting. I did not find any point that could be further improved but this two minor suggestions:

I would like to thank the reviewer for these encouraging words. The reviewer gives an accurate summary of the work despite being, as he/she claims, a layman.

I had difficulty following Figure 1a. I think the lines represent isolines, but I am not sure. I would color or number the lines and provide a legend to facilitate understanding.

This is correct and I have clarified this in the figure caption. I however believe that colors and/or numbers would clutter the figure while not contributing significantly to the story, and therefore avoid to include them.

Please check “Figure ??” in lines 286 and 288

This is a LaTeX related typo that I now have corrected.

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

1 Apr 2022

PISA data clusters reveal student and school inequality that affects results

PONE-D-21-35674R1

Dear Dr. Neuman,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jacob Freeman

Academic Editor

PLOS ONE

Additional Editor Comments (optional): Thank you for making excellent revisions and for clarifying concepts. The paper now meets Plos One criteria for publication. Best of luck on your future research.

Reviewers' comments:

14 Apr 2022

PONE-D-21-35674R1

PISA data clusters reveal student and school inequality that affects results

Dear Dr. Neuman:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jacob Freeman

Academic Editor

PLOS ONE