Computer science skills across China, India, Russia, and the United States.

We assess and compare computer science skills among final-year computer science undergraduates (seniors) in four major economic and political powers that produce approximately half of the science, technology, engineering, and mathematics graduates in the world. We find that seniors in the United States substantially outperform seniors in China, India, and Russia by 0.76-0.88 SDs and score comparably with seniors in elite institutions in these countries. Seniors in elite institutions in the United States further outperform seniors in elite institutions in China, India, and Russia by ∼0.85 SDs. The skills advantage of the United States is not because it has a large proportion of high-scoring international students. Finally, males score consistently but only moderately higher (0.16-0.41 SDs) than females within all four countries.

RCT Entities:   Population Interventions Outcomes

The rapid proliferation of information and communication technologies (ICTs) in economic, political, and social life has led to an increasing demand for computing professionals worldwide (1–4). In the United States, it is projected that over half a million ICT jobs will be created within the next decade, and by 2024 almost three-quarters of science, technology, engineering, and mathematics (STEM) job growth will be in computer-related occupations (1, 3, 4). The excess demand for ICT workers in Europe is further expected to double between 2015 and 2020 (5). To meet growing demand, employers are competing for computing professionals not only domestically but also internationally (6, 7).

The rising demand and competition for computing professionals has seen a corresponding expansion in undergraduate computer science (CS) programs. Undergraduate CS enrollments in doctoral research institutions in the United States and Canada tripled between 2006 and 2016 (4). The number of CS graduates in Europe increased by ∼150% between 1998 and 2012 (8). The number of CS graduates in China and India—approximately three and three and a half times more than the United States—also increased by 33% from 2011 to 2015 alone (see for more details).

Despite rapid increases in the quantity of CS students and graduates, however, little is known about their quality. In particular, little is known about the major-specific competencies, knowledge, and skills (henceforth “skills”) of individuals from different countries and types of CS programs. International rankings, although widely regarded by the public and in the press as indicators of quality, largely focus on elite programs across countries and, more importantly, do not consider skills in the formulation of ranks (9). Ignoring skills, the 2018 US News and World Report: Best Global Universities for Computer Science claims that 45 CS programs in the United States, 34 in China, 3 in India, and 0 in Russia rank in the top 200 (10). Although international programming competitions, such as TopCoder and HackerRank, assess coding skills, they only reflect the ability of a small number of self-selected individuals and do not measure CS skills among a wider population of students (11). No large-scale study compares standardized measures of CS skills across countries and types of programs (12).

Similarly, little is known about how CS skills differ by important background characteristics, such as gender. In many countries, female students enter and finish CS programs at lower rates than male students (13, 14). Female CS graduates also earn lower wages than male CS graduates (15, 16). Evidence on CS skill levels by gender may help explain gaps in enrollment, graduation, and employment that contribute to social inequality and economic inefficiency (17, 18).

Evidence of how CS skills compare among CS students from different countries, programs, and backgrounds can ultimately inform employers seeking to hire computing professionals within a globally competitive labor market, as well as policymakers and administrators seeking to improve the quality and diversity of programs in an international context. As such, this study compares the skills of fourth and final-year (senior) CS undergraduates from different backgrounds and programs across four major economic and political powers that train half of the world’s STEM graduates: China, India, Russia, and the United States (13).

Data and Methods

The Institutional Review Board approval for this research project was approved by Stanford University (IRB#31585). We selected nationally representative, random samples of seniors from undergraduate (bachelor’s degree) CS programs in China, India, and Russia (see for more details). We first identified all undergraduate CS programs from China, India, and Russia that had similar course requirements and content with each other and with undergraduate CS programs in the United States. In choosing the sampling frame for each country, we did a careful review of all potential CS majors in each country, and only included majors that taught core CS coursework. Like the United States, the standard number of years for bachelor’s degrees in CS programs is 4 y in China, India, and Russia. While it is true that many bachelor’s degree majors in India are 3 y, this is not true for technical (CS and engineering related) majors, which are 4 y.

Using administrative data on the population frame of all higher education institutions with undergraduate CS programs in each country, we then sampled institutions that offered these comparable CS programs. From China, we randomly selected six institutions from each of six representative provinces (36 institutions). From India, we purposefully sampled five institutions from each of three representative states (15 institutions). From Russia, we took a stratified national random sample of 34 institutions. Our sample of CS students from China, and perhaps India, may be of slightly higher math and science ability (∼0.20–0.25 SDs) than the population of CS students in those countries. As such, the estimates of CS skill levels of CS seniors in those countries may be slight overestimates. We provide further details on the national representativeness of the China and India samples in .

The national samples covered elite and nonelite programs in each country. In China, elite programs were identified as those in Project 985 or 211 universities. In India, elite programs were identified as those in India Institutes of Technology, National Institutes of Technology, and other institutions that ranked in the top 100 of the National Institutional Ranking Framework rankings. In Russia, elite programs were identified as those in National Research Universities, “5–100” universities, and Federal universities. These high-profile elite programs teach different proportions of the total number of CS undergraduates in each country (see for more details). The comparisons of elite universities favor India because students attending elite CS programs in India are approximately among the top 4% of CS undergraduates nationally, while students attending elite CS programs in China, Russia, and the United States are approximately among the top 19–26% of CS undergraduates in their respective countries.

We next randomly sampled smaller administrative units (departments and classes) within each of the sampled programs in China, India, and Russia and selected all seniors in those administrative units (see for more details). We randomly assigned half of the selected seniors to take the same standardized CS examination. Altogether, 678 seniors from China (119 from elite programs), 364 seniors from India (71 from elite programs), and 551 seniors from Russia (116 from elite programs) took the examination. To ensure representativeness, we adjusted our analytical estimates and SEs for survey design features, including multistage sampling and probability sampling weights (see for more details).

We also obtained assessment data on 6,847 seniors from a representative sample of CS programs in the United States (607 from elite programs). The sample and population of CS programs in the United States were similar in terms of the number and percentage of CS degrees awarded (see for more details). The distributions of average ACT/SAT equivalent scores of admitted students in 2015–2016 were also similar across the sample and population of CS programs (see for more details). Elite programs in the United States were identified as those from colleges with average ACT/SAT equivalent scores of 1,250 (of 1,600) or higher; these programs produce ∼19% of the country’s CS graduates (19).

Sampled seniors in the four countries all took a 2-h, computer-based, standardized CS examination from the “Major Field Test” suite of assessments designed by Educational Testing Service (ETS). The examination assesses how well CS seniors master CS-related concepts, principles, and knowledge. It consists of 66 multiple-choice questions, some of which are grouped in sets and are based on materials such as diagrams, graphs, and program fragments. The test does not assume knowledge of any particular type of software or programming language. In fact, it uses pseudocode that is meant to be easily understood by CS students regardless of program or country. Examination content areas include discrete structures, programming, algorithms and complexity, systems, software engineering, information management, and “other” (). Content areas and their proportions are aligned with the Association for Computing Machinery (ACM)/Institute of Electrical and Electronics Engineers (IEEE) authoritative international standard, Computer Science Curricula 2013, 2008, and 2003 (20) () and with the official curricula guidelines for domestic CS programs in China, India, and Russia ().

We took several steps to ensure that examination-taking conditions were similar for all students. First, we provided the same incentives to students. In particular, students were given the option of receiving an individualized report of their examination performance. Second, to address concerns about student motivation in taking the examination, we conducted robustness checks in which we excluded a small minority of students (1.7%) that did not answer at least 75% of the items. Results are substantively the same whether or not we exclude these students. Third, the examination was translated into the language of program instruction. To minimize bias due to differences in language, we followed a rigorous multistage translation and translation review process (see for more details). Fourth, examination scores were scaled to be comparable across countries (see for more details). To examine relative skill levels between countries and institutions in terms of effect sizes, we converted each student’s examination score into a z-score by subtracting the mean and dividing by the SD of the four-country sample.

The de-identified dataset and analysis code for replication have been deposited at Open Science Framework (https://osf.io/c78wb/) (21).

Results

Seniors in the United States exhibit much higher levels of CS skills than seniors in China, India, and Russia (Fig. 1). Specifically, seniors in the United States score 0.76 SDs (P = 0.000) higher than seniors in China, 0.88 SDs (P = 0.000) higher than seniors in India, and 0.77 SDs (P = 0.000) higher than seniors in Russia. In contrast, differences in CS skills between seniors in China, India, and Russia are small and statistically insignificant. [The results remain virtually unchanged when we drop students from CS majors with nonstandard names (in particular, Information Security or Information Engineering in China or Information Security in Russia) from the analysis.]

Fig. 1.

CS skills across China, India, Russia, and the United States. Mean estimates for China, India, and Russia are each statistically lower than the mean estimate for the United States (P = 0.000). Mean estimates are not statistically different between China and India (P = 0.435), China and Russia (0.914), and India and Russia (P = 0.509). Estimates are reported as effect sizes (in SD units). Scaled CS examination scores were converted into z-scores using the mean and SD of the entire cross-national sample of examination takers. As such, the overall mean of the standardized score across all four countries is zero. SEs are adjusted for clustering at the institution (university/college) level.

Although seniors in elite programs score much higher than seniors in nonelite programs in China, India, and Russia, they still score lower than seniors in the United States (Fig. 2). Specifically, the average senior in the United States scores 0.15–0.25 SDs higher than seniors from elite programs in China, India, and Russia (P > 0.100). Seniors from elite program in the United States score much higher than seniors from elite programs in the other three countries (0.85 SDs, P = 0.008).

Fig. 2.

CS skills by elite and nonelite institutions: China, India, Russia, and the United States. Within each country, the mean estimate for elite institutions is higher than the mean estimate for nonelite institutions (China, P = 0.063; India, P = 0.174; Russia, P = 0.084; United States, P = 0.000). The mean estimate for elite institutions in China, India, and Russia combined is lower than the mean estimate for elite (ACT/SAT equivalent >1,250; approximately the top quintile) institutions in the United States (P = 0.008). Mean estimates for nonelite institutions in China, India, and Russia are each lower than mean estimate for nonelite institutions in the United States (P = 0.000). Mean estimates for elite institutions across China, India, and Russia are not statistically different (P > 0.100). Mean estimates for nonelite institutions across China, India, and Russia are also not statistically different (P > 0.100). Estimates reported as effect sizes (in SD units). Scaled CS examination scores converted into z-scores using the mean and SD of the entire cross-national sample of examination takers. As such, the overall mean of the standardized score across all four countries is zero. SEs adjusted for clustering at the institution (university/college) level.

The substantial advantage of CS students in the United States is not driven by the presence of international students. We distinguish between domestic (versus international) students in the United States sample in two ways: (i) students who reported that their best language is English or English and another language equally, 94.4% of all sampled United States students; and (ii) students who responded that their best language is English (only), 89.1% of all sampled United States students. [We proxy for “domestic” versus “international” in the United States sample by using a survey question on the self-reported best language of test takers. Specifically, the survey question asked students: “Do you communicate better in English than in another language?” Student’s had three response options: (i) English; (ii) other language, and (iii) both equal. Furthermore, by way of comparison, the National Science and Engineering (NS&E) Indicators define “domestic” CS students as having US citizenship or permanent residence. According to the NS&E indicators, 95% of CS bachelor’s degree graduates from the United States from 2011 to 2015 (the years that correspond to the United States sample data) were reported by colleges as being “domestic” (13). We use the additional, stricter definition of “domestic” as students who report their best language as English only in Fig. 3, because it is possible that some students designated as “domestic” CS graduates in the NS&E indicators may have become citizens or permanent residents before graduating from college.] Fig. 3 reports the average CS skill levels for the two groups of domestic students (English/bilingual and English only) in the United States sample, along with that for the total United States sample. The average CS skill levels are extremely similar among the three groups (0.157 SDs, 0.164 SDs, and 0.192 SDs). Given the small differences, the magnitude and significance of the gaps between each group of United States students on the one hand, and China, India, and Russia, respectively, on the other, are virtually the same.

Fig. 3.

CS skills across China, India, Russia, and the United States after adjusting for United States student’s’ self-reported best language. The mean estimate of CS skills among United States students (“All”) is substantively the same as both (i) United States students who reported their best language is English or English and another language equally (English/Bilingual: 94.4% of all sampled United States students); and (ii) United States students who reported their best language is English only (89.1% of all sampled United States students). The mean estimates of CS skills for each of these categories of United States students are higher those of China, India, and Russia (in each case, P = 0.000). Estimates reported as effect sizes (in SD units). Scaled CS examination scores converted into z-scores using the mean and SD of the entire cross-national sample of examination takers. As such, the overall mean of the standardized score across all four countries is zero. SEs adjusted for clustering at the institution (university/college) level.

Finally, we find consistent but moderate differences in CS skills between female and male students within all four countries. Males score 0.15 SDs higher than females in China (P = 0.093), 0.24 SDs higher in India (P = 0.077), 0.25 SDs higher in Russia (0.022), and 0.41 SDs higher in the United States (P = 0.000). The within-country gender gaps in CS skills, while significant, are generally smaller than the skill gaps between the United States and other countries as well as between elite and nonelite programs. Females in the United States score 0.35–0.42 SDs higher than males in China, India, or Russia (P = 0.000) and 0.52–0.67 SDs higher than females in China, India, or Russia (P = 0.000). Female students from the United States also, on average, score comparably with students in elite programs in the three other countries (P > 0.100).

Discussion

The above results indicate that undergraduate students at the end of their CS programs in the United States have much higher levels of CS skills than their counterparts in three major economic and political powers: China, India, and Russia. Seniors from the average CS program in the United States score far ahead of CS seniors from the average program and are on par with seniors from elite programs from these three countries. Furthermore, seniors from the top quintile of CS programs in the United States are far ahead of seniors from elite CS programs in the other countries. Notably, the advantage of the United States is not because its CS programs have a large number of highly skilled international students.

The results, when viewed in the context of the number of CS graduates emerging from different CS programs across countries, have implications for the global supply of computing professionals. The ∼65,000 CS graduates from the United States are outnumbered, but are much more skilled, on average, than the graduates from China (∼185,000), India (∼215,000), and Russia (∼17,000) (see for more details). United States graduates only face competition from a much smaller cadre of elite program graduates in China (∼33,000), India (∼8,000), and Russia (∼4,000). A substantial number of CS graduates from selective programs in the United States further face little competition, even from the other countries’ elite programs.

The results also suggest that the CS skill gains made in CS programs vary considerably across countries. The math and science skill levels of entering CS freshmen are much higher in China than in Russia, somewhat higher in Russia than in the United States, and much higher in Russia than in India.* [Although no comparative cross-country data have been collected on the math and science skills of United States CS freshmen, we can approximate differences in the math and science skills of prospective CS freshmen in Russia and the United States by using the 2015 TIMSS Advanced dataset (22). Using the dataset, we find that among “advanced” high school seniors reporting an intention to major in CS in college, students in Russia score ∼0.335 SDs higher in math and 0.732 SDs higher in physics than students in the United States. The larger gap in physics compared with math makes sense since high school students in Russia have several years of coursework in physics, while high school students in the United States generally have 1 y of coursework in physics (23). According to the regular 2015 TIMSS data, before high school, the average eighth grader in Russia scores 0.20 SDs higher in math and 0.15 SDs higher in science than the average eighth grader in the United States (24, 25). Similar comparative data do not exist between the China and the United States or between India and the United States.] That China, India, and Russia have comparable CS skill levels by the end of college even though they start with different levels of math and science skills, suggests that program quality is lowest in China and highest in India. Although there is a much greater degree of self-selection into and out of CS programs in the United States than in the other three countries (23), the fact that prospective college students in the United States have likely similar math and science levels as students in Russia, as well as little pretertiary training in CS, implies that skill gains associated with attending CS programs in the United States are high. [In the years during which the students in our United States sample attended high school, the percentage of US high school students that earned any CS course credit was relatively small (19% in 2009) (26). Most prominently, an average of 20,934 high school students took the AP CS examination each year from 2007 to 2011 (27). If we were to assume that all AP CS examination takers from 2007 to 2011 majored in CS in college, then approximately one-third of senior CS students from 2011 to 2015 received some preparatory CS training in high school.] Although we are unable to explore possible reasons here, the potentially higher skill gains of CS students in the United States compared with the other three countries could be due to higher quality teaching or stronger linkages between college performance and employment outcomes.

Finally, despite the substantial focus of policymakers and researchers on gender inequality in CS, within-country gender gaps in skills are moderate compared with skill gaps across countries or programs (Fig. 4). The gender gap in skills does indicate that more effort is needed to attract higher-achieving female students into CS and ensure that they have equal opportunities to receive a quality education. The within-country gender gaps in skills are small enough, however, that they may explain little about gender gaps in CS graduates’ labor market outcomes (28, 29).

Fig. 4.

CS skills by gender: China, India, Russia, and the United States. Within each country, males score significantly higher than females (China, P = 0.093; India, P = 0.077; Russia, P = 0.022; USA, P = 0.000). Estimates reported as effect sizes (in SD units). Scaled CS examination scores converted into z-scores using the mean and SD of the entire cross-national sample of examination takers. As such, the overall mean of the standardized score across all four countries is zero. SEs adjusted for clustering at the institution (university/college) level.