Out of the laboratory and into the classroom: the future of artificial intelligence in education.

Like previous educational technologies, artificial intelligence in education (AIEd) threatens to disrupt the status quo, with proponents highlighting the potential for efficiency and democratization, and skeptics warning of industrialization and alienation. However, unlike frequently discussed applications of AI in autonomous vehicles, military and cybersecurity concerns, and healthcare, AI's impacts on education policy and practice have not yet captured the public's attention. This paper, therefore, evaluates the status of AIEd, with special attention to intelligent tutoring systems and anthropomorphized artificial educational agents. I discuss AIEd's purported capacities, including the abilities to simulate teachers, provide robust student differentiation, and even foster socio-emotional engagement. Next, to situate developmental pathways for AIEd going forward, I contrast sociotechnical possibilities and risks through two idealized futures. Finally, I consider a recent proposal to use peer review as a gatekeeping strategy to prevent harmful research. This proposal serves as a jumping off point for recommendations to AIEd stakeholders towards improving their engagement with socially responsible research and implementation of AI in educational systems. © Springer-Verlag London Ltd., part of Springer Nature 2020.

Introduction

AI has received substantial attention in the twenty-first century by powerful business interests, researchers, governments, and the public. Along with efforts to implement AI in organizational settings and economize on its value, the ethical, legal, and social implications have come into the forefront of discussion. Much of the focus of public, private, and academic discourse surrounding AI has addressed labor shifts and displacement, military and cybersecurity concerns, ethical issues such as bias, and economic benefits.

However, less public attention has been given to other areas of AI in social policy, especially AI in education (AIEd). This paper attempts to address that gap by characterizing and then exploring the implications of AIEd as well as providing recommendations to AI researchers and developers towards a responsible research agenda. Moreover, this analysis brings a forward-looking policy lens that can help situate the work of AI developers and researchers in the larger picture.

In Sect. 2, I contextualize the role of AIEd historically and discuss the excitement, fears, and often faulty predictions that have accompanied new forms of education technology. Digital distance education serves as a historical example from which lessons can be learned regarding AIE’s capacities and implications. Section 3 reviews three reasons why a belief in the technological inevitability of distance education fared poorly against social realities: a failure to consider implementation of education technology, a mistaken assumption of technological linearity, and the inability of distance education to reproduce teacher–student interaction. I highlight the role of teachers in quality education to explain why massive open online education has been unable to achieve its vision of high-quality, universal education.

Section 4 offers an analysis of AIEd and argues that AIEd is a candidate to overcome this barrier through the simulation of teachers, including complex social behaviors previously thought non-reproducible. I provide an introduction to AIEd and its most prominent tool, intelligent tutoring systems and then pay special attention to anthropomorphized educational agents, which most commonly simulate teachers. Section 5 discusses possible futures as AIEd begins to penetrate educational systems. I explore, both optimistically and critically, implications for pedagogy, curricula, international development and equity, flexibility and ownership of educational systems, the role and possible automation of teachers, and the use of nudging/manipulation by AIEd.

In light of the risks posed, Sect. 6 reviews recommendations to promote responsible research in AIEd and AI generally. I consider a recent proposal in computing to use the peer review process to prevent harmful research. I suggest revisions and expansions of this proposal towards a responsible approach to AIEd research and implementation. Section 7 concludes.

Education technology and sensational history

With new technology come utopian reformers and dystopian cynics. This is true in the realm of education, where the stakes are believed to be nothing short of the future of society. Here, reformers and cynics posit, respectively, that young learners will be radically empowered to reshape the world or will be placed into the service of unfeeling, technocratic forces (Feenberg and Hamilton 2012). Similar competing progressive and dire predictions have accompanied the advent of media forms that affect education, including radio, television, the Internet, and even writing. For instance, Plato, speaking through Socrates in Phaedrus, feared that writing would lead to forgetfulness and the appearance of wisdom without the reality. Like a painting, writing was static and distant and would “go on telling you just the same thing forever.” Ultimately, Plato worried that writing was a poor replacement for ‘true education’ which instead required a teacher capable of instructing and facilitating, of defining and defending ideas, and knowing “when to speak and when to be silent” (Jowett 1901). Not dissimilar critiques around educational implications of technology have accompanied radio (Siepmann 1941), television (Emery 1972; Schwartz 1974), the Internet (Iseke-Barnes 1996), and most recently, virtual reality (Evans 2003), and social media (Sunstein 2018).

In the realm of education technology, a stereotypical progressive reformer is a technologist, entrepreneur, or philanthropist who believes that the status quo is failing us and so invests time or money to promote a new vision.1 The reformer’s counterpart is then a ground-level practitioner, philosopher, or sociologist who has witnessed or studied how so-called progressive ideas are, in fact, counterproductive.2 Yet, advocates and skeptics of educational technology often shared a key proposition. Regardless of their support or opposition, many agreed that the pace of change would be rapid, the scale of disruption substantial, and the transformation inevitable (Agre 1999; Hamilton and Feenberg 2005). Ironically, this lone area of agreement often missed the mark, with its adherents failing to anticipate social and technical constraints and ultimately make faulty predictions. The mode of writing has been dynamic and productive, contrary to Plato’s expectations, and video has been successfully integrated into educational settings. More recently, while brick-and-mortar universities have not yet been supplanted by online ones, the COVID-19 virus has accelerated such a transformation (LeBlanc 2020). Simply, predicting the limits of technology—and its eventual social implications—is difficult.

While prediction is imperfect, this familiar cycle suggests that history is a useful lens with which to understand AIEd’s possible future. One such antecedent for AIEd is distance education or (digital/online) learning, which is not typically AI-based. Emerging into prominence along with the early Internet in the 1990s, distance education promised to automate, streamline, and universalize education (Feenberg 2002; Li 2002). AIEd is, in some ways, a continuation of this vision of educational technology: both are software-based, challenge the role of teachers, and potentially upset traditional understandings of classroom size, pedagogy, and more. Most importantly, both distance education and AIEd advocates imagine technology’s role in providing the highest quality instruction for learners around the world. However, distance education typically replicates existing classroom structures such as textbooks and lectures, often putting them into a video format, and lacks ‘intelligent’ features that characterize AIEd. In contrast, the intelligent tutoring systems that are central to AIEd seek to reproduce one-on-one human instruction, with personalized and real-time feedback (Self 1998). Section 4 reviews AIEd’s characteristics in more detail.

By the late 1990s, the majority of universities had an Internet-based educational option in place. Yet with the early potentials of distance education recognized, criticism followed the more optimistic aspirations. David Noble argued vehemently that distance education was being gleefully serviced towards commodification of education, deskilling and displacing professionals, and harming students, all in the name of economic efficiency (Noble 1998a, b). He feared that educational technology would turn schools into “digital diploma mills.” These criticisms echoed those of industrial and postmodern critics like Andrew Ure and Jean-François Lyotard, who saw the automating forces of their times as similarly reductive of human capacities (Lyotard 1984; Ure 1836).

A particularly salient case in distance education was the cascade of excitement about the universalizing potential of Massive Open Online Courses (MOOCs) in the 2010s, an online educational technology that is not typically AI-based (Bond et al. 2019). Unlike AIEd, MOOCs have little capacity for real-time or differentiated instruction, and instead aim at mass delivery of lecture content (Scagnoli et al. 2019). Even so, MOOCs were imagined to play a key role in ‘flattening’ education (Friedman 2005), opening up the best courses and instructors to anyone with an Internet connection. This access would rapidly shift educational infrastructure towards skill-based online education and away from long-term, credential-based education in physical spaces. Universities were among those most concerned about losing their clientele, and those most eager to adapt MOOC-style education into their approaches to evolve with the new market. While these questions have yet to play out in full, the perception of inevitability was tempered rapidly as it became apparent that dropout rates were staggeringly high and that participants tended to be well-educated and from wealthier countries (Christensen et al. 2013; Rivard 2013). MOOCs seemed to reach students who had already been reached, and employers and students were not ready to abandon the traditional degree and educational experience for a series of online videos. The predictions of MOOC’s inevitability went unrealized.

Not dissimilarly, some artificial intelligence commentators have blundered into this context, seemingly forgetting AI’s own history of failed predictions including “AI winters” in the 1970s, 80s, and 90s wherein slowdowns of technological progress and funding impeded the sense of inevitability (Hendler 2008).3 Famously, Hubert Dreyfus’ (1979) discussion of the limits of AI had to be walked back two decades later (Dreyfus 1992) after his assumptions about AI’s core focus on symbolic methods and subsequent limitations were undermined. Nevertheless, general AI-related predictions today continue to be just as, if not more extreme. Surveys of experts demonstrated substantial consensus that AI will achieve ‘superintelligence’ (surpassing humans) by the end of the century. Leading scholars, scientists, and technologists predict variously a golden age wherein all major civilizational challenges have been solved and humans are entirely free to pursue creative interests, or alternatively, the literal end of the humanity (Armstrong et al. 2014; Müller and Bostrom 2016).

Within the realm of AI and education, predictions are less sweeping but no less bold. It is widely believed by AIEd advocates that “AI will be a game changer in education,” wherein global classrooms will enable “individuals and groups to learn remarkably better than if they are taught by a single human teacher” (Woolf et al. 2013). The Department of Education’s STEM 2026 vision states that intelligent tutoring systems, the primary AI-based educational tool currently, “may play a key role in the future of education” (Tanenbaum 2016). Founder of Squirrel AI Derek Li argues that only AIEd’s adaptive systems can produce “miracles” and that entrusting teachers may risk “damaging geniuses” (Hao 2019).

The above examples do not entail that prediction is a foolish enterprise. Though it is difficult to place bounds on AI’s capacities or social implications, technological forecasting is a rich and valuable toolkit with qualitative and quantitative aspects. It can help us to envision possible (extreme) scenarios that may serve as idealistic goals or cautionary tales and can help us chart a more considered course. For example, not all commentators of distance education took polarizing positions; some offered moderating principles and strategies to address possible challenges, noting the ways in which developmental trends are positive or negative and work that remains (Beldarrain 2006; Chickering and Ehrmann 1996). The value of future-oriented thinking is also recognized in the AIEd community, including in a special edition of the International Journal of Artificial Intelligence in Education (Lane et al. 2016).4 It is in this moderating spirit that I hope to contribute to the conversation on AIEd.

Technological inevitability and evitability

What then is the gap between the ostensible technological inevitability (or fatalism) and the much clunkier reality in the historical case of education technology? I highlight three candidates which are neither mutually exclusive nor exhaustive:

The failure to consider implementation;

The assumption of technological linearity; and

The need for human interaction, especially teacher–student interaction

Implementation barriers

I will not overly focus on the first in this paper, as the topic has been discussed thoroughly.5 I observe, however, that this remains a perennial problem. Both national and international education technology initiatives have failed to sufficiently consider cost/funding, teacher training and ongoing professional development, timely rollout of complementary curricular and pedagogical supports and strategies, necessary school and system-level structural changes, and student and family context among other issues (Argueta et al. 2011; Warschauer and Ames 2010; Nye 2015).

Design-based implementation research (DBIR) is one framework that has emerged in response to these failures, focusing on collaborative multi-stakeholder design and development of sustainable capacity to address previous shortcomings in implementation (Fishman et al. 2013). Practitioners in DBIR as well as user-centered design, human–computer interaction, and other fields are increasingly expounding on contextual, regional, and demographic factors necessary for successful design and implementation. However, the fact that major initiatives like purchases of millions of tablets/laptops still fail to take account of the lessons of education technology implementation provides a cautionary note.

Technological linearity

As for the second point, the nonlinearity of technological development was pointedly observed by Pinch and Bijker in their seminal work on the social construction of technology. Technology could not be said to be linearly entailed by the nature of its underlying science. Instead, prospective technologies are open to a kind of interpretive flexibility, wherein the contextualized relevance to certain social groups and a bit of historical accident leads to closure and a stable technological form that we later accept (mistakenly) as obvious and inevitable (Pinch and Bijker 1984). Feenberg has carefully demonstrated the ways in which the relationship between technical capacities of a new educational technology and its applications is neither linear nor deterministic (Feenberg 2002). Writing has long since emerged as a powerful educational tool capable of displaying complex traits that would surprise Plato, and the Internet has developed social-interactional mechanisms that early theorists could hardly speculate. Why should we expect AIEd to be any different?

Even this criticism of retrospective closure and stability in light of sociotechnical mediation is incomplete. Following Ihde, Rosenberger argues that education technology is additionally subject to prospective reinterpretation/reinstrumentalization and thus multi-stability (Ihde 1986). Education technologists who believe their products will be used in a certain way are bound to be disappointed or at least surprised as teachers and students modify and innovate through a process of strategic adaptation.6 A tool imagined for one purpose becomes repurposed for another; closure may never actually obtain. However, these potential usages are still circumscribed by the technical nature of the tools and the social context, leading to at least temporary stabilities, hence multi-stability (Rosenberger 2017). Education technology, then, is deterministically realized neither in the past nor in the future.

A final point is relevant to the assumption of technological linearity. This is that technologies interact not only with a local social context (e.g., a classroom) or with consumer behavior, but also more pointedly with our politics and our values. AIEd, like other education technologies, is inherently political and value-laden. It implicitly assumes much about the structural norms of education—the nature of curricula, pedagogy, classrooms, educational trajectories and national goals, the local culture, and so on. Subsequently, then, educational technology reinforces these norms, reifies structures, and even preserves authorities (Winner 1980). These structural assumptions as embedded in technology are critically important to how the technology is eventually mediated. Unpacking and challenging these assumptions in technology design can reveal nonlinear possibilities, a point to which we will return.

Teacher–student interaction

Third, the nature of human interaction becomes especially salient for educational technologies that radically change the teacher–student relationship, e.g., distance education, and as we will see, AIEd. Distance education explicitly re-envisions the role of teachers and the teacher–student ratio, and less intentionally, the phenomenology of student experience. The critique is this: while distance education advocates have realized the potential of near-zero marginal cost scaling and access to ‘high-quality’ education, they failed to sufficiently appreciate how much educational quality depends on human interaction and teachers. Content was imagined above form, and educational ‘quality’ as measured by the best instructors and best schools was abstracted away from the original context which enabled that quality.

This account helps to explain why distance education has failed to universalize equitably or without substantial dropout rates. As educational philosophers, developers, and evaluators have noted, simulated, impersonal digital learning environments just do not seem to be able to replace the classroom (Mackness et al. 2010). Will AIEd be different, if it can simulate student–teacher interaction or teachers themselves? Or are there other abstractions involved in the design of AIEd which leave more to be desired?

Predictive failures, then, are explained in part by an overestimation of the quality of the novel educational technology compared with the norm it purports to supplant. This overestimation can result from a formalized abstraction of the previously successful setting (the classroom) which leaves out critical elements of quality (teachers). The gap between hypothesized and actual educational effectiveness explains the gap in implementation success, as education stakeholders are forced to revert back to previous methods when facing suboptimal results, or are simply savvy enough to be skeptical in the first place (Norris et al. 2012).

Explaining this quality gap further, Dreyfus focuses specifically on the nature of teaching and learning. He argues that basic competence can be achieved with online education, but to achieve true proficiency, expertise, and mastery, students must be exposed to risk and failure in a way that is only achievable in person. Students must be able to propose and defend ideas, make fools of themselves, and face social approbation and reckoning from the teacher. An online ‘like’ or comment does not engender this kind of emotional investment and thus learning. Moreover, Dreyfus suggests that the highest levels of expertise and mastery require apprenticeship, not only to learn complex skills like perspective- taking, but also to be socialized into professional norms and cultures. Students cannot be encouraged, motivated, challenged, or ultimately acclimated without the teacher, and the Internet provides a weak approximation (Dreyfus 2002, 2008). It may be unsurprising then that MOOCs often have limited student engagement (Cohen et al. 2019).

There are, of course, various pedagogical and design strategies being tested to promote motivation and even social interaction in online education. Notifications and quizzes during lectures, reminder emails, and suggestions on planning the course workload are all efforts towards increasing attention, motivation, and successful course completion. These efforts have been able to draw on big data, tracking individual student clicks and attention to videos, to optimize MOOC design (Simonite 2013). Small discussion forums, peer review, and even in-person meetups have been attempted to improve elements of social interaction (Goel et al. 2016). To Dreyfus, these laudable steps are nevertheless tweaks on a fundamentally flawed model. Without the teacher, digital education simply does not have the tour de force to achieve its ambitious aims.

More optimistically, Feenberg asserts that online education can achieve the quality of in-person education, but only provided it incorporates teachers more robustly. What online education needs to match the quality of traditional education is facilitation, moderation, synthesis, and leadership: “raising topics, recognizing participants’ contributions, and summarizing discussion at key points” (Feenberg 2008). Similar critiques are manifested in the Distributed Open Collaborative Course (DOCC) movement, a MOOC alternative based on feminist pedagogy. DOCC’s pedagogy emphasizes distributed rather than central expertise, collaboration and community rather than isolation, and diversity in participants and perspectives. A clear expression of this philosophy is the idea of digital office hours, a personalized interaction that is incompatible with typical MOOC structures (Koh 2017).

Differentiation

It should also be pointed out that these conceptualizations of the critical role of teachers offered above imply an additional need: individualization or differentiation. To provide effective guidance, motivation, facilitation, and emotional investment, a teacher must know their students. They must understand the students’ social, emotional, and educational past and present. Only with this knowledge and skillful practice can educators inspire students with unique backgrounds, learning styles, and interests. This vision is clearly not embedded in MOOC-style distance education, as asynchronous posting, passive reception of video content, and even the use of pseudonyms (usernames) exacerbates the issue of impossible student–teacher ratios.

Importantly, while in-person meetups and digital office hours are plausible at small scale and do start to address the concerns of Dreyfus and Feenberg, these teacher-centered approaches to digital education contradict the raison d’être of massive online education—large scale and reduction of human capital costs. In the face of this tension of quality versus scale, scale dominates. The result of this model is that scale-oriented MOOC education model often leaves students untethered in the face of “the lack of structure, support and moderation” (Mackness et al. 2010).

In contrast to MOOCs and other distance education tools which do not traditionally involve AI, AIEd technologies focus heavily on the perspective of learners (Lane et al. 2016), on pedagogy, and on differentiated, one-to-one instruction. AIEd tools are becoming capable of social–emotional engagement and can even be integrated into scale-focused distance education technologies. Moreover, while AIEd has been used in applied settings for decades, its prospects (Nye 2016) and usage (Hao 2019) are now greater than ever before, thanks to research advances and especially increased access to personal computers and the internet. In short, AIEd is leaving the laboratory and entering the classroom. In light of this, the next section reviews the argument that AIEd is a candidate to bridge the quality gaps surfaced by the previous round of education technology.

AI and educational agents

Distilled to a single term, the argument for the quality gap presented above is teachers. Without teachers (or teaching), we have good reasons to think that the quality gap between traditional and digital education will not be so easily closed and hence implementation will fall utterly short of the hopes of educational technology’s strongest proponents. This is why AIEd is so enticing: it offers to close the quality gap by tackling the most insoluble barrier—teaching.7 I argue that the central ambition of modern AIEd is to simulate teachers.

A most useful definition of artificial intelligence8 in the case of education is “the study of how to make computers do things at which, at the moment, people are better” (Rich and Knight 1991). The ‘thing’ referred to in this paper is teaching, and those people that AI hopes to emulate and surpass are most often teachers (otherwise tutors, mentors, or even educational administrators). Below, I provide a brief overview of AIEd technologies (both extant and emerging), focusing on the predominant class of tools known as intelligent tutoring systems.9 I characterize their basic design, functional capacities, and expected utility, demonstrating how they increasingly encroach on the traditional niche of human teachers. I then highlight a subset of intelligent tutoring systems known as educational agents, effectively anthropomorphized software-based characters. Educational agents represent the attempt to tackle some of the hardest challenges in simulated teaching, namely perceiving and portraying emotion in a way that engenders authentic student response.

AI is used in a wide variety of educational applications, including: adaptive assessments (Heffernan and Heffernan 2014), educational and social robots (Timms 2016; Wang and Johnson 2016), lifelong intelligent mentors (Woolf et al. 2013), and more (McArthur et al. 2005; Czopek and Pietrzak 2016). AI is also being applied to traditional educational tools, with intelligent features being incorporated into ‘adaptive’ learning management systems (Nenkov et al. 2016), ‘adaptive’ MOOCs and distance education tools (Sonwalkar 2013; Xiaohong 2017; Rosé and Ferschke 2016), and predictive analytic systems in higher education, for example, aimed at improving graduation rates and career outcomes (Dimeo 2017; Tastimur et al. 2016). For a use case to qualify as an example of AIEd, it must involve some ‘intelligent’ component. Finally, AI is important to related educational fields such as educational data mining (Romero and Ventura 2010) and computer-supported collaborative learning (Dillenbourg et al. 2009).

However, the most prominent form of AIEd is intelligent tutoring systems (ITS) (Niculescu 2016; Roll and Wylie 2016). ITS are defined as “computer-based learning systems which attempt to adapt to the needs of learners” with the “scientific goal to make computationally precise and explicit forms of educational, psychological and social knowledge which are often left implicit” (Self 1998). ITS are, therefore, aimed at opening the black box of teaching and learning and simulating it computationally. While this paper focuses on ITS, ITS are often connected with other AIEd tools, such as educational robots and adaptive assessments, creating broader AIEd ecosystems (Nye 2016). An ITS is typically constituted by some variation of the four following software components (Samuelis 2007):

An expert domain, which holds the relevant subject knowledge, e.g., contains all the information related to high school algebra that will be employed by the ITS.

A learner domain, which keeps track of how the student has engaged with the ITS, e.g., what questions the student has answered correctly and incorrectly and, via prediction, what the student (probabilistically) understands.

A pedagogical domain, which structures strategies for providing targeted feedback and material to students, addressing mistakes and questions as would a teacher.

An interface through which the student interacts with the software, potentially including various forms of multimedia or different representations of the digital learning environment. An educational agent, for example, would manifest as a quite different kind of interface, one in which the student interacts with an anthropomorphized character, even a robot.

The resemblance of this four-prong structure to our model of formal education is no coincidence. Indeed, the goal is to unpack—and, therefore, eventually simulate—what it is that happens within the black box of teaching and learning. Importantly, our interpretation of that black box is not a straightforward or unambiguous representation; how we conceive of teaching and learning and what they unpack into is shaped by the social and political context of educational settings at the micro-level and our educational systems writ large at the macro-level.

Among the most evident influences of our social context on AIEd is the fundamental emphasis on scale. In theory, an effective ITS can serve any number of students all at the same time, assessing student answers and differentiating instruction instantly. It can store a student’s entire history of mistakes and growth, their patterns of learning, and even micro data about engagement, motivation, and more. The capacity for differentiation is especially notable in contrast with MOOCs. MOOCs cannot at present differentiate at all; all students watch the same videos and complete the same assignments with no attention paid to student characteristics, educational history, or learning style. ITS can differentiate readily and en masse in ways that arguably surpass the capacities of human teachers. In sum, ITS promise impressive capacities, filling the very gaps that plague traditional non-intelligent distance education.

Furthermore, ITS are becoming richer and more robust, increasingly attempting to simulate what were once thought to be ‘uniquely’ human capacities, namely emotional/sentiment measurement, analysis, and representation (Woolf et al. 2010; Arroyo et al. 2014). ITS are attempting to track student attention (D’Mello 2016), detect emotions such as frustration, and even teach physical motor skills (DeFalco et al. 2017; Harley et al. 2017). Along with improving the ability of ITS to measure and interpret emotional data through affective computing, efforts are increasing to portray emotion, encourage motivation, and “care” about students (Du Boulay 2011; Du Boulay et al. 2010; San Pedro et al. 2013).

Among the most interesting and cutting-edge ITS tools which often use social–emotional capacities are advanced versions of educational or pedagogical agents. These agents had early precursors in the 1990s but are now reaching far higher levels of sophistication (Dinçer and Doğanay 2017). Educational agents can be represented visually, with sound or with text (Baylor et al. 2003). They may appear as human faces, full human bodies, cartoons/animations, or may even be represented in embodied robots (Dunsworth and Atkinson 2007). Text-based educational agents must respond to students in a convincing human-like manner with words, while auditory agents and embodied robots must master inflection and facial expressions. In light of these features, educational agents are theorized to foster socio-emotional connection, interest, and engagement, and thus produce superior learning outcomes. Evidence even in these early stages supports the claim that that humans do bond with nonhuman entities (Tanaka et al. 2015) and that educational agents have positive impacts on motivation and academic outcomes (Bickmore, Pfeifer, and Paasche-Orlow 2009; Domagk 2010; Woolf et al. 2010).10

On top of the different interfaces through which an educational agent can manifest, the agent can also take on different roles. I adapt work from Chou et al. (2003) and Payr (2003) on offering a simple ideal type depiction of educational agents (Table 1). Educational agents can be parsed by role (teacher, teacher’s assistant) learning companion, peer tutor), by content expertise (expert to beginner), by emotional stance (from strict/authoritative to neutral to highly emotional), and by activity (from direct instruction to guidance to co-learning). Example ideal-type agents are the authoritative teacher, who contains all the relevant domain knowledge but portrays limited emotionality; the coach or tutor, who provides more mentorship, guidance, and support; and the learning companion, who has more restricted expertise and serves as a co-learner and peer.11

Table 1

Ideal-type educational agents

Role	Teacher, professor	Tutor, coach, teaching assistant	Learning companion
Content expertise	Expert, full domain knowledge	Quasi-expert, substantial domain knowledge	Peer, similar to student within some range
Emotional positionality	Strict, neutral, authoritative, older	Supportive, motivating, semi-authoritative, older peer	Student-like, engaging, non-authoritative, same-age peer
Activity	Instruct, challenge, hold accountable	Guide, facilitate, encourage	Excite, relate, foster learning-by-teaching

This is just a sample of the forms that an educational agent might take, as professor-like agents need not be emotionless nor learning companions juvenile, and indeed thoughtful design might suggest a hybrid of these kinds of characteristics. Nevertheless, these efforts to simulate humans as educational agents are revealing in terms of how they are encroaching on the traditional niche of human teachers and what that might mean for the future of education.

At this point, it is worth reviewing the nature of AIEd and ITS in light of our prior discussion of distance education:

There are strong analogies between distance education and AIEd as forms of digital education. Both purport to achieve massive scale at low marginal cost. Both promise to challenge traditional educational structures, especially with regard to the role of teachers. And of course, both are subject to hype and skepticism.

However, the disanalogies are also important. While MOOCs traditionally lack intelligent features and provide effectively no individualization or differentiation, AIEd is built on the AI paradigm of individualized, immediate responses. AIEd thus explicitly tackles one of the great barriers of distance education by simulating teachers, promising something closer to what Feenberg desired in a small classroom distance education setting. In essence, it is a candidate for the best of both (but not necessarily all) worlds.

Moreover, while MOOCs offer little in the way of socio-emotional engagement and support, especially with respect to the student–teacher relationship, AIEd is advancing unafraid towards this goal, especially via educational agents. AIEd is thus explicitly addressing the very gaps proposed by Feenberg and Dreyfus regarding why distance education just can not replace traditional human teachers.

Feenberg (2002) writes that “whenever a new educational technology is introduced, arguments emerge for substituting interaction with the technology for the process of intellectual exchange. But there is something about dialogue, and the active involvement of the teacher, that is fundamental to the educational process and that should be woven into the design of every new instructional tool.” Is it possible that something about AIEd technology is different, that it breaks with the typical cycle of substitution, overreach, retreat, and eventual synthesis?

If it has become accepted that computers can store more information and perform calculations more quickly, it has certainly not been accepted that computers can replace teachers as mentors and motivators. And yet, this perennial expectation is being challenged. While it is yet to be seen how robustly, convincingly, and effectively emotionality can be assessed and portrayed by AI, those who assume the irreplaceability of teacher interaction, differentiation, facilitation, and emotionality ought to revisit these assumptions. I discuss a related issue of replacing or otherwise transforming the work of teachers in Sect. 5.3.

Even if AIEd’s simulation of teachers is less-than-perfect and clear quality gaps remain, as they will initially, advocates for technological substitution will have increasingly strong arguments for highlighting the advantages of AIEd—scalability, differentiation, instant adaptability, and lower cost. For this reason, it is important to review possible models of AIEd in the future. What sociotechnical considerations will impact the success of AIEd? What are the risks if the technology is pushed too far too soon? What ethical and design questions need to be asked and answered before major implementation efforts are undertaken? In the next section, I consider these questions.

Possible futures for AIEd

I review possible futures for AIEd and educational agents in particular with respect to two competing visions. These resemble what Feenberg calls the factory and city models. The factory model represents “a society reflecting in all its institutions the logic of modern production, obsessed by efficiency achieved through mechanization and management.” In contrast, “the city is the place of cosmopolitan interactions and enhanced communication. Its god is not efficiency but freedom. It is not dedicated to rigid reproduction of the same, ‘one best way’, but to the flexible testing of possibilities” (Feenberg 2002). If the factory model is a technocratic dystopia, then the city model is its progressive counterpart.

My approach emphasizes the factory or dystopian model and, therefore, appears closer to the viewpoint of the skeptic. The purpose is to emphasize social and ethical risks by making them especially salient (a risk-averse approach). As a result, there are many benefits from AIEd that are not fully explored here. Even some of the negative scenarios imagined here could be viewed more charitably.12 I encourage the reader to keep this lens in mind even where I have not.

I discuss social and ethical implications of AIEd regarding bias, curricula, international development, flexibility and ownership of educational systems, the role and potential replacement of teachers, and the use of nudging/manipulation by AIEd. In each case, I consider the optimistic ‘city’ perspective alongside a more critical ‘factory’ analysis.

Pedagogy and bias

Fully realized, an optimistic AIEd vision of pedagogy under the city model is one of robust differentiation. AIEd systems will be able to deeply understand students, including their cognitive development and history, emotional reactions, level of engagement, and even their future educational pathways. While differentiation remains a major challenge in classrooms today, with the flexible and responsive technology applied by AIEd, students would learn optimally, supported cognitively, structurally, and socio-emotionally with unprecedented levels of sophistication (Tomlinson 2000).

In contrast, an emerging fear in artificial intelligence is that of algorithmic bias, which may favor or disfavor certain subgroups by gender, sex, race, ethnicity, socioeconomic grouping, and other more subtle factors (Friedman and Nissenbaum 1996; Howard and Borenstein 2017; Bolukbasi et al. 2016; Buolamwini and Gebru 2018). AIEd systems are developed through training data, and those data will be extracted from dominant and early user bases. AIEd, therefore, learns about student engagement, learning, and cognitive styles from a subset of potential future users. Plausibly, these users will come from more highly-resourced school systems that can afford to purchase and experiment with new tools. These users may also come from wealthier countries and especially from countries where AIEd development is predominant, likely in high-income countries. This issue of intercultural use and bias is also considered by Pinkwart’s (2016) dystopian scenario and likely to be a real concern. This is unfortunate given that low-income countries are the places that could be most positively impacted by AIEd.

What sorts of assumptions are implicit in the educational settings of wealthier nations, in the structure of their classrooms, and in the backgrounds of their students? ITS and educational agents built on training data from one part of the world may not speak to or effectively educate students from another (Anderson and Adams 1992; Fejerskov 2017; Kak 2020).13 Indeed, challenges with technological access and literacy affect a huge portion of learners around the world (Nye 2015). Thus, when AIEd favors certain pedagogies, learning styles, and educational systems, it ultimately dis/advantages certain students and their communities. Finally, implicit and explicit biases in AIEd development also reflect the slanted demographics of its researchers—their own backgrounds and assumptions about education are already shaping AIEd (Blanchard 2015).

Curricular emphasis on STEM subjects

A major promise of online education has been its potential to take any subject material, perhaps taught by the ‘best instructors’ in the field, and make that available to anyone at any time. This complete universalization of curricula, coupled with the powers of AIEd, could ostensibly enable the original promise of MOOCs but with critical educational quality issues such as the role of teachers/teaching largely solved. This would represent the true flattening of education.

However, a similar risk obtains as with pedagogy. The AIEd development ecosystem will make strategic choices about what content is developed first, and as AIEd moves out of the laboratory and into classrooms, corporate developers and large purchasers will have an increasing say. Education technology companies have a critical role in translating ‘laboratory’ research into practice, but their financial motives may favor development decisions that neglect nuances important to AIEd researchers.

Moreover, the development of AIEd systems is itself conditioned by the nature of the technology. In particular, some subjects are easier to model computationally than others. Algebra is quite easy to create an ITS around; history, not so much. Mathematics relies on well-established equations and symbols, and even standard problem-solving approaches and student misconceptions are well understood and easily modeled by AIEd. Understanding meaning and context via natural language processing is far more challenging than simple mathematical representations. Evaluating the quality of a student’s personal reflection on current issues or providing feedback on poetry are far more ambitious goals for AIEd. This is true both with respect to the expert domain (the content itself) and the pedagogical domain (how to teach the content). For example, data used to train an ITS can be obtained from both student answers and teacher feedback, and these data are much more straightforward to obtain from simple question–answer pairs in mathematics than from a winding and complex teacher–student conversation about historical perspectives or literary criticism. Indeed, it is no coincidence that current AIEd reflects this bias towards modeling STEM content (Roll and Wylie 2016).14

There is a further corollary of this logic related to subject bias having to do with the popularity of the subject material. More training data can be obtained from larger, more common courses as compared to smaller and more esoteric ones. Large introductory courses will provide robust and consistent data sets; esoteric and upper-level courses will be too rare, inconsistent, and expensive to capture data from. Many AIEd introductory algebra courses already exist, while courses on philosophy and art history are unavailable. Given educational and developmental pressures to produce content, developers may not wait to produce a full suite of English, social studies, science, and math courses, much less anthropology or gender studies. They will produce what can be done more easily and cheaply and marketed in turn.

The upshot of these technical realities and developmental pressures is that large, introductory courses in STEM will be favored while courses in arts, social sciences, humanities, and other areas may be further relegated to second-tier importance (Olmos-Peñuela et al. 2015). This outcome, also discussed by Pinkwart (2016), is highly reminiscent of complaints that labor force, economic, and even military motives have pushed well-rounded, liberal education out of the way in the pursuit of STEM. This educational emphasis has further implications for student moral and civic development, diversity, equity, and more (George et al. 2001). The technical nature of AIEd may provide even more momentum towards this imbalance.

Teachers’ role and displacement

One more well-established concern with AI is that of technology-induced role change and displacement, with subsequent effects for the exacerbation of inequality and economic and political instability (Autor 2015; Makridakis 2017; Manyika et al. 2017). The AIEd community is often careful to argue that its projects are tools to support and augment rather than displace teachers (Corbett et al. 1997; Pinkwart 2016). Many AIEd systems are designed to be used in coordination with teachers and cannot function without teachers’ active facilitation and complementary instruction. For example, the ASSISTments project (Heffernan and Heffernan 2014) recognizes the mantra “put the teacher in charge, not the computer.” This view is one of positive complementarity (Hannafin and Savenye 1993).

Under this paradigm, AIEd will serve as a tool to empower teachers. It will provide detailed information about student learning, enable unprecedented levels of differentiation, and enhance the status of the profession. Teachers will be freed up from the most rote tasks, and afforded the flexibility, creativity, and time to pursue advanced professional development or other educational goals in one’s school and career.

There are a few reasons to be hopeful about this trajectory. One is the strong intentions of many AIEd researchers themselves, an important force shaping design. A second reason is that teachers do have some agency over whether and how tools are used; if they are not supportive of AIEd, the tools are unlikely to flourish in classrooms. Finally, some evidence suggests that teaching is among the hardest professions to automate (Frey and Osborne 2017), particularly because of its focus on social–emotional skills.

However, some empirical and socio-political considerations cut in the other direction. First, the intentions of AIEd researchers may not be enough. As discussed, designers may have only partial control over the multi-stable usages of a design. This is especially true when commercial interests develop tools based on economic logic. Education technology companies and governments may find it profitable and efficient to emphasize labor-saving versions of AIEd, after initial versions are transferred by AIEd researchers.

The aspiration to support, not replace, teachers is also undercut by research efforts to learn from and model human teachers, to function in environments without teachers, and to be evaluated against the performance of teachers (Boulay and Luckin 2016; Cumming and McDougall 2000). Comparative studies have found that ITS are rapidly converging on the effectiveness of human teachers.15 Van Lehn’s (2011) meta-review of 28 comparison studies found that ITS were essentially equal to human tutors in driving student learning gains (VanLehn 2011).16 Kulik and Fletcher (2016) find similarly in a meta-analysis of 50 studies that students taught by ITS outperformed control students who received no ITS in 92% of studies, stronger than typical human tutors. The very fact that the AIEd community is proactively aware of this possibility of replacing teachers in at least some settings speaks to its plausibility.

For example, amongst the largest uses of AIEd out of the laboratory is Squirrel AI (or Yixue), an education technology company in China (Cui et al. 2018). Squirrel has registered over a million students in more than 200 cities, and is growing. For both students and instructors, the laptop is the main vehicle of learning. Squirrel’s founder, Derek Li, argues that when AIEd technologies reach their ultimate vision, “human teachers will be like a pilot,” playing a passive role, monitoring computer dashboards until a student is flagged, and even then focusing on emotional (not academic) communication (Hao 2019). Thus, while AIEd researchers may believe that their tools require the involvement of teachers in some way, it is not merely wholesale replacement of teachers that is important. Squirrel AI still involves a role for teachers, but it makes it easier to increase student–teacher ratios, displacing or removing a need to hire teachers in the first place. Moreover, it puts teachers into an arguably lower-level role of facilitating computer systems and managing student social and emotional needs.

Second, teachers may not have as much agency in making choices about the usage of AIEd as imagined. The history of education technology and its intersection with teachers reveals a more pained relationship (Cuban 1986). Teachers tend to be (perhaps rightly) resistant to the emphatic insistence of technology reformers, and implementation, use, and eventual outcomes fall far short of the optimistic education technology predictions. Some teachers thrive, leveraging new tools and innovating beyond even the creator’s original intentions, but other teachers may lack comfort with the tools or reject the implied pedagogical and structural changes. This resistance itself feeds justifications for resentment and blame of teachers, and arguments for more aggressive enforcement and deprofessionalization of teachers. Moreover, increasing public expenditures in education, concerns about economic competitiveness and inequality, and actual or perceived stagnation of educational outcomes all increase pressure on teachers.17 While it is true that teachers have some agency over whether AIEd tools are used, they are often overruled by administrators, policymakers, and other powerful interests.

Third, while some projections of automation suggest that teachers are relatively insulated, this insulation is granted by the focus of teachers on social–emotional tasks, which are currently hard to automate. Yet, even these projections are based on only the presumed near-future capacities of machine learning (Frey and Osborne 2017). This suggests caution, as other tasks previously thought non-automatable, like driving, can now be computerized. It is impossible to predict the limits of AI. Moreover, as we have seen, the AIEd community is undertaking very deliberate efforts to close the social–emotional gap, and to computerize the very tasks that currently insulate teachers.

If the complementarity thesis fails then, the alternatives of displacement, deprofessionalization, and deskilling may dominate. The eventual outcomes of the substitution paradigm are larger class sizes, job displacement, and even relegation of the teacher role to a mere facilitator job, where human decision-making is replaced by algorithms (Guilherme 2019; Croy 1989). In this role, low-skill instructors walk up and down the aisles of a large computer lab, policing student behavior and ensuring logistical smoothness and basic student engagement. This fear, dating back to Nobel’s concern of digital diploma mills, it is not so far-fetched. It is evidenced by Squirrel AI’s rapid growth, as well as in classrooms in low-income countries which currently employ prepackaged curricula coupled with low-skill teaching guides (Rangan and Lee 2011). Even success stories like Georgia State University, with increased graduation rates and closure of achievement gaps, have relied on large computer-based courses with high student–teacher ratios to achieve educational goals inexpensively (Dimeo 2017).

International development and premature automation

The above analyses can be extended to consideration of education across different regions, socioeconomic settings, and countries (Nye 2015). The utopian view is, again, universalized access to top-quality educational materials and now AIEd-supported instruction (Pinkwart 2016). While digital education and MOOCs were previously unable to address issues of differentiation, local context, and socio-emotional engagement, AIEd promises to improve on these challenges. However, even a successful long-term complementarity might be preceded by short-term sufficiency. Perhaps, the best we can do given established challenges with the educational systems and teacher workforce in many countries is to provide highly packaged instructional tools (Birdsall et al. 2005).18

On the other hand, investing in technological solutions and accepting packaged education facilitated by low-skill classroom guides may in fact be the short-term good destroying the long-term better. While educational aid has recently stagnated, the promise of cheap, effective educational tools could lead to a revolution in educational foreign aid (UNESCO 2017). If AIEd technologies are viewed as relatively superior to the status quo without demanding major infrastructural spending and change, the expansion of AIEd may come at the cost of long-term educational development. We might call this the premature automation of education.19 That is, a country’s investment in its educational infrastructure, systems, and perhaps most importantly preparation and ongoing professional development of high-quality teachers may suffer, because AIEd is viewed as a sufficient short-term solution.

This risks precluding long-term educational effectiveness and international equity, locking certain communities or countries into a kind of permanent second-tier educational system, particularly if low-income countries are viewed as a laboratory for AIEd and commercial interests (Fejerskov 2017). Therefore, even if we accept deprofessionalization of teachers and increased teacher–student ratios as possibly ‘better’ short-term strategies endorsed by the likes of USAID, we ought to be very careful with long-term implications for educators and educational systems, especially when our proposed solutions are imposed from the outside.

Flexibility and ownership of educational systems

Accompanying the vision of finely individualized pedagogy and curricula is a view that local communities, families, and students themselves will have unprecedented control over their own educational destinies. Students will have significant flexibility in choosing the courses they desire at the pace they want towards their anticipated academic and professional trajectory. Schools will be able to experiment under a supportive regulatory environment, tailoring educational experiences to the needs of individuals and their communities. Under this view, educational ownership becomes an increasingly local project, engaging and inspiring teachers, families, and students towards better educational, social, and civic outcomes.

The contrasting view reflects challenges already presented in the pedagogy and curriculum sections above. Under this scenario, curricula will be packaged with aforementioned biases toward certain content, cultures, and learning styles. Official entities at higher levels of aggregation, such as school districts, states, and countries, will determine educational priorities for progression and graduation. These priorities may reflect national interests, rather than individual or local ones, and may additionally reflect unbalanced private sector market influence and lobbying.20 The irony here is that a technology lauded for its ability to individualize could be used instead for mass standardization. If AIEd enters the classroom in such a way that it decenters and supplants teachers, it could detract from the creativity and diversity made possible by the esoterica of individual teachers, and even decrease educational freedoms of students and families.

Nudging and manipulation

One last element worthy of review is the idea of nudging, or framed less positively, manipulation. If we anticipate increased abilities of AIEd systems to predict, measure, and respond to emotional data, we raise questions about the form those responses will take. Ideally, these emotional systems are used to support and improve students educationally as well as socio-emotionally, taking into account the kinds of broader goals we might have for students beyond test scores and graduation.

However, the possibility of nudging raises a suite of challenging ethical issues, presented in a slightly different context by Borenstein and Arkin (2017), which I expand on here. For example, do we expect or want students to be consciously aware of, or be able to decide whether a nudge occurs? Do we want nudges that harm, even in service of calculated long-term benefit, such as an AI system shaming a student for laziness or mistakes?21 More generally, is it appropriate to treat student emotion as a means to an end? Do we want AIEd systems to manipulate student cognition with flashy colors and potentially addictive gamification techniques?22 Are there particularly vulnerable students, “due to age, physical or mental characteristics, socioeconomic status, or other relevant factors” toward which we would want to pay special attention (Borenstein and Arkin 2017)? A further issue compounding these nuanced challenges is the so-called black box problem, wherein some AI systems built on big data rely on complicated and fundamentally opaque decisions (Castelvecchi 2016). In such cases, even the programmers of AIEd systems may not understand where and why students are being nudged, rendering legitimacy even more problematic.

Final notes on the city and factory

Readers interested in other depictions of possible futures of AIEd should see similar work by Rummel et al. (2016) and Pinkwart (2016). Both present utopian and dystopian scenarios, with the latter serving as a cautionary tale. Rummel et al. imagine a future of computer-supported collaborative learning where AIEd systems lack flexibility, are opaque and intrusive, and ultimately undermine student collaboration and motivation. Pinkwart imagines an ITS which is culturally biased towards US learners, pedagogically favors subjects like math rather than ethics, and where companies commercialize student data and even profile students as dangerous. In response, both papers offer suggestions for research. Pinkwart proposes increased attention to issues such as intercultural education, achieving practical impact in classrooms, and student privacy. Rummel et al. emphasize a need for more transparent and flexible AIEd tools. In the next section, I move to recommendations to address some of the ethical and social risks raised in this paper.

What’s an AIEd researcher to do?

These possible futures represented as the factory and city model may ultimately be as inaccurate as some of the predictions cited previously, but they minimally help to situate AIEd in the present and reveal novel ways in which AIEd as a technical tool can interact with social and political realities. Awareness is a good starting point, but how is this information and speculation useful to us beyond mere consciousness-raising? The proximal audience for this paper is those with some interest in shaping the future of AIEd systems, and AI more broadly as it impacts social and policy outcomes. As such, the reader may have some influence over the possible futures of AIEd and AI generally. How might this influence be exerted in a responsible and prudent manner?

One promising and growing framework in this vein is Responsible Research and Innovation (RRI), which “asks researchers to take on a practice of critical reflection considering the potential societal impacts of their research outputs, and second to include the general public in a dialogue around the development of research goals and strategy” (Schomberg 2013). RRI has been applied to a number of domains and regions, and can be applied to computer science research and AI in particular.

Aligned with the sentiment of RRI, a proposal from the ACM Future of Computing Academy (FCA) argues that in light of increasing recognition of possible harms of computing, it is incumbent on researchers to “work much harder to address the downsides of our innovations” and that the failure to do so has represented “a serious and embarrassing intellectual lapse” and arguably an ethical one (Hecht et al. 2018). The recommended response is to leverage the gatekeeping function of the peer review process, requiring papers to include robust broader impact discussions that consider (1) potential harms as well as potential benefits of the research, and (2) technical and policy options that would mitigate those harms. The proposal also suggests that research that entails net harms and researchers who repeatedly pursue such research should face strong resistance and rejection.

While this use of the peer review process represents an interesting and potentially viable mechanism to promote socially responsible research, there are also significant limitations as well as reasons to favor other approaches. Tapan Parikh rebuts that pursuit of harmful research may be important for prophylactic reasons, and points out that researchers are not wholly responsible for how the technologies they develop are used politically.23 Even seemingly beneficial or harmless research can be misused, and these misuses cannot be controlled easily by researchers (Waddell 2018b). Moreover, the ability of both authors and reviewers to accurately and unbiasedly capture, predict, and review risks in such uncertain domains and potentially far outside of their field of expertise is highly questionable (Parikh 2018). Is it reasonable to expect an AI researcher to provide accurate models of labor displacement, produce a cost–benefit analysis, or to engage with critiques in education implementation or sociology?

Consequently, attempting to assess net harms may not be practical and thus is unlikely to be a reliable rubric for critical gatekeeping decisions. I would add that the focus on net harms versus benefits itself misses the point in an important sense. A net harm/benefit discussion emanates from the tradition of optimization and efficiency and is often orthogonal to critiques regarding distribution and equity (Callahan 1964). Ironically, these are often the very intellectual and ethical lapses that computing research has been accused of in the first place. For example, it may very well be that a certain job-displacing automated technology leads to net benefits, measured by economic growth, GDP, and perhaps even net jobs. But concerns about automation were never purely about GDP in the first place; the concern is that many people in certain job types, sectors, or demographic groups will lose out, irrespective of society’s net benefit. Similar points can be made about racially biased facial recognition or criminal sentencing AI systems, which are opposed not primarily on aggregate efficiency grounds, but on ethical ones related to fairness, transparency, and accountability (Buolamwini and Gebru 2018). Therefore, again, even if peer review is a viable gate, net harm and benefit are not appropriate rubrics for gatekeeping.

Another challenging question is the scope of implications that a researcher is responsible for. Must a researcher be accountable to the implications of their entire field or sub-field (the large-scale implications of AIEd discussed previously)? Must every AI paper about image processing include a boilerplate discussion of the risks of facial recognition technology? This seems both unfair and wasteful. However, equally problematic is the risk that few individual papers (e.g., review papers, meta-analyses) synthesize and encompass the set of developments and risks inherent in the broader field. As such, most single papers will not and perhaps should not discuss these issues in depth.24 Instead, collective gatekeeping of a domain of research or new technology seems more effective and appropriate than gatekeeping of individual papers. Along these lines, one of Parkih’s helpful proposals is to leverage existing research fora outside of the peer review process. As is done in some cases, conferences can host interdisciplinary panels, and journals can encourage integration of extradisciplinary ideas even if they do not advance ‘core research.’25

Ultimately, the FCA proposal raises an important issue, and is certainly correct that research should not be presented through solely rose-tinted glasses. Minimum consideration of risks is a good idea, and while thoughtful expansion of broader impact consideration is a great start, the research community needs even more robust ways to engage with these complex technoethical issues and their associated uncertainties. Instead of individual gatekeeping then, responsible research and implementation suggests broader thinking of impacts at all stages of the research process, from brainstorming to publication to implementation and evaluation.

A key element of this responsible research is the inclusion of the subjects, targets, and decision makers who surround the research/technologies in question into the research process, implementation, and associated fora. Gatekeeping should not be done solely by academic reviewers. While incorporating broader voices is complex and can be especially challenging for technical communities (Shilton 2018), work on participatory methodologies and DBIR has demonstrated that inclusivity is perfectly feasible (Schuler and Namioka 1993). For example, the healthcare research community is currently working through the micro- and macro-challenges of incorporating patients as co-equal researchers, and has valuable lessons and models from which the computing and AI communities could learn (Domecq et al. 2014).

In light of all of this, what might improving our capacity to conduct thoughtful and socially responsible research look like in the case of AIEd?

Currently, teacher collaboration in AIEd research is only partial, with teachers involved most often as accessories during implementation (Roll and Wylie 2016). Instead, teachers and other educational stakeholders (students, parents, administrators, education researchers, and policymakers) should take a much greater role in the development and implementation of AIEd research. Fortunately, there is extensive work on design in educational technology that provides methods and frameworks to better incorporate teachers and other stakeholders in design (Luckin et al. 2013). The incorporation of diverse perspectives is beneficial both for researchers and developers who want their products to be truly useful, while also encouraging development of technologies along socially responsible pathways.26

AIEd research publications and platforms should actively discuss how exactly they interface with larger scale educational systems and affect social and policy considerations, not merely how the technology works in abstract or within a single classroom. This suggests an emphasis on implementation studies, perhaps along the lines of DBIR (Fishman et al. 2013), as well as increased work with social science and policy scholars outside of education. Academic and corporate developers should articulate clearly how exactly their platforms interact with existing (and future) settings, including the projected degree of collaboration with and the anticipated role of teachers and in what kinds of environments the tool is appropriate and inappropriate.

Importantly, parallel considerations also hold for those funding and procuring AIEd technology. School administrators, school districts, regulators, etc., should be transparent about how exactly the technologies they procure are meant to interface with their educational systems and stakeholders. This, at a minimum, should open these decisions to transparency and accountability. Thus, if a developer’s tool deprofessionalizes teachers, or a school district uses technology to justify cutting class sizes, these decisions can more easily face public scrutiny and accountability.

Finally, journals, conferences, private and public funders, graduate programs, and commercial parties interested in economizing AIEd tools should emphasize social responsibility in research, development, and implementation. Such emphasis should be deeply integrated into processes and cultures, not offered superficially or nominally, as can be the case in computing and engineering education (Cech 2013; Leydens and Lucena 2018). For example, research fora should discuss the ‘corruptible potential’ of their technologies, mitigative strategies, and how they might engage positively with subjects, targets, and policymakers connected with the technology. And then, they should do so.

Conclusion

In Dede’s prognostications about the future of artificial intelligence in education 30 years ago, he offered that “the two most common errors in technology assessment are overestimating the speed of diffusion of an innovation and underestimating its eventual consequences and side effects” (Dede 1988). While predicting the future of educational technology often lends itself to uncertainty and extremes, a socially and historically grounded consideration of educational technology helps to reveal possible trajectories for AIEd. In particular, AIEd and ITS in particular promise (or threaten) to encroach on the roles of teachers and close perhaps the most prominent barriers keeping distance digital education from achieving its lofty goals—differentiation, and social–emotional engagement.

In this light, it is important to evaluate the implications of AIEd for our educational systems, including considerations of pedagogy, curricula, the role and possible automation of teachers, international development, ownership of educational choices, and behavioral manipulation. By reviewing these potential futures, I hope to promote critical reflection on possible developmental pathways of AIEd as it comes closer to mainstream implementation. Beyond that, I have offered preliminary strategies towards a more thoughtful engagement with AIEd’s future and how the AIEd community can exercise the social responsibility incumbent on it. Moreover, these considerations and strategies are relevant to other research and practice communities, such as those in AI, computing, and technology design generally. It is my hope that through sustained interdisciplinary dialogue and research, and continued reflection on our progress, we can realize the best of promising technologies like AIEd while avoiding the worst harms.