Designing research to inform sustainability and scalability of digital technology innovations.

Considerably more research needs to be done to understand how successful technological innovations and change processes are sustained and scaled to new learning contexts. Without a better understanding of successful technological innovation, the wider field and education community are unable to benefit and build capacity. A model is presented that explores how research can better inform sustainability and scalability in technological innovations. It is represented as four loops: Organization, Innovation, Research and New Contexts. Three international case studies of technological innovation are analyzed to demonstrate use of the framework and model. Results show that research can be designed to support sustainability and scalability, but that this needs to be balanced with other factors to support a successful technological innovation. Implications for supporting technological innovations are explored. © Association for Educational Communications and Technology 2021.

Introduction

In contemporary education and educational research there is growing attention being paid to digital innovations in new flexible learning environments, educational partnerships, informal and non-formal learning experiences – just to name a few. The integration of digital technologies in educational innovation can result in a range of fundamentally different ways learners and teachers’ access, engage with, and build knowledge. However, where good practices and successful implementations of technological innovations have been identified, understanding how these can be sustained within contexts and scaled to new contexts has proven difficult (Hubers 2020). It is important to understand it innovations have the potential for sustainability and can be scaled so other contexts can benefit.

To better understand technological innovations, ongoing research that can inform success over time and across a range of contexts is needed. Research approaches such as participatory and design-based research, streaming educational data, and rich observations can be designed to capture meaningful data across these various spaces, to begin to unravel some of the complexity of contemporary learning and digital technology use.

To support technological innovation and capacity in the field of education, research needs to be able to show if, under what circumstances, and how an innovation has been successfully sustained and scaled to new contexts. In the following discussion, we present an initial model to explore how research can contribute to and support digital innovation in education and inform ongoing sustainability and scalability. Moreover, the model can be used to support decision making about how research can inform change and innovation in an educational organization, such as a school or university. Using the model, an initial analysis of three technological innovations is presented. The main contribution of this study will be demonstrating use of the framework to address the issues of sustainability and scalability in research design, as an explicit component of innovations. Findings will inform how research approaches and methods can each attend to these issues in ways that are meaningful and relevant to educational contexts. Implications for educational change and research will be discussed.

Technological innovations in education

Technological innovation has had long and difficult relationship teaching and learning (Niederhauser and Lindstrom 2018). To begin to unpack this problem, it is first necessary to understand what we consider to be a 'technological innovation' in education. The Organisation for Economic Co-operation and Development (OECD) defines innovation as: “significant change in selected key practices in education” (OECD 2016, p. 8). In many cases, innovations are enacted to improve learning and/or learning processes. Change can be defined in three ways. First-order change is initiated by a teacher, such as an individual wanting to change their practice. Second-order change is at the school level. This could be in the form of a school-wide initiative to engage in blended learning. The final is a third-order change. This is change at a district or state-level, such as a major curriculum redesign (see Hubers 2020). In this discussion, we are specifically looking at technology-related innovations, but across all levels of change. A technological innovation is an innovation that is driven by or has a significant focus on use of digital technologies.

It has been difficult to understand if technological innovations are sustainable and scalable, in that research has not typically addressed these questions. Further, many of our traditional research methods are not necessarily able to capture change processes in technology-enhanced teaching and learning (e.g. Blikstein 2013; Cukulova and Luckin 2018). We argue that new approaches and methods of research are able to afford and support sustainability and scalability in technological innovation initiatives, through purposeful design of research and evaluation. Purposeful designs to address sustainable and scalable innovations, to collect empirical data on these issues, can then inform how they have succeeded, how to sustain those effects and how others can also enjoy these benefits.

Sustainability and scalability

Building on the work of Niederhauser et al. (2018), we define sustainability simply as the “ongoing change” of an innovation, which would continue in response to needs and intention of stakeholders. We define scalability from the point of: “dissemination of change across different contexts” (p. 508). On-going change may be sustained at a single level, such as a teacher changing their practice. Scalability may mean an innovation staying at one level, such as transferring a second-order technological innovation from school to school or scaling from a second-order school change to a third-order district-level initiative. Drawing on Coburn (2003) we consider scalability and sustainability to be related. Coburn points out that many studies of scale and sustainability separate the two dimensions, however for an innovation to be scalable it must first be sustained in a context. Innovations that are not sustained, either not adopted at all or not enduring in practice, are unlikely to be scalable or scaled. Therefore, they argue that sustainability is actually a dimension and requirement of scalability. Kampylis et al. (2013) have made the same argument, that given sustainability and scalability are so closely related, and often draw on the same change strategies, they can be treated as one construct.

In an effort to better understand the key factors of successfully sustained and scaled technology innovations, Kampylis et al. (2013) reviewed seven large-scale ICT-enabled innovations across Europe and Asia. A key finding of this study was the importance of context in understanding sustainability and scalability, at all levels of education innovation. With that in mind, they go on to state that regardless of where an innovation is initiated, to be successful it requires adaptation and change in 'factors, provisions and priorities' across a range of levels (Kampylis et al. 2013, p. 124). Supporting sustainability and scalability of new practices requires critical leadership support across levels; specifically, explicit 'top-down strategies to support bottom-up innovations' and measures in place to promote connectivity among stakeholders (p. 130). Research in the sustainability of school improvement initiatives found a similar pattern. School administrators did not necessarily play a direct role in innovations, but they provided the structures to support the innovation, such as promoting connectedness and supporting the creation of teacher networks within an innovation (Kampylis et al. 2013). Connectedness and the creation of networks were identified as being able to play an important role in supporting professional learning and connecting teacher communities of practice. Where these types of strategies are in place, sustainability is more probable, which can then lead on to scalability of practices.

To fully understand these mechanisms and their relation to sustainability and scalability, quality empirical evidence is needed (e.g. Kampylis et al. 2013). However, research actually addressing sustainability in educational change and innovation continues to be limited (Hubers 2020). For example, sustainability can only be addressed through longitudinal research, which provides the opportunity to capture experiences and change over time (see Hargreaves and Goodson 2006). However, in many cases change and innovation are only examined in the beginning of an innovation or initiative, which does not actually capture how new practices are adopted and implemented over time.

The role of research and 'new' approaches

Broadly across education innovation and change, ‘one of the most important knowledge gaps related to sustainability is insight into both the individual and collective learning processes that are required to successfully implement change’ (Hubers 2020, p. 9). This is particularly important in technological innovation, considering that technology use can lead to new pedagogies and new types of learning experiences, that the rapid rate of changes in technology and innovation can be resource intensive. There is a significant need in the field for methodological approaches that are able to understand change processes, the outcomes and sustainability of those changes. In this discussion, we focus on two specific examples to demonstrate some of the affordances of participatory research and the use of new technologies in data collection to serve this purpose. The combination of these two approaches is able to support the longitudinal research needed to study sustainability and be flexible enough to inform and include new and different contexts.

Participatory research

We first address forms of participatory research as an approach to specifically understanding sustainability and scalability. We consider forms of participation research that range from design-based research (e.g. McKenney and Reeves 2012) to action research (e.g. Kemmis et al. 2014). These types of methodologies place participants and stakeholders at the front of research and evaluation of their own context. In placing participants in a position of power in innovation and research, it is likely to promote a 'bottom-up' and collaborative approach to innovation, which is more likely to be sustained in an organization if identified and developed from a genuine need in the context (Kampylis et al. 2013; Bellei et al. 2020). Given the complexity of technology-related educational change and innovation, a participatory approach has the flexibility to be responsive to individual contexts to account for variation in experience, resources, priorities and practices that are available and meaningful to participants (e.g. Phelps and Graham 2010).

Longitudinal research

Second, while participatory research is able to engage practitioners and other stakeholders, it is still necessary to design research to be longitudinal to address sustainability (Hargreaves and Goodson 2006), and to identify the necessary data to address practitioner's needs. Sustainability change is a longitudinal process, which begins when change is contemplated (Hubers 2020), extends through the life of an innovation and continues into the scaling of practices. New technologies offer affordances that may more fully support longitudinal research into sustainability (e.g. Blikstein 2013). An example of new technologies applied in research is Davis et al.'s (2017) year-long study of students in makerspaces. They developed an automated method to code video observations of students solving puzzles, which allowed students' problem-solving processes to be revealed visually over time. Using automated processes reduces time and resource costs associated with longitudinal research. This type of new approach, embedded in innovation, has the potential to capture change processes and outcomes.

New approaches are only useful if aligned with teaching and learning goals (Bellei et al. 2020). It is essential that more appropriate research is conducted across all levels of innovation and the role of this research in understanding sustainability and scalability is clear. To further this effort, we present an initial framework to illustrate the role of research in technological innovation sustainability and scalability.

An initial model

The model presented here draws on systems thinking, in that it is characterized by having definable factors and interactions. Specifically, we use an adapted causal loop model. A causal loop model is used to demonstrate key relationships in a system, where ‘loops’ represent components of a system. Arrows between the loops represent an effect or change occurring, where one component of the system is causing a change in another. This results in a dynamic relationship between the components. In the following model, the ‘loops’ representing components of a system of technology innovation in education and each includes a set of factors. The four main loops proposed are: Organization (L1), Innovation (L2), Research (L3), New contexts (L4; see Fig. 1).

Fig. 1

Framework for supporting sustainability and scalability in innovation research

Research has shown that there are dynamic relationships between educational organization, technology innovations, research and new contexts (e.g. OECD 2016). We draw on this insight to understand how the four components of the initial model would be expected to interact and contribute to sustainability and scalability of innovations.

We see Organization (L1) at the core of this dynamic model, as it begins with leadership. It is envisioned that technological innovation begins with a decision to change at an educational organization, whether that be at the first-order level of practitioner, second-order school/organization level or even at the third-order system level. Therefore, it is placed at the centre of the model.

Understanding the context of educational change and innovation is critical in sustaining change (Kampylis et al. 2013; Fullan 2015; Hubers 2020). It influences how research is conducted, innovation implementation and how innovations are adapted to new contexts. Moreover, leadership's knowledge of technology innovation will significantly affect the success of innovation and change in practice (Dexter 2018). Leadership can also engage stakeholders to foster a culture of change and create communication channels (Howard et al. 2018), which all contribute to participants' collaboration in and sustainability of the innovation. The most effective leadership model to support educational change and innovation is a bottom-up approach supported by and aligned with top-down policies (Howard et al. 2018).

The second component is the Innovation (L2). The Innovation is conceptualized as the expected or desired technology-related educational change. Currently, digital technologies are at the heart of most educational change and innovation, in some capacity (Kampylis et al. 2013). It has been difficult to understand why an innovation is successful or unsuccessful. To be successful, innovations need to have a clear purpose, the new practices/expected change and the use of digital technologies need to be valued by practitioners and specifically relate to learning. An essential component of sustaining change is the provision of professional learning, built into the innovation (Hubers 2020). From the sustainability of new practices and educational change, the scalability of the innovation can be considered (Coburn 2003).

The third component is the Research (L3), which is the main contribution of this model. Currently, there is a lack of research looking specifically at sustainability and scalability in technological innovation in education – and in education more generally (Hubers 2020). Importantly, to be meaningful and effective, this research needs to be led by participants and co-constructed in collaboration with researchers, to draw on both bodies of expertise and knowledge (Kemmis et al. 2014). A range of research methods and approaches need to be used to support longitudinal research designs and to capture the complexity of educational change, to be able to inform sustainability (Hargreaves and Goodson 2006). Research outputs and dissemination need to be aimed at practitioners (McKenney and Schunn 2018) and able to show a strong connection to learning, to support sustainability and scalability.

The final component is New Contexts (L4), which is essentially scaling an innovation after the change in practice has been sustained. This may be a different teacher adapting use of a new technology in their classroom, or it may be the implementation of a whole-school initiative to increase the use of authentic tasks in learning. This may be a teacher to teacher first-order change, or it may be a teacher change that is being scaled up to be school-wide or at a system level (Hubers 2020). Importantly, scaling an innovation from one context must include important considerations of fidelity to the innovation and flexibility in local adaptation (Kampylis et al. 2013; Anderson 2017). This negotiation is rooted in the organizational context and will strongly affect professional learning and other support structures of the innovation.

The aim of this study is to explore the use of the model to better understand the role of research in sustaining and scaling technological innovations in education. The main research questions addressed in the following analysis was: What were the key factors of research relating to sustainability and scalability of technological innovations?

Approach and methods

The model

The model comprises key components of technological innovations, embedded research and implications for sustainability and scaling. The current focus of research is applying the model to further define the factors, consider high-level interactions and identify directions for further development. Models such as this can be used to aid decision-making by providing ways to hypothesize choices, compare possible outcomes and examine the effects of changes in a system (Martinez-Moyano and Richardson 2013). They have proven to be effective in understanding educational change and innovation (e.g. Hirsch et al. 2007). The model and individual factors were identified and refined by a thematic working group of educational technology experts, as part of EDUsummIT 2019, held in Quebec, Canada. The group specifically examined the nature of sustainability and scalability in technological innovations. The initial discussions and outline of this work can be found in the EDUsummIT 2019 book (see Howard et al. 2020). The model was then refined based on current literature in the area of educational change, school effectiveness and improvement, and sustainability and scalability of educational innovations.

Analysis

To build cases for analysis, the three lead researchers were provided with the model to guide reporting on their case. To do this, they drew on peer reviewed publications from the cases, reports and their own professional experience. They were instructed to use the model as a guide, but to include any and all description about the case that they felt it was relevant to understanding why it was successful, why aspects were sustained and how scaling occurred. The cases were then theoretically analyzed by a fourth researcher, using key factors from the framework as a coding structure (Flick 2014). All three cases were coded by one researcher and then reviewed by two additional researchers for agreement and reliability. Coding was negotiated until full agreement was reached among all three.

Results from the three cases

In the following sections, three case studies of technological innovations are presented. The cases are drawn from three different countries and all address different types of innovations: teacher training, educational experiences and teachers' integrated practice. Each represents a different technological innovation, which in combination with organizational/contextual factors resulting in implications for sustainability and scalability.

Case #1, Netherlands

Pilot project addressing the professionalization of preservice teachers in technology integration

The organization (L1)

Teacher Education Institutes in the Netherlands face an enormous challenge to prepare their students for teaching in the digital age. Despite the fact that the new generation of prospective teachers is familiar with using technology for private purposes, many of them do not have the basic technology skills needed for becoming competent in using technology for educational purposes (e.g. Tondeur et al. 2012). The three-year long survey showed that between 60% and 70% of the freshman reported that they lacked basic technological skills, as measured by the Technology Knowledge scale (see Schmidt et al. 2009).

The Teacher Education College in this case has four departments: Primary Education, Secondary Education, Vocational Education (all bachelor level) and Graduate Courses (master level). The research groups in the college are more or less independent but are administratively situated immediately under the College Director. The research group Educational Innovation and ICT has as its main aim to conduct applied research in collaboration with and relevant to faculty of the College. The joint goal of the research group and the leadership is to contribute to relevant and useful educational innovations supported by technology.

Innovation (L2) and research (L3)

Design-based research (McKenney and Reeves 2012) guided the design and research of the interventions, which results in a close relationship between the innovation and research. Because of concerns about the preparation of future teachers to use technology from technology support staff and the leadership of the College, the research group Educational Innovation and ICT was asked to survey student teachers’ knowledge, skills and attitudes towards technology. The three-year long survey showed that between 60% and 70% of the freshman reported that they lack basic technological skills. A conclusion of this study was that providing students with useful experiences in technology integration would foster their technology integration skills (Farjon et al. 2019). In close collaboration the faculty and research group decided to implement two practically relevant and feasible interventions, to provide student teachers with experiences of technology use in teaching to foster technology integration competencies. Theoretically the interventions were grounded in the Technology Pedagogy and Content Knowledge (TPACK) framework (Koehler and Mishra 2008), pedagogical reasoning about technology, (e.g. Heitink et al. 2016), affordances of technology (e.g. Hunter 2015) and the Synthesis of Qualitative Data (SQD) model (Tondeur et al. 2012). The first intervention aimed to develop student teachers’ TPACK in an undergraduate English literature course, for future English language teachers. The second intervention was conducted in a graduate-level technology integration course for practicing teachers.

From existing practice to sustainable new practice

The first intervention took place within the Secondary Education Department and was an initiative of the research group and the English language teacher educators. In this intervention the development of student teachers’ TPACK in an English literature course consisted of lectures, book analysis and discussion. The class was compulsory for third year undergraduate teacher education students preparing to become English language teachers. The teacher educators leading this class noticed that many students copied summaries from the Internet instead of reading the literature. The teacher educators and the research group teamed up to design and investigate an intervention aimed to overcome this problem and to contribute to student teachers’ TPACK in a series of iterations (2016–2019). Students had to produce multimedia blogs, digital magazines, gamebooks, animations, trailers, digital stories (they could choose the technology of their own choice). Research outputs: artefact analysis of student products, observations of the lessons, interviews with students, guided subsequent iterations of the intervention: teacher educator modelling of technological pedagogical reasoning to address specific affordances of technology, collaborative production of digital artefacts and finally practicing technological pedagogical reasoning and action during internships to provide students with authentic practice. Research results showed that student teachers’ TPACK improved, as did their pedagogical reasoning. However, the will to use technology and pedagogical reasoning about technology depended on beliefs about education (Smits et al. 2019a). The final intervention is now a sustainable practice, in that the task to produce a multimedia product has been retained as part of the course.

In the second intervention, the technology integration course for graduate students arose from a lack of attention to technology use in the master Educational Needs and increased worries about technology use in Dutch schools (Smits and van Koeven 2018; van Dongen and Voogt 2019). Therefore, the department head suggested that the research group together with faculty design a technology integration course. This course aimed to develop the technology integration skills of graduate students, who were typically practicing teachers. It was designed and improved between 2016 and 2018, as an optional course as in collaboration with teacher educators from the Master program and the research group. Graduate students become acquainted with core models for learning and technology integration, observe technology use of colleagues, identify and discuss good practices, and design and enact technology-rich lessons. The accompanying research included a student questionnaire, an artefact collection and reflective interviews with students. Results showed that at the end of the course most students produced lessons with a rich quality of technology integration taking advantage of a (sufficiently) rich array of technological affordances. The course has been a sustainable practice and compulsory for all master students since 2019 (Smits et al. 2019b).

The vulnerable road from sustainability to scalability

Both practices were considered model practices to address technology integration in the Teacher Education College. Sustainability of the practices prompted the initial stakeholders—the research group and the collaborating teacher educators—to think about how to scale what we had learned in both practices, rather than replicate the practices, to other subject areas within the secondary education departments and to the other departments within the College. The aim of scaling up lessons learned from the practices was to provide student teachers with a range of technology integration experiences during their program relevant to the subject domains (e.g. STEM, social studies etc.) they would be expected to teach. It was realized that scaling the interventions as such would not be possible, as many teacher educators in the college often expressed that they do not feel comfortable with using technology in their teaching practice

New contexts (L4)

In response to this, a plan was developed in early 2019 with two distinct but related elements: (1) to support the discussion on relevant technology use in the various (discipline-based) teams of teacher educators and (2) to foster the technology integration skills of teacher educators by creating teams of experienced and non-experienced teacher educators in technology use. During 2019 the plan was extensively discussed with the leadership. It was well received, considered affordable and it fit well with innovation ambitions of the college. Also, teacher educators were positive about it. However, these positive signals never led to concrete decisions about the implementation of the plan by the leadership. Unclear decision structures in the College resulted in leadership not arriving at a final decision about the plan. When the College was faced with extra budget cuts the plan was then easily dropped, as primary tasks were prioritized. This situation changed significantly when all universities in the Netherlands closed mid-March 2020 because of COVID-19. Teaching became online and support to do so was offered to the teacher educators. Although we have to wait to understand what this means for the quality of technology use in the long run, it shows that barriers in using technology can be easily overcome during such a disruptive change.

Case #2, United States

The sustainability and scalability of a grassroots technology-related multi-school technology-change initiative, in both urban and rural contexts

Beginning in 2013, a large group of organizations began a joint effort to improve educational experiences in their region, within the United States. These included urban and regional school districts, non-profit organizations, foundations, and educational entities who shared a common goal to help create and support innovative educational learning perspectives.

The innovation (L2)

Operating within the broad aim to ‘Remake Learning,’ leaders specifically came together to promote the concept of ‘Spaces for Active Learning.’ The specific innovation was the creation of a network of educational and community organizations to support a combination of small grants, collaborative interactions, and top-down/bottom up structures, to support a broad set of innovations related to active learning. Each educational organization receiving funds was expected to identify and enact technology-related plans relevant to its population, educational needs, overarching goals, and individual contexts. The key underlying pedagogic approach was the integration of authentic, project-based activities into educational contexts, bringing together long promoted ideas that support learners creating, developing, and inventing (Hatch 2013). These concepts include Papert’s (1986) theory of constructionism, in which learning is facilitated through the construction and sharing of physical artefacts. A shared focus was on moving instruction from teacher driven to authentic, learner driven, and hands-on activities (Costa et al. 2017).

Research (L3)

Within the larger Remake Learning innovation, individual projects employed a modified design-based research approach (McKenney and Reeves 2012) to understand their own progress, which was reviewed and modified throughout the life of each program. At times, projects included researchers conducting observations in authentic settings; other data were collected through leaders or educators’ self-reports and project meeting notes. An overarching mixed methods action research project was conducted to collect data across the Remake Learning initiative (see Freeman et al. 2017). This primarily took the form of a set of embedded cases, within a combined case study participatory design (Scholz and Tietje 2002; Cooper et al. 2015), and included multiple methods of data collection and dissemination across the different project implementations.

From existing practice to sustainable new practices

Each individual school and/or educational organization had the opportunity to design its own programs and was responsible to ensure its continuation and growth. In many cases this resulted in the development of makerspaces and integrating components of engineering, computational thinking, coding and robotics into curriculum. Innovations were locally designed, so there was a wide range of projects occurring across the network. Libraries, museums, and out of school programs were part of the program and could partner with schools on projects. These entities serve citizens of the region and all were intended to benefit by sharing best practices and lessons learned. Continuous robust professional development was required as part of every project, across the network, with educators treated as professionals who were encouraged to follow their personal interests.

Sustaining these activities required ongoing funding sources, which was supported by the Remake Learning initiative. The support and leadership of this overarching structure was precisely why the implementations were successful and have continued to evolve, grow and spread (e.g. Bellei et al. 2020). It also required strong support from leaders in each organization, and leaders networked to share experiences and best practice. Learner engagement also provided great insight into which projects have been the most successful and most likely to be sustained. Educators and families reported evidence of learners’ engagement when curriculum included choice, active learning, and participation related to time in the Spaces for Active Learning (Hohlfeld et al. 2017). Students, in and out of school, articulated the impact of the opportunities as influencing their options for careers or lifetime interests. For example, in one rural area, learners explained their year-long project on waste management led to creating recycling efforts that solved an authentic problem in their community, which led to the manufacturing of recycled materials into objects for use in new products. Students reported that this experience introduced them to future work and learning possibilities in engineering that they previously would not have considered.

From sustainability to scalability

Key positive impacts in the system that contributed to sustainability of the program, individual innovations and scalability of practices include: recognition of cultural and contextual identity, strong leadership, commitment to collaboration, and engagement with a broad range of community and regional organizations. The Remake Learning initiative began with a few small implementations in 2013 and expanded into an ongoing regional network of learning organizations. This is probably the strongest representation of a sustained initiative, which has continually scaled to incorporate additional educational organizations and projects. Collaboration with colleges, universities, businesses, museums, and other similar organizations were fostered within this project. Moreover, links were created with local business and industry leaders and they were encouraged to spend time in schools. Importantly, the sustainability of this initiative is not necessarily in the individual projects, but in the wider Remake Learning network. The growth of this network provides access to and support to apply for continuous funding from: the foundations, businesses, regional service centres, state priorities, and universities, which are all committed to funding local initiatives. Additionally, this builds capacity for individual projects to seek further funding from national or state sources.

The goal of the Remake Learning network was that organizations learned from each other and then designed projects that responded to their own contexts, while having the support of network experience and knowledge to be able to sustain any changes implemented. One small rural school began by its leaders and technology coordinator attending meetings with others already involved in implementing “spaces for active learning.” Drawing on network expertise and previous successful projects, they began small by creating a robotics club and then a shop class with an engineering curriculum. This was followed by adding a television and journalism program and a wide variety of other credit and elective courses. The success of each project encouraged the development of others. The school leaders began to help other schools rethink their programs and offer advice on best practices. Also, the network included ex-school leaders to mentor new educational organizations into the initiative and how to gain funding. These strategies were implemented in ways that focused on change and learning rather than fidelity to a particular project implementation, which is a more effective approach to scaling (see Hubers 2020).

Case #3, Australia

Developing sustainable and scalable data-informed approaches to support technology-related classroom practice

With increased availability of digital data through new technologies, such as web enabled cameras and eye gaze tracking, it would seem possible to develop a better understanding of technology integration in teaching and learning. However, even though a wide range of data is available, understanding the complexity of technology integration in classroom teaching and learning has continued to be difficult (Howard et al. 2018). The technological innovation presented in this case study broadly addressed how digital technologies were used in teaching and learning, in the classroom. To do this, participatory research was designed to capture educational data in the live classroom to inform learning. Researchers worked in close collaboration with a secondary school in Sydney to develop the research and test new research approaches using digital technologies. The methods developed were initially developed through work at a single school. Plans for scaling up to a larger group of schools in the New South Wales (NSW) Department of Education were underway when the COVID-19 outbreak occurred and ceased all school-based research.

The initiative to improve educational technology integration was a grassroots initiative driven by three teachers participating in the original study, one from Science and two from History. All three had worked with the researchers previously on a different observation data collection. They wanted to continue with classroom observations and focus on improving how they used digital technologies, such as Microsoft OneNote as a class resource and electronic workbook, in their practice. They were keen to participate in long-term observations of their practice to understand how digital technologies were used in their classrooms over time. The school leadership was very supportive of research and teachers continuing the classroom observations, in whatever capacity they wished. Digital technology innovations in their teaching practice were driven by the teachers and were loosely aligned with the school's overarching aim to provide high quality teaching to the students and to increase technology integration. The teachers decided to use the research as part of their professional learning, specifically how to support students' learning processes through digital technologies.

The research (L3)

An incredible amount of educational data is continually collected by education departments, institutions and digital technology providers. However, identifying meaningful uses of educational data to inform teachers' practice has proven difficult, particularly technology integration. The researchers have argued that for data to be meaningful to stakeholders it must be close to the practice being studied and/or changed, such as capturing full learning and teaching processes and practices (Howard et al. 2019). To identify how to do this, stakeholders need to be involved in the research design and data collection process to ensure what is captured will be representative of what they wish to change. The aim is to use educational data to inform technology-enhanced learning, in ways that can be meaningfully reported and visualized to support educators (Merceron et al. 2015).

The research began with collecting continuous video and audio data in the classroom, looking at speaking patterns and the teachers' movement during lecture and small group tasks using laptops (see Howard et al. 2019). Observations were conducted over weeks using a web-enabled observation kit that can be controlled through an online interface. The system does not require any resources after being installed and observations can continue indefinitely. The teachers wanted to know more about how Microsoft OneNote was being used by their students and interaction occurring in the physical space at the same time, such as student discussions or soliciting support from the teacher. Separately, methods of analysis of streamed educational data developed in another project were being adapted for this initiative. This would require user-profile log data. All parents and students had given their permission for log data to be used in research. However, to access streaming educational data it was necessary to engage with the Department of Education, as this type of educational data was held centrally in NSW.

Analysis of the physical classroom movements and speaking patterns when using digital technologies were developed in consultation with teachers. Some findings resulted in teachers changing their practice when students were using the laptops to improve classroom arrangements, methods of group questioning, providing student support and their own lecture style. They were able to link these changes to their own desired pedagogy, e.g. more student-centred questioning. School leadership was also given the analysis for their own interpretation, which they discussed with the teachers as part of their professional learning.

The new practice of observing technology-enhanced learning and using the outputs to change classroom practice has been sustained in the school for the last four years, in that the teachers have continued to work with the researchers to collect data on their practice to observe how students experience and engage in new digital technology integrated lessons, such as a new unit on pulleys in Physics. Teachers have given feedback on visualizations and the relation of data to their practice, and they have shared visualizations with their students for feedback. In each year the researchers and teachers negotiate what will be observed, what they would like to know about in their teaching. Observations, collaborative analysis and development of data collection methods are ongoing.

In 2019, discussions started with the school and the NSW Department of Education to scale the classroom observation methods to other individual schools and to groups of schools across the state. As a second pilot study, scaling the classroom observation to another school context was planned. A primary school using a flexible learning space set up was selected, where the teacher wanted to answer similar questions about how students were using laptops. Specifically, how different groups of students, e.g. gifted and talented, students who likely to be distracted, used chromebooks in lessons. Observations were delayed to the last term of 2020 because of the COVID-19 pandemic.

The NSW Department of Education wanted to explore how educational data can be used to inform teacher's technology-related professional and school development. It was determined that classroom observation data of teaching practice and learning processes would be triangulated with department log data, to provide context for patterns observed in Microsoft OneNote data. The Department wanted to trial the approach in 20 schools to inform School Improvement Plans, the new state-level Digital Transformation Project and to support an incoming Digital Maturity Framework which was planned for implementation in 2020. However, while close collaboration with one or two schools is not resource intensive, setting up the same relationship with 20 schools would require considerable time and resources, which makes scaling up to a system level difficult. However, with the COVID-19 pandemic this project is currently on hold until 2021.

The model and discussion

The aim of this study was to explore the use of the model to better understand the role of research in sustaining and scaling technological innovations in education. Specifically, the study aimed to explore the question of which key factors of the research are related to sustainability and scalability of technological innovations. All three case studies were sustained and scaled, to a degree. The main finding of this study has been that the model is able to provide a way to explore the role of research in technological innovation and support comparison among cases. The results are also able to indicate where the use of new research approaches may have had an effect on sustainability and scalability.

Below, initial results exploring the role of research in each of the three cases has been mapped to the framework (see Figs. 2, 3, 4). It is important to keep in mind that this analysis is based on what was reported by the researchers in the case descriptions. In the figures, the key factors in each loop that are strongly present are dark, while those that are weakly represented are greyed out. The arrows showing progress in sustainability and scalability by design, the arrow is solid green. If progress was not by design, the arrow is dashed. If progress was not observed, the arrow is solid red. The figures are followed by a comparison of the three cases.

Fig. 2

Netherlands case analysis

Fig. 3

United States case analysis

Fig. 4

Australian case analysis

In all three cases, there was a high level of collaboration among participants and researchers, strong leadership and the innovation was supported through policy, strategic goals and/or funds. (L1). The technological Innovations (L2), which focused more on learning and teaching than technology use, were valued and had strong relations to learning. These positive contextual factors significantly contributed to success of the innovations, which is particularly important to develop positive attitudes towards technological innovation (Dexter 2018). The approaches to research varied widely. The Netherlands and Australia were both strongly participatory and focused on the innovation, while the approach to research in the United States case was specific to each project. In terms of sustainability and scalability, the Netherlands case (see Fig. 2) did show progress, which may have been an affordance of the iterative design-based research. The iterations improved the innovation and built a sense of ownership in the teacher educator; however, this needs to be validated. Sustainability was a result of the success of the innovations and leadership support. However, there were decision making issues within the leadership, which delayed scaling up of the innovations. If empirical research informing scaling had been available, rather than anecdotal information, this process may have been improved (Jamaludin and Hung 2016). Given that scaling was also delayed because of teacher trainers' knowledge of technology integration, clear understanding of the change processes related to integrating the new innovations in courses, could have supported these individuals and improved their confidence (Kampylis et al. 2013).

The United States case was very successful. It has been sustained and scaled through the network; but, this was not necessarily related to the research (see Fig. 3). It was reported that the research was largely focused on progress of the projects and fulfilling obligations of the various funding agencies. Co-construction with researchers or collaboration are not clearly identified. Practices were more likely to be sustained and scaled through leadership and teacher communication networks (e.g. Howard et al. 2018), than by research. However, the projects did not seem to scale up from second-order change to third-order, system change. They stayed at the level of individual schools and partnerships with industry partners. If the research had been designed to explore the change processes and better understand the sustainability, successful practices could have benefited a wider range of schools and moved beyond the immediate context. This is particularly important given the wide array of projects and the embeddedness of digital technologies in the learning spaces, to be able to extract how the use of technology related to student experiences, project design and learning.

The Australian case was slightly different (see Fig. 4), in that the innovation was tightly integrated with the research design. The aim of the research design was to support sustainability and scale up to include more schools, as the methods were refined. Affordances of the new approaches were able to support long-term research, e.g. observing the classrooms over four years, which supported sustainability. However, the actual innovation and changes in practice; how the teachers used the observations classroom data to inform their practice was not scaled. This becomes apparent in the absence of professional learning built into the innovation, in that the teachers were adapting their practice as they wished. These processes were not captured in the research or in the organization. Therefore, while collecting data was sustainable and scalable, the resulting changes in teachers' technology integration were not necessarily being sustained or able to be scaled to other contexts. This could have been improved by including data collection points, such as interviews and questionnaires, to better understand and document how teachers implemented educational data to inform their technology integration.

The above analysis demonstrates use of the framework to understand the role of research, its relation to key factors, sustainability and scalability, and provides a way to hypothesize possible points of improvement in sustainability and scalability. There is clearly much more work to be done validating the key factors and understanding their relationship to research. In particular, results suggest a productive association among research, professional learning and leadership (Voogt et al. 2016). Findings suggest that purposeful design of research can inform sustainability and scalability, but it is well documented that professional learning and leadership are critical components of sustaining educational change and innovation (Dexter 2018; Christensen et al. 2018), particularly technology integration which can often strongly affect how learning occurs and even result in new learning. The combination of these factors needs to be better understood to be able to draw on the strengths of all three to improve technological innovation and capture change processes. Without research, leadership processes, change processes and professional learning are left opaque and cannot be used to build capacity in other organizations (Cukulova and Luckin 2018).

Future research and conclusions

It is necessary that successful technological innovations and associated change processes are understood better, as digital technologies underpin most contemporary learning. Ultimately, the model has introduced more questions than provided answers. As this is only an initial testing of the model, it is apparent that considerable work is needed to fully validate the key factors and relationships among the loops. With that said, the current version has been able to unpack each case study, provide a basis for investigation into technological innovation and compare the three. This, to a point, does validate the use of the framework and model at a tool to consider the role of research.

Future research to develop the model will initially take two forms. The first will be applying it to a wider range of cases to validate the key factors and relationships among these, sustainability and scalability. It is also necessary to explore the possibility of reciprocal/two-way relationships among factors and research. This is suggested in the Netherlands case, and is an important aspect of understanding affordances of the research designs and their effects in innovations. In that the three cases used here also played a role in the development of the framework and model, there is bias in the key factors and ways that change processes are understood. Additional researchers, not connected with the work, need to be asked to apply the framework to their own cases. The second step would be to then conduct research more deeply investigating and tracking how aspects of the research, supported by new approaches to data collection, interact with the loops and key factors. In particular to better understand the balance between research and leadership. While the Australian case shows research can support sustainability and scalability, it is not enough for the innovation to be successful. Leadership is an incredibly strong factor in the success of an innovation, particularly technological innovations and their understanding of technology, that it can sustain and scale practices without research. However, as observed in the results, this leaves processes relatively opaque and does not contribute to the wider field. This is likely a role that well-designed research to understand technology change and innovation can play, to communicate and support these processes in conjunction with good leadership.

In conclusion, by improving research associated with technological innovation more organizations would be able to benefit from successful practices and change processes. Given the highly variable nature of technology integration, across education, it is essential that the field develops a better understanding of successful innovations that support context and learning. Clearly research can be designed to support sustainability and scalability, but other contextual elements are needed for technological innovations to be sustained and adapted to new contexts. The framework and the model provide a starting point for making decisions about this work, when innovations are being conceptualized, as they progress and how they are evaluated at the end.