Evaluating Digital Instructional Materials for K-12 Online and Blended Learning.

With the large increase in online instruction, including remote instruction with online materials during the COVID-19 pandemic, there also was an increase in the use of instructional materials that were made to be displayed online or were digitized for online use. However, teachers have not had access to guidance about how to select and evaluate online instructional materials for classroom use. The lack of guidance has the potential to harm historically excluded populations of students and could frustrate teachers as they learn to teach with digital materials. The purpose of this paper is to share the 4A Framework for evaluating online instructional materials. The framework is organized around the premise that quality online instructional materials are accessible, promote active engagement, advocate for inclusion, and are accountable for their relationships to standards and data privacy. Each feature is discussed and examples of teacher work in applying the framework are shared. © Association for Educational Communications & Technology 2021.

Instructional materials that are developed for and hosted in online spaces are distinct from traditional materials (Choppin & Borys, 2017). Digital instructional materials have the potential for increased individual interactivity (Choppin & Borys, 2017). Also, digital instructional materials contain a broader array of adaptivity and personalization features, and they have more potential to be accessible to students with disabilities (Choppin & Borys, 2017). Finally, digital instructional materials have a greater occurrence of built-in assessment programs (Choppin & Borys, 2017).

The mere fact of being a digital material does not guarantee these perceived benefits (Greer et al., 2014; Rice, 2018). Even so, teachers are expected to use these digital materials in teaching, even when most instruction occurs in person (Kalolo, 2019; Selwyn, 2007; United States [U.S.] Department of Education, 2016). With the increased pressure to acquire and use them, digital and digitized instructional materials are a rapidly expanding market within the educational technologies sector. This market is expected to grow from $85.5 billion (USD) in 2020 to $181.3 billion in 2025 (Markets Insider, 2020). Examples of digital instructional materials include videos, content presentations, animations, educational games, online articles, and educational materials from traditional print texts that have been scanned and uploaded. The increased awareness and availability of such materials was likely aided by the onset of the COVID-19 pandemic, which closed school buildings in the United States and abroad (Education Week, 2020; UNESCO, 2020). During this time, remote online learning replaced in-person learning in many schools. Teachers who may have been infrequent or non-users of digital instructional materials suddenly had to gather and use them for their lessons.

As students and teachers return to in-person learning, the educational technology sector is unlikely to relinquish their profits or decrease their product offerings. Some schools will continue to use fully online learning, but many more will use forms of blended learning—where instruction can be provided through both traditional and online modalities in ratios that meet the needs of the students and match the resources in the school (see Martin et al., 2020).

The surge in popularity and availability of online educational resources and the necessity to use them during the pandemic required teachers to make quick decisions about what digital instructional materials to use to teach their students (Gallagher et al., 2019). Selecting materials off the internet simply because they are inexpensive or even free is unlikely to result in materials that embody the benefits of using digital materials (Reidenberg & Schaub, 2018). Moreover, it could encourage content developers and vendors to make materials that meet corporate ends without considering what teachers and students need to have educative experiences together. Using digital materials with deliberation and agency requires teachers to ask very basic but very important questions, such as what materials will I teach with? And why these materials?

Before the pandemic, much growth in digital learning came from traditional public schools who had technology integration initiatives (Digital Content Goes to School, 2016; Digital Learning Collaborative, 2019). In many cases, teachers made decisions about digital materials for online teaching with little to no guidance from initial teacher preparation programs or professional learning (Crouse et al., 2018; Gallagher et al., 2019). Teachers in these studies described the stress involved in not knowing how to find digital instructional materials or make them (Kundu & Rice, 2019; Kundu et al., 2021). In such contexts, teachers made decisions based on a few factors such as compatibility with current software, the perceived technological simplicity of the materials, or faith in a recommendation from a colleague (Fang et al., 2010). While none of those reasons are inherently problematic, teachers deserve stronger guidance for informing their decision-making about the quality of instructional materials. Moreover, this guidance should be efficacious without being too complex to remember and apply (Cheon & Grant, 2008; Chiu & Churchill, 2015; Kim et al., 2017).

In response to the need for practical guidance about choosing instructional materials, the authors of this paper developed a conceptual tool based on implications from research and policy guidance for high quality digital instructional materials in K-12 settings. This tool is called the 4A Online Instructional Material Evaluation Framework. It is abbreviated in this document as the 4A Framework (see Fig. 1). The 4A Framework consists of four elements:

Fig. 1

4A Framework for evaluating digital instructional materials

Accessibility addresses whether users can open, view, and interact with the materials according to applicable laws.

Active Engagement addresses whether learners are invited to invest effort into learning from behavioral, cognitive, and emotional perspectives.

Advocacy for Inclusion addresses whether diverse peoples are represented with contextual nuance, compassion, and respect.

Accountability addresses whether the materials share their origins and purposes, are based on standards or principles, and are open about personal information and user data collection processes.

The Development of the 4A Framework

The 4A Framework was built using principles of generalized invariance structure theory (GIST; Vigo, 2013). Humans organize concepts by determining a minimal number of useful but distinct rules they can apply across an array of situations (Feldman, 2000). In GIST theory, conceptual understandings emerge as qualitative patterns are detected and judged to resonate with humans based on how they have made sense of their experiences. When new knowledge is encountered, it can be applied to the present framework and placed in an existing category or determined to belong to a new one.

Use of the GIST Process to Develop the 4A Framework

To make a qualitative framework of ‘rules’ for digital instructional materials, we, the authors of this paper located information about what constituted quality in digital instructional materials in K-12 online and blended learning settings. This included research studies in K-12 settings where digital instructional materials were being used, the U.S. Section 508 law, and the Web Content Accessibility Guidelines by the Web Accessibility Initiative of the World Wide Web Consortium (2020). Then, we made lists of features of quality instructional materials based on that literature. Next, we used the lists to organize the items into groups based on conceptual matching (see Table 1). We alternated roles of organizing items and giving each other feedback, striving to reduce the number of broad categories until we had the smallest number possible with minimal overlap (Watkins, 2017). This left four categories: Accessibility, Active Engagement, Advocacy for Inclusion, and Accountability.

Table 1

Key support for 4A framework

4A Framework Element	Author/Year	Key Ideas
Accessibility
	Behling (2017)	Decrease access points and task steps to increase accessibility (e.g., logging on, finding assignment, submitting assignment)
	de Witte et al. (2018)	Assistive technologies should be affordable and available to students who will benefit from them
	Edyburn (2004)	Assistive technologies include text-to-speech and speech-to-text, but the definition of assistive technologies is wide
	Federal Communications Commission (2021)	Under U.S. law, electronic devices and technologies must be accessible
	Information Technology Industry Council (2020)	Digital devices and technologies should be easy to obtain and use
	Rice (2017)	Teachers should evaluate text complexity in online instructional materials and add support for comprehending complex text
	Rice (2018)	Accessibly evaluation requires more than relying on the accuracy of Voluntary Product Accessibility tables; it requires information from actual users with disabilities
	Rice and Deshler (2018)	Teachers should provide vocabulary support for online texts even when definitions are available by ‘hovering’ or ‘clicking’
	Rose (2018)	Students can and do make complaints to the U.S. Office of Civil Rights about online instructional materials when they are not accessible
	[U. S.] Office of Civil Rights (2011)	Technology-based learning opportunities should be available with equal ease to those with and without disabilities
	W3C (2020)	Accessible content is perceivable, operable, understandable, and robust
Active Engagement
	Rice et al. (2019)	Parents locate links to digital instructional materials, including activities and videos to promote engagement
	Boettcher and Conrad (2016)	Teachers need to be proactive about monitoring the completion of online work and providing engaging materials
	Borup et al. (2020)	Online engagement is an ecology of various types of engagement with many actors, rather than an individual psychological activity
	California State University Center for Distributed Learning (1997)	Digital instructional materials should lead to positive cognitive outcomes
	Cheon & Grant (2008)	Digital instructional materials should avoid presenting large amounts of content because of the risk of cognitive overload
	Chiu and Churchill (2015)	Digital instructional materials should contain some variability of presentation and behavioral response
	Crouse et al. (2018)	Online teachers used technological tools and programs to engage students, such as allowing children to use drawing tools
	Jung et al. (2016)	Digital instructional materials should support cognitive engagement and outcomes
	Kundu et al. (2021)	Elementary children in India engaged quickly with blended curriculum when teachers changed the materials to meet students’ interests and needs
	Shapiro (2008)	Too many choices of activities may overwhelm students and diminish engagement
	ter Vrugte et al. (2017)	Computer-mediated games, simulations, and worked examples facilitated knowledge acquisition
	Tonks et al. (2020)	Students engaged online when emotional needs were met
	Watson et al. (2017)	Feedback is critical for online learners’ cognitive engagement
Advocacy for Inclusion
	Archambault et al. (2021)	Educational marketplaces contain online materials rife with misrepresentations, bias, and stereotyping
	Chu (2000)	Cultural bias is common in digital multimedia materials; biases are reduced by including multiple perspectives and support for thinking
	Galia & Mulyana (2020)	Online English language teaching materials need to be revised to include gender groups other than heterosexual male
	Gallagher et al. (2019)	Teacher education should support teachers in noticing racism and other bias in online instructional materials
	Lewis Ellison (2017)	Learners need to find identity and representation in digital learning experiences
	Mann (2019)	Identifies patterns of whiteness and economic advantage in multiple aspects of online schooling
	Shelton et al. (2020)	Teachers should not buy materials from other teachers on websites unless they evaluate them first for racism and other biases
Accountability
	Attai (2020)	Many teachers do not understand issues of educational data privacy
	Beienberg (2020)	Online instructional materials should disclose information to parents about content development, ideological orientations, and data
	Boniger et al. (2017)	Data privacy is a pressing issue that needs to be addressed in schools
	Hodges and Barbour (2021)	Digital assessment is enabled by digital tools and strategic planning by teachers who are seeking to meet goals and standards
	Lazar et al. (2017)	Students with disabilities are more vulnerable to data misuses and breeches
	Selwyn (2007)	The digital learning marketplace should not exploit learners
	Wang et al. (2019)	Obtaining content expert and user feedback are important for ensuring ethical data use
	Vargo et al. (2003)	Instructional media should align with local or state content standards

During this process, we also compared our findings to previous frameworks to ensure such a framework did not already exist (see Table 2) (California State University Center for Distributed Learning, 1997; Chu, 2000; Vargo et al., 2003; Jung et al., 2016; Information Technology Industry Council, 2020). The 4A Framework is the only framework to include both issues of accessibility for students with disabilities and advocacy for inclusion of groups in addition to students with disabilities. Also, no other framework that was reviewed addressed research on the need for materials to be accountable for the user data and personal information they are collecting and using for various purposes. While the other frameworks that were reviewed called for materials that were aligned to standards, they did not make space for educational goals that were not academic (Nesbit et al., 2009; Vargo et al., (2003). Finally, other frameworks that included concerns about user experiences lacked descriptions of multiple types of engagement (e.g., behavioral, cognitive, emotional) (Chu, 2000; Jung et al., 2016).

Table 2

Existing frameworks for evaluating digital instructional materials

Framework	Creators	Target Audience	Strengths	Challenges
Multimedia Education Resource for Learning and Online Teaching (MERLOT)	California State University Center for Distributed Learning (1997)	Peer reviewers; higher education faculty and staff	Includes a set of 30 individual questions in three domains including quality of content, potential effectiveness, ease of use	Does not directly address accessibility as a set of legal and technical concerns; uses terms like ‘high quality’ without acknowledging the subjectivity of such statements or that notions of quality might have hidden exclusionary premises or outcomes; does not include a robust discussion of engagement; lacks a K-12 focus
A critical content approach to evaluating multicultural multimedia materials	Chu (2000)	Librarians, teachers, parents, and others who are responsible for selecting and providing multicultural multimedia materials (text, sound, and graphics)	The proposed criteria require the examination of four components: objectivity, language, subject mastery, and resources. These criteria are designed to be applied broadly in evaluating multicultural content of any multimedia, for any audience and level of coverage, and in any language	Focuses solely on advocacy criteria for multimedia without acknowledging other considerations; operates from a multimedia orientation but not necessarily an online one
Learning Object Review Instrument (LORI)	Nesbit et al. (2007); Vargo, et al. (2003)	Instructional designers, media developers, and instructors	Measures include nine items with brief descriptive rubrics. Domains include aspects of design, access, alignment to goals and standards, and user feedback	Does not directly address accessibility as a set of legal and technical concerns; assumes goals and standards do not have potentially exclusionary premises or outcomes
A framework for assessing fitness for purpose in open educational resources	Jung et al. (2016)	All educators; product designer or developer	Five critical selection criteria that includes the following topics: 1) Creative Commons License is attributed, 2) easily used with other materials, 3) clear goals and purposes, 4) content accurate and up to date, and 5) educationally significant	Does not address accessibility as a set of technical and legal concerns; does not address aspects of engagement other than cognitive; assumes goals and standards do not have potentially exclusionary premises or outcomes
Voluntary Product Accessibility Template (VPAT)	Information Technology Industry Council (2020)	All educators; product developers; consumers	Section 508 Functional Performance Criteria	Does not address instructional materials beyond legal and technical accessibility concerns

Intended Audience for the 4A Framework

Multiple audiences for the 4A framework are necessary and acceptable in the digital realm because of the many different groups that design, choose, teach with, and purchase digital instructional materials. Other groups have also named multiple audiences in their frameworks for reviewing instructional frameworks (see Table 2). For example, Jung et al. (2016) specifically addressed their framework to educators and developers. Also, Chu (2000) indicated that the critical content approach to evaluating materials could be used by anyone who uses multimedia and then lists librarians, teachers, and parents.

Classroom teachers who use digital instructional materials can use the 4A Framework to evaluate materials created by others, support the modification of materials for new purposes, or as guidance to make their own materials. Because K-12 teachers are not always allowed to choose which materials they use (Archambault & Crippen, 2009; Crouse, et. al, 2018), Course Designers might also benefit from using the 4A Framework. This might be especially important for the asynchronous, teacher-less or teacher-low instruction that is becoming increasingly popular in the K-12 context (Rice & Ortiz, 2020). In such contexts, the only instruction students receive is from the materials. Finally, since Administrators are often the ones who hold the purchasing power, they might also be interested in the 4A Framework to support teachers as instructional leaders.

The rest of this paper conceptualizes the 4A Framework. Each section begins with a definition of each of the elements. Then, a justification using previous research and/or policies is provided. After that, examples appear as vignettes the authors of this article developed for educators to use in professional learning and future research. Appendix contains a checklist of example questions that further clarify the four elements of the 4A Framework.

Accessibility

Definition:

Digital instructional materials are accessible when they adhere to applicable legal standards, and users can open, view, and interact with digital material.

Justification

Students cannot learn from materials that they cannot open, use, view, and interact with (Rose, 2018). To ensure students can access digital materials, specific design features and supports must be present (Rose, 2014). The purpose of supports is to ensure that all students, regardless of disability status, can access the same information and have the same interactions with similar amounts of effort (Dear Colleague Letter from the Office of Civil Rights, 2011). In many countries, students have the right to equal opportunities to benefit from using school-adopted technologies and digital materials (Wisdom et al., 2007).

Common Legal Complaints about Accessibility

According to Rose (2018), most legal complaints about accessibility claim that digital instructional materials: (1) lack alternative text (short descriptions embedded into images), (2) have been posted in formats that are inaccessible, (3) lack captions for video materials, (4) include improperly structured data tables, (5) use improper formatting and labeling of information, and (6) use improper contrast between background and foreground.

Access Points as an Element of Accessibility

In addition, most digital instructional materials require many steps to access them (Behling, 2017). For example, a student who needs to do a simple math worksheet online might have to (1) enter a learning management system, (2) locate the correct folder, (3) open the document, (4) print the document, (5) do the math problems, (6) scan the document with their answers, (7) upload the document, and (8) attach the document in a place where it might be graded. The more steps there are, the less accessible the materials are.

Text Complexity and Access

Some materials might not be accessible because the text is too linguistically complex for students. Text in online instructional materials can shift from low complexity to high complexity without notice to the student. These shifts in complexity can occur between lessons or web pages (Rice, 2017). Support for navigating text complexity, such as vocabulary support, is often unavailable or inaccessible (Rice & Deshler, 2018). Thus, much of the text in instructional materials can be too difficult for students to comprehend.

Supports for Improving Accessibility

Supports to increase accessibility in digital instructional materials include text-to-speech/speech-to-text software or features, display change options (e.g., changing the background color, font type, or size), and alternative text (i.e., explanatory information added to describe embedded images) (Edyburn, 2004; Rose & Meyer, 2006). Students should not be required to sign in multiple times or engage in many successive tasks (e.g., downloading, filling in, re-uploading). Students do need multiple options for file types when they are required to upload or download materials. Screen readers cannot read portable document files (PDFs) unless they have been saved to support screen reading. When teachers make images containing text, such as a.png or possibly a.jpg file from a screenshot of a presentation slide or word processing document, the text will not be readable with screen-reading technologies, either. Alternative text must be added in these cases. Also, some learning management systems block parts of uploaded images on the screen with items like menus or virtual pens and pencils. Students are impeded from working efficiently when they must maneuver around menus and virtual writing instruments to add responses. Ambient sounds from their peers’ devices or offline sounds where they work at school or home may also be distracting and limit access to the materials.

Assistive technologies, including larger computer monitors, special keyboards and mouses, and audio equipment (e.g., headphones or microphones), will help students access materials. Sometimes professional captioners or sign language interpreters are required. Teachers cannot assume students already have access to assistive technologies when they are choosing digital instructional materials. Gathering information about what students already have, what they need, and collaborating with special educators is important for achieving accessibility (de Witte et al., 2018). The example below illustrates accessibility issues that teachers might encounter and how they might begin to address these issues.

Accessibility Scenario: Synchronous Classroom Discussion

Corey teaches eleventh grade economics and government to students who are currently learning remotely due to a state public health order. Corey feels that discussion, even when it occurs over a videoconferencing platform, is important for developing key ideas and understandings of historical events. Corey wanted students to use an educational website with information about economic systems to answer questions and prepare for a discussion. While learning about accessibility, Corey acted on advice to teachers recommending that they place links to educational websites and the discussion questions in a shared document so students do not have to look in multiple places. With the shared document, Corey’s students could follow one link, open the website in a new tab, and toggle between the website and the document, or they could open a new window and place the website and the questions side-by-side. Corey also realized the website had both dark colors and dark font. There was no way to change this. Corey found a different website with the same information that had high contrast between the background and the font. This website had a cleaner appearance, more headings, and was not disrupted by advertisements and images. Finally, Corey learned that the screen readers could not read screen captured questions as a pasted picture file. To ensure that students using screen readers could access the materials, Corey decided to type the questions directly into the document.

In this scenario, Corey made strategic decisions about what digital materials to use with a focus on accessibility. Rather than memorizing a long list of accessibility technicalities, Corey focused on key ideas about access points, provided visual and auditory support by enabling screen reading technology, and anticipated problems students might have. Although institutions need to provide some supports (e.g., when human captioning is required), teachers can make many efforts to ensure instructional materials are accessible on their own.

Active Engagement

Digital instructional materials attend to multiple dimensions of active engagement (i.e., behavioral, cognitive, emotional) when they invite students to invest effort and energy into learning concepts.

Engagement is a commonly used term in educational research that refers to the effort and energy students invest to learn the concepts presented in the learning environment (Fredricks et al., 2004; Hughes et al., 2008). Digital instructional materials can be designed to elicit engagement from students (Cheon & Grant, 2008; Mayer, 2019). Engaging digital materials are especially important when in-person learning contact is reduced or eliminated, which is common in remote, fully online, and even some blended learning environments (Boetcher & Conrad, 2016). Three major types of engagement from research include (1) behavioral, (2) cognitive, and (3) emotional (Lim & Lerner, 2013; Zimmerman & Schunk, 2001).

Behavioral engagement occurs when learners complete measurable and observable actions to learn the content (Zimmerman & Schunk, 2001). Students actively engage in online learning activities by opening pages, clicking on links, giving answers, writing comments, submitting assignments, using emoji, and more (Chapman, 2003; Lawson & Lawson, 2013). When teachers use a variety of digital instructional materials, such as pre-recorded videos, screencasts, games, simulations, and interactive webpages, students can engage with the materials at times convenient to them and as many times as they want (Chiu & Churchill, 2015). Also, students should be able to choose from among multiple types of materials, instead of having to use every type of instructional material in a lesson (Rose & Meyer, 2006). However, merely providing many materials and activities does not universally result in engagement. Students can become overwhelmed by many choices (Shapiro, 2008). They may also fail to behaviorally engage without a clear understanding of the purpose of lesson and may even stop logging on to class (Archambault et al., 2013). Teachers should choose a few types of materials that meet their purposes and satisfy the other elements of the 4A Framework to maintain engagement (see Fig. 1).

Cognitive engagement is “the expenditure of thoughtful energy needed to comprehend complex ideas in order to go beyond the minimal requirements” (Finn & Zimmer, 2012, p. 102). Cognitive engagement often emerges alongside a behavioral response, such as asking a poignant question or completing an assignment. Cognitive engagement is supported when students have access to information about all the relevant parts of the concept rather than overly simplified definitions or incomplete explanations (Meyer, 1993; Sanders & Makosta, 2016). Worked examples and simulations have also proven useful for learning concepts with digital materials (Roblin et al., 2018; ter Vrugte et al., 2017).

Emotional engagement requires the “presence of task-facilitating emotions such as interest and the absence of task-withdrawing emotions such as distress” (Reeve, 2012, p. 150). Sometimes, students have negative emotions in response to digital instructional materials about tragic events. Presenting content that consistently evokes negative emotions, like distress, impedes cognitive and behavioral engagement (Nabi, 1999). How learners are treated by teachers and peers also affects emotional engagement with instructional materials (Ruzek et al., 2016).

Content presented on the internet with powerful visual media also provides more opportunities for students to exhibit unexpected emotional responses to materials (Chen et al., 2015; Lim & Childs, 2020). Teachers should understand that students’ emotional responses might not be predictable because of past experiences, culture, disability, language, or other factors. For example, students might respond to a video of the falling World Trade Center Towers with cheers and laughter, rather than with horror and sadness, because they are intrigued by the visual images of large structures collapsing. They might also be confused by many adults’ negative reactions to footage of the towers collapsing because there are many videos on the internet of building implosions that were safely planned and carried out. The fact they were not yet born in 2001might make it more difficult for young people to emotionally respond to the building collapse in ways adults expect.

Overlap in Types of Engagement

Highly engaging instructional materials simultaneously attend to multiple types of engagement (Chiu & Churchill, 2015). Such materials tend to lead with intellectually interesting questions (cognitive engagement), require a recorded response (behavioral engagement), and invite appropriate feelings about the topic (emotional engagement). Materials designed for engagement also can include elements from games, such as earning points, gathering materials, or going on a quest, and using feedback (Reeves & Reed, 2009; Watson et al., 2017). Feedback can support engagement when it includes features like leaderboards, interactive task lists, and partial credit (behavioral); information about performance, including missed items and worked examples (cognitive); and upbeat sounds, inspirational images, or short positive phrases, like “good job” (emotional) (Watson et al., 2017; Yeo & Fazio, 2019).

Active Engagement Scenario: Using Instructional Videos

Avery teaches seventh grade English. Avery’s students use blended rotations to complete some of their work online. The students also engage in person with each other in small groups and as a whole class. Avery wanted to include a short video in an online lesson developed to introduce key concepts about the industrial revolution in England. Several colleagues recommended a website that contained an online video about the topic and included a free worksheet with many specific questions about minor facts and events in the video. When previewing these materials, Avery found the video very long with slow narration. Avery became concerned the students would become disinterested before the main content of the video was presented. The students might also find it difficult to attend to the detailed points of the worksheet while watching the video, and they might give up on both. Avery located a shorter video with more lively dialogue and images. Instead of a worksheet asking for many details, Avery prepared a simple organizer for students to download and record information while they watched. The organizer asked a few general questions and left space for short answers (1-3 words). The students could also respond to the questions using emoji from a bank at the top of the sheet. Also, Avery worried that some of the images of children covered in soot, looking tired, or suffering with major physical injuries may be emotionally disruptive. Avery decided to pre-record a short video segment to prepare the students for the fact that images of some of these children might make them sad. Avery sent this video to parents using a school email list feature, so parents would know that their children were being presented with sensitive images.

In this example, the goal was for students to engage with key ideas and not just view images of children in distress. Avery did not merely find a video that was short enough to hold the learners’ attention. Instead, Avery made efforts to monitor engagement with digital instructional materials to ensure students had multiple ways to behaviorally demonstrate what they were learning cognitively while also receiving appropriate emotional support.

Advocacy for Inclusion

Definition.

Digital instructional materials promote advocacy for inclusion when they represent diverse peoples with contextual nuance, compassion, and respect.

Instructional materials presented to students should advocate for inclusion. Materials represent an inclusive mission when they present people as having complex lives, motivations, and interests (Chu, 2000; Lewis Ellison, 2017). Inclusive instructional materials promote engagement, particularly among students historically from excluded groups, whose stories and perspectives are often absent or mischaracterized in school spaces (Bishop, 2012).

Digital instructional materials reflect the hidden curriculum that permeates teaching and learning at all levels of education (Alsubaie, 2015; Margolis, 2001). Hidden curriculum creates conditions where the materials and activities students are exposed to advance an implicit argument that characteristics such as whiteness, heteronormativity, and hegemonic masculinity are preferable to other characteristics (Selvi & Kocaman, 2020). Students also receive messages about what languages and dialects are valued, what their bodies should look like physically, what counts as intelligence, and which emotions are valid.

Inclusion and Representation in Digital Instructional Materials

Digital instructional materials allow for high quantities of easily shared content that has been encoded in multiple simultaneous modes (e.g., visual and auditory) to be transmitted to nearly limitless audiences with little oversight (Chu, 2000). Because there is no governance ensuring that instructional materials will fairly represent all groups, teachers must ensure they are identifying and evaluating the materials for racist and biased messaging (Gallagher et al., 2019). For example, many materials for sale on sites where teachers sell materials to other teachers presented content from racist perspectives (Archambault et al., 2021; Shelton et al., 2020). Another recent study of online materials for English language instruction found gender bias, such as stereotypes, under-representation, and gender-exclusive language use (Gailea & Mulyana, 2020).

Instructional Materials as Conduits for Advocacy

When instructional materials promote exclusion, they are not quality materials, even if the materials are otherwise accessible and engaging for students. Teachers must actively review and choose materials that support valuable opportunities for challenging deficit views of social groups and promoting community in the classroom and beyond. Teachers should not assume students will automatically make connections in their thinking that will provide the opportunity to notice and denounce social injustices on their own. Instead, an ethos of inclusion is built by presenting multiple perspectives on events, providing multiple opportunities for student response, and then supporting students in examining and dismissing perspectives using various approaches to reasoning (Gallagher et al., 2019). Below is an example of how teachers might confront online educational texts that are not inclusive.

Advocacy for Inclusion Scenario: Indigenous Storytelling

Harper teaches first grade using a blended learning format where the students are divided in half, and they alternate weeks in coming to school in person and staying home to learn online. Harper’s school district serves over 8,000 students from Native American groups. These students in the district represent over 100 indigenous communities from all over the world, but many students and their families claim membership in one or more of the 20+ local groups. Harper wanted to represent Native American peoples in her teaching and make strong school-to-home connections. Therefore, Harper searched for materials about Native Americans to use with students when they were learning online at home. Most of the materials she found featured images of native peoples who were depicted as children with feathers in their hair. Harper also found a few sites about Native American storytelling. Harper immediately dismissed the feather images as misrepresentative but decided to use some of the Native American stories about animals and pair them with a science unit about animals. In continuing to search, Harper found information from a local indigenous people’s cultural center and learned that many of the groups in her community only engaged in storytelling during the winter months. To do so outside of that time of the year, even in school, was considered problematic and disrespectful by many families. Further, Harper learned that some of her students claimed membership in groups that had restrictive expectations for storytelling about certain animals, such as spiders and snakes. Considering this information, Harper decided to use the digital materials provided by the local cultural center rather than write new ones or buy them from a site that was not sponsored or endorsed by native peoples. Harper also decided to search for additional materials that were developed by, or in collaboration with, members of these groups.

The example above illustrates the complexity involved in finding and evaluating digital instructional materials and the need for collaboration with local communities. Sometimes teachers might wish that there was a concise list of Dos and Don’ts somewhere that they could memorize or have on hand. Other teachers might become exasperated by what they perceive as too many rules. Instead of an annoyance, prioritizing inclusion in selecting digital instructional materials should be conceptualized as a shared responsibility and part of an on-going effort to align educational practices with community ethics.

Accountability

Digital instructional materials demonstrate accountability when they are transparent about their origins and purposes, based on standards or principles, and are open about personal information and user data collection sharing processes.

Teachers should seek digital instructional materials designed to foster a learning experience that aligns with learning goals and objectives (Vargo, et. al. 2003). Explicit alignment with goals is an important element of digital assessment practice (Hodges & Barbour, 2021). Standards alignment is common in other digital materials evaluation frameworks (California State University Center for Open and Distributed Learning, 1997; Chu, 2000; Jung et al., 2016; Nesbit et al., 2007; Vargo et al., 2003). In the 4A Framework, accountability is more than ensuring that students are accountable for learning standards and teachers are accountable for teaching the standards. Digital materials must also be held accountable to the students and families for what is done with the information gathered during the learning process.

Origins and Purposes

Instructional materials should reveal their goals and values to users (Beienberg, 2020). For example, the Learning for Justice organization (2021) created instructional materials for both online and in-person learning. It was easy for teachers to ascertain the organization’s values and goals because there is a page on their website titled “About Learning for Justice,” where the group clearly reveals their connection to the Southern Poverty Law center. They do not claim they are politically or socially neutral in their aims. Instead, they are transparent about their alliances.

Connections to Standards and Principles

Instructional materials that connect to local, state, or national standards should describe how they are aligned, instead of merely stating that they are (National Academies of Science, Engineering, and Medicine, 2018). Not all materials have to be aligned to state standards to be considered high quality, and digital assessment is much more than standardized testing (Rice, 2021). Some digital instructional materials might have been created to support non-content goals, such as social/emotional development. Other materials might operate from a position that content standards represent a baseline of knowledge only, and students should be given the opportunity to go beyond the standards (Noddings, 2013). Such extensions might be considered important advantages in online learning settings so long as these are explained to students by the developer, the teacher, or hopefully both.

Data Collection and Security

Digital instructional materials should reveal what personal information and user data they are collecting (Boninger et al., 2017). When students create accounts, they give their personal information to a company who collects data about them (Attai, 2020). Collecting some data might be important for vendors to support and maintain the application. However, sometimes account agreements allow vendors access to search histories and other information that is outside of the original application and not important to the functioning of the program or its materials.

Educators, including teachers and administrators, need to carefully consider what data they are asking young people and their families to give to companies when they require registration for accounts as part of a class. Sometimes, teachers might get excited about new applications and send their students links to create accounts before they have considered how to use these materials (Beienberg, 2020). Students might end up with many user accounts for materials that the teacher never uses or only uses a few times. For a student with a disability or who has trouble remembering and keeping track of passwords, this can add frustration. It also creates a need for an adult to assist in the storage and retrieval of all these passwords and makes data more susceptible to misuse (Lazar et al., 2017). While single sign-on options alleviate the problem of remembering passwords, they also increase the need for vendors to share what data they are gathering within and across sites and what they plan to do with the data they collect (Pangrazio & Cardozo-Gaibisso, 2021; Pötzsch, 2021).

Vendors might also give schools access to students’ personal information beyond what the school already collects in addition to user data about how the students perform. In such cases, schools should also be transparent about how they are using student data to distribute opportunities, contribute to databases, make decisions about resources, and so forth. Some district user agreements explain the rules for students and families, but many might not yet include reciprocal protections where the district promises to protect personal information and user data. Policies at local and state levels regarding the terms schools can enter with vendors to share students’ data is emerging but incomplete (Krueger, 2014; Reidenberg & Schaub, 2018; (Unauthorized Release of Personally Identifiable Information, 2015). The goal should be to give students and their decision-makers as much information as possible and allow them to make decisions about their personal information and data. In the meantime, schools and families need to navigate accountability for data privacy and related protections together as best they can (Reidenberg & Schaub, 2018). Below is an example of a teacher working to address accountability challenges in online instructional materials.

Accountability Scenario: Math Textbook Adoption

Jan needed to adopt a math textbook for students in a distance learning school that pairs physical textbooks with some online instruction. The school where Jan worked offered three choices. The first book had a seal on the cover that said it was aligned to state standards recommendations, but it had no description of how the instruction was linked to the standards. The second book offered worked examples for students online through the company website, but the school needed to pay a licensing fee for students, or families could pay separately. The publisher of the third book was also offering worked examples and other activities online; some of these online resources were free and did not require registration, while others did require registration and did charge a fee. When Jan contacted the publishers of choices two and three about data collection practices, the publisher of choice two said that privacy information was not available, but assured Jan that they only used the data they needed. The publisher of the third choice sent a disclosure about what data was collected, how it was used for program improvements, and how it was stored. The disclosure also indicated that both the free and the fee-based materials collected some user information, but the fee-based materials gathered data that could be shared with students about their progress and offered students more options to opt-out of some data collection features. Jan also saw that the book had been endorsed by American Mathematical Society for its strong documentation of standards alignment. Jan decided to recommend that the school principal purchase the third book and budget enough money to pay the fee to register students, so they could have access to and make more decisions about their data.

The example above demonstrates the importance of mutual accountability in evaluating digital instructional materials. While it is important to advocate for open materials, it is problematic when schools choose free materials from vendors that gather students’ personal information and user data, use it, and then sell it, while denying students access to and control over the data. Students lose potential benefits from working with digital standards-based instructional materials when the materials do not provide useful information to them and when the students’ data as a commodity takes primacy over the learning experience.

Limitations and Affordances of the Framework

The 4A Framework enables decision-makers to evaluate individual items intended for use in instruction (e.g., a single video or website) and make informed decisions about using the material as it is, adding additional support, or modifying the materials. Educators could also use the 4A Framework in groups to make decisions about what instructional material to use with classes, small groups, and individuals. While some users of the 4A Framework might see how these elements can apply to instructional materials that are not online, it is important to remember that the tool was developed using research and policies that considered the affordances of digital materials in K-12 settings across a variety of digital settings (e.g., fully online, remote, blended). In fact, existing conceptual overlaps may make the tool highly suitable for learning spaces where both traditional and online materials are used.

Conclusion

Teachers deserve access to instructional materials that they can use with students with confidence (Kim et al., 2017). The 4A Framework for evaluating digital instructional materials provides teachers and those who support them with important guidance for making decisions about teaching and learning (Mayer, 2019). Even after teachers choose materials, they must continue to be vigilant. Informing teachers’ judgement about major concerns can help them make decisions for their students (Chu, 2000). Of course, more research is needed that focuses on supporting teachers and other decision makers in evaluating learning materials for children with different needs, strengths, and interests. Specifically, more research is needed to investigate how teachers use the 4A Framework and whether and how it improves their ability to evaluate instructional materials. Even so, this framework adds a needed update and consolidation of ideas about quality online instructional materials in K-12 settings.