Designing a Framework to Facilitate Metacognitive Strategy Development in Computer-Mediated Problem-Solving Instruction.

Metacognition is a key component of ill-structured problem solving. As such, there is a need for instructional design guidance for metacognitive skill development. This study addressed this need through the creation of a computer-based interactive content design framework to guide the design of metacognitive scaffolds for ill-structured problem-solving instruction. It utilized a type II design and development research approach and formative design principles to create a comprehensive and generalizable instructional design framework. The framework was composed by synthesizing research and practical literature and evaluated by experts in related fields. Key components of the framework include metacognitive strategies, instructional design strategies, interactive media types, question prompts, and feedback. Instructors, course developers, and other key stakeholders could follow the guidelines proposed in this framework to create metacognitive-based ill-structured problem-solving instruction using e-Learning authoring tools. This study bridges the gap between theory and practice, as well as adds to literature in media research with focusing on utilizing various media types to create effective learning materials. © Association for Educational Communications & Technology 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Introduction

Problem solving is an essential skill for success in the twenty-first century (Belland, 2013; Kim & Tawfik, 2021; Law et al., 2020). It has been studied extensively in mathematics (Jonassen, 2011; Wilson et al., 1993), engineering (Pappas, 2002; Pappas & Pappas, 2003), and online information searching (Walraven et al., 2008). Metacognition is regarded as a key component in problem solving (Lin et al., 1999; Tarricone, 2011), especially in solving ill-structured problems (Ge & Land, 2003; Jonassen, 2000; Shin et al., 2003). However, there is a lack of guidance for developing metacognitive skills as a means of supporting ill-structured problem solving.

The term metacognition was first introduced by John Flavell in the 1970s, and it is also called second-order cognition and cognition of cognition (Flavell, 1987; Papaleontiou-louca, 2003; Tarricone, 2011). There are three major components of metacognition, which are metacognitive knowledge, metacognitive experience, and metacognitive skills (Efklides, 2008). Metacognitive knowledge is the learner’s stored world knowledge (Flavell, 1979), knowledge about oneself and cognitive tasks (Ku & Ho, 2010). Metacognitive experience refers to the feeling and judgment towards completing tasks (Efklides, 2008; Flavell, 1979). Metacognitive skills are also called metacognitive strategies (Efklides, 2008, 2014; Veenman & Elshout, 1999), including planning, monitoring, and evaluating thinking processes (Akturk & Sahin, 2011; Ku & Ho, 2010).

Currently, many computer-based learning materials are produced using e-Learning authoring tools, such as Articulate Storyline, Adobe Captivate, and Camtasia. Functional attributes of those e-Learning authoring tools, including multiple choice questions, short answers, and auto-grading, could be utilized to create metacognitive scaffolds in e-Learning instruction. By engaging in metacognitive-based instruction, learners can acquire metacognitive skills to plan, control and monitor their learning processes.

However, the literature is sparse in studying the application of interactive media attributes in creating computer-based instruction to support metacognitive development. In consideration of the critical role that metacognition plays in enhancing learner’s problem-solving skills, there is a need to produce evidence-based computer-based learning materials to teach metacognitive skills to an increasing number of online learners (Brown et al., 1994). The shift to virtual learning during the COVID-19 pandemic underscores the growing need for online approaches to problem solving instruction.

An instructional framework that guides the design of computer-based metacognitive materials will be beneficial to instructors and practitioners when creating problem-solving instruction. In line with these goals, this study is conducted to create an interactive content design framework that serves as guidelines for instructors and practitioners to create computer-based problem-solving instruction. By embedding metacognitive scaffolds in the self-paced instruction, learners are trained to monitor and control their thinking process efficiently during problem-solving process.

Methodology

Research Purpose and Question

The focus of this study is the design of an interactive content development framework with utilizing formative design principles to teach and train metacognitive skills. The guidelines proposed in the framework consolidate prior research from metacognition, multimedia design, instructional design, interaction design, and message design. Three key components were identified from the literature and included in this framework. Those components are metacognitive strategies, instructional strategies, and media attributes. The interactive content will be developed using e-Learning authoring tools with following the guidelines proposed in the framework. The research question framed this study is:

What features would a framework have to facilitate metacognitive strategy development in computer-mediated instruction?

Study Design

This study employed a design and development research (DDR) approach with implementing formative design principle throughout the design process. DDR is defined as “the systematic study of the design, development and evaluation processes with the aim of establishing an empirical basis for the creation of instructional and non-instructional products and tools and new or enhanced models that govern their development” (Richey & Klein, 2007, p.1). There are two types of DDR, including type Ι product and tool research and type Π model research (Richey & Klein, 2005, 2007, 2014; Richey et al., 2004). To create a framework that guides the design of metacognitive-based interactive content, this study adopts type Π model research method with suggesting key steps in designing metacognitive scaffolds in ill-structured problem-solving contexts.

This study consists of four phases, which are (1) Analysis, (2) Design and Development, (3) Evaluation, and (4) Revision. A formative design approach is adopted throughout the design process. According to Glaser et al. (2017) and Hokanson and Kenny (2020), iteration is a key parameter in design research. This study implements an iterative process to continuously design, refine, and evaluate the framework based on experts’ feedback.

Study Procedure

Analysis

A systematic and thorough review and analysis of the existing literature on metacognitive strategies, instructional strategies, and interactive attributes of the e-Learning authoring tools were performed, related questions and literature sources are presented in Table 1. Publications on these topics were obtained from journal articles, dissertations, theses, and e-Learning design practices. Key search terms included “metacognition,” “metacognitive skills,” “metacognitive strategies,” “interactivity,” “computer-based instruction,” “interaction design,” “message design,” and “screen design.”

Table 1

Design elements of interactive content in web-based instruction

Questions	Sources
What are the metacognitive strategies?	Published Literature
What are the instructional strategies to teach metacognitive strategies/skills?	Published Literature
What media attributes could be used to create interactive instruction?	Web sources and investigation of interactivity attributes of multiple e-Learning authoring tools

Design and Development

The design and development phase consisted of the construction of components and operationalization of the design procedures of interactive content design framework. This phase was carried out based on findings from the Analysis phase where relevant research on metacognitive strategies, instructional strategies, and media attributes was identified and synthesized.

Evaluation

Expert review was adopted as an internal validation procedure (Ross et al., 2007) to evaluate the effectiveness of the developed framework. This process occurred by asking experts to evaluate components, structure, and application of the model (Ross et al., 2007). The evaluation process consisted of two phases: pilot study and formal study. Preliminary feedback was collected from two scholar-practitioners in the pilot study to improve the framework and adjust expert review invitation messages and evaluation survey. Selected experts in instructional design, problem solving, and metacognition were invited to participate in the formal phase of the evaluation.

Revision

During the revision phase, feedback from the experts was implemented to refine the framework. The modified framework is presented in the next section.

Framework for Designing Metacognitive Scaffolds in Ill-Structured Problem-Solving Instruction Using e-Learning Authoring Tools

Scholars believe that a continuum exists between ill-structured and well-structured problems, and there is no clear boundary between them (Simon, 1973; Voss, 1988). Ill-structured and well-structured problems share similar problem-solving procedures (Gick, 1986), such as represent the problem, generate solutions, present a solution, evaluation, and reflection. This framework presents detailed instruction to guide the design of metacognitive scaffolds to facilitate learners in solving ill-structured problems. Metacognitive scaffolds help learners monitor and reflect on their thinking process (Crescenzi et al., 2021; Kim & Lim, 2019; Zhou & Lam, 2019). Commonly used metacognitive scaffolds in problem solving instruction are prompting and modeling (Hollingworth & McLoughlin, 2001; Lin et al., 1999). The following sections introduce key problem-solving stages, metacognitive scaffolds, instructional strategies, examples, and interactive media types identified from existing literature.

Ill-Structured Problem-Solving Stages

Ill-structured problem-solving instruction can be created by following general ill-structured problem-solving procedures. Those stages are representing problem (Kale et al., 2018; Kim & Tawfik, 2021), generating solutions (Ge & Land, 2004; Sinnott, 1989), presenting a solution (Ge & Land, 2003, 2004; Kim & Tawfik, 2021), evaluating the solution (Ge & Land, 2003, 2004; Jonassen, 1997), and reflecting on the problem-solving process (Kauffman et al., 2008; Kim & Hannafin, 2011). Related literature that underpins each stage is listed in Table 2.

Table 2

Ill-structured problem-solving stages

Stages	Literature
Represent problem	Decide about the essence or nature of the problem (Sinnott, 1989); Define, generate and pursue learning issues to understand the problem (Kim & Tawfik, 2021); Presentation, understanding, or definition of problem (Swanson, 1990)
Generate solutions	Develop a possible solution (Kim & Tawfik, 2021); Search for a solution (Chi & Glaser, 1985); Choose and generate solutions (Sinnott, 1989); Solution development (Ge & Land, 2004)
Present and evaluate solution	Present their solution and the evidence to support it (Belland et al., 2008); Provide justifications (Ge & Land, 2003, 2004; Kim & Tawfik, 2021); Monitor and evaluation (Ge & Land, 2003, 2004); Solution evaluation (Davidson et al., 1994); Assess problem solution (Jonassen, 1997)
Reflect	Reflection and negotiation (Kim & Hannafin, 2011); Reflect on work (Kauffman et al., 2008)

Different from well-structured problem solving, ill-structured problem-solving procedures are situated (Jonassen, 1997, 2000; Voss, 1988), dynamically intertwined, circular, and iterative (Ge et al., 2016). The initial stage of creating metacognitive-based ill-structured problem-solving instruction is to present the problem. Problems presented in the learning materials should be situated, real-world problems to foster knowledge transfer (Lin et al., 1994; Liu et al., 2012; Park & Hannafin, 1993). Once an ill-structured problem is presented, learners can follow the problem-solving process to solve the problem. The problem-solving stages include represent problem, generate solutions, present and evaluate a solution, and reflect as shown in Fig. 1.

Fig. 1

Metacognition-based ill-structured problem-solving process

Represent Problem

The purpose of problem representation is to understand and form the initial state of the problem (Voss, 1988), such as identifying cause(s) of the problem and setting goals. This process is complex, which requires the learner to discover missing information, differentiate relevant information from irrelevant information, and express personal opinions of the problem (Jonassen, 1997, 2000). To assist learners with developing a problem representation, metacognitive scaffolds could be created to prompt learners to identify the causes and constraints of a problem (Ge & Land, 2003; Jonassen, 1997; Voss, 1988).

Generate Solutions

Once the initial state of an ill-structured problem is constructed, the learner moves on to generate solutions. According to Shin et al. (2003), an ill-structured problem could possess more than one solution with multiple solution paths. Jonassen (1997) posited that multiple solutions resulted from different problem representations. Sinnott (1989) described this process as a creative exercise that cultivates divergent thinking. This process requires information identification, strategy selection, data collection, and action plan formulation.

Present and Evaluate Solution

Once a solution is chosen, it should be verified and evaluated (Garofalo & Lester, 1985; Kwang, 2000). Huttenlock (2007) recommended to evaluate both the process and goal completion in an ill-structured problem domain. The evaluation of goal completion focuses on checking if the learners have enough information to confirm their claims or reach the preset goals. In contrast, process evaluation emphasizes on choosing appropriate data and strategy during the process (Huttenlock, 2007). Jonassen (1997) proposed to evaluate the viability of solutions by asking questions, such as “Was the problem solved? Did it adhere to the constraints?”.

Reflect

Reflection is the last stage in ill-structured problem-solving process. During this stage, learner revisits ideas, process, adopted strategies, and solutions (Collins & Brown, 1988; Kim & Hannafin, 2011). Learning occurs when learners reflect on their own mistakes, make plans for improvement, and prepare for knowledge transfer in other contexts. Computer-based learning environment could be used as an effective medium to facilitate the reflection process (Collins & Brown, 1988).

Metacognitive Strategies and Instructional Activities

Metacognitive strategies are essential for solving ill-structured problems by monitoring and controlling cognitive processes (Brown et al., 1994; Flavell, 1987). According to Shin et al. (2003), metacognitive skills, such as plan and monitor, are strong predictors of the success in solving unfamiliar ill-structured problems. Five metacognitive strategies are presented in this framework, which are Orient (Garofalo & Lester, 1985; Meijer et al., 2006; Molenaar & Chiu, 2014), Plan (Brown, 1987; King, 1991; Pintrich, 2002), Monitor (Brown, 1987; Efklides, 2014; Zimmerman & Campillo, 2003), Evaluate (Bannert & Reimann, 2012; Schmidt & Ford, 2003; Yıldız-Feyzioğlu et al., 2013), and Reflect (Meijer et al., 2006; Molenaar & Chiu, 2014; Zimmerman & Campillo, 2003).

Orient

Garofalo and Lester (1985) defined orienting as a strategic behavior to understand and assess a problem. Meijer et al. (2006) asserted that experts spent more time than beginners on orientation activities. Orienting occurs during the problem representation stage, and orienting activities include understanding the problem (Kale et al., 2018), examining the causes of the problem (Jonassen, 1997; Kauffman et al., 2008; Voss, 1988), identifying relevant and irrelevant information (Hollingworth & McLoughlin, 2001; Swanson, 1990), as well as associated constraints, variables, and relationships (Ge & Land, 2004; Kale et al., 2018; Voss, 1988).

Orienting skills can be trained using question prompts, such as “What are the causes of the problem?” (Kauffman et al., 2008), “What information is relevant to solve this problem and what information is irrelevant?” (King, 1991), and “What are the key variables of this problem and how are they related?” (Meijer et al., 2006). More examples can be found in Table 3 in Appendix 1.

Table 3

Revised design of computer-based metacognitive scaffolds in represent problem stage

Metacognitive strategies	Instructional strategies	Examples		Interactive media type
		Questions	Feedback
Orient (Bannert & Reimann, 2012; Garofalo & Lester, 1985; Meijer et al., 2006; Molenaar & Chiu, 2014)	Understand problem (Belland et al., 2008; Kale et al., 2018)	Do you understand the problem?	If learner answers “Yes”, he/she will be prompted to explain the problem using their own words (Kim & Hannafin, 2011; King, 1991) If learner answers “No”, he/she will be prompted to discuss the problem with the Chatbot or their peers in Discussion Forum	True/False; Checkbox; Radio Button; Audio Input; Notes; Hyperlink; Web Objects (Chatbot; Discussion Forum; AI-based Feedback)
	Examine causes of the problem (Jonassen, 1997; Kauffman et al., 2008; Voss, 1988)	What are the causes of the problem? (Kauffman et al., 2008) or The causes of the problem are___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Fill-in-the-Blank; Audio Input; Notes; Web Objects (AI-based Feedback)
	Identify relevant and irrelevant information (Hollingworth & McLoughlin, 2001; Swanson, 1990)	What information is relevant to solve this problem and what information is irrelevant? (King, 1991)	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Drag and Drop; Multiple Choice; Short Answer; Text Entry Box; Radio Button; Audio Input; Notes; Web Objects (AI-based Feedback)
	Identify constraints, parts, variables, relations (Ge & Land, 2004; Kale et al., 2018; Voss, 1988)	What are the key variables of this problem and how are they related? (Meijer et al., 2006)	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Drag and Drop; Radio Button; Audio Input; Notes; Web Objects (AI-based Feedback)
	Identify available resources/information (Bannert & Reimann, 2012)	What resources or information are available to solve the problem? Or Available resources/information to solve this problem include___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Notes; Checkbox; Radio Button; Audio Input; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Activation of prior knowledge and experience (Huttenlock, 2007; Meijer et al., 2006; Shahbodin & bt Bakar, 2010; Yıldız-Feyzioğlu et al., 2013)	Do you have any related experience with this kind of problem? (Shahbodin & bt Bakar, 2010)	If learner answers “Yes”, he/she will be prompted to elaborate on his/her relevant experience to this problem If learner answers “No”, he/she will be prompted to discuss the problem with the Chatbot or his/her peers in Discussion Forum	True/False; Text Entry Box; Audio Input; Notes; Radio Button; Hyperlinks; Interactive video; Animation; Simulation; Web Objects (Chatbot; Discussion forum; AI-based Feedback)
	Goal Setting (Bannert & Reimann, 2012; Garofalo & Lester, 1985; Yıldız-Feyzioğlu et al., 2013)	What is/are your goal(s) and/or sub-goals for solving this problem? Or My goal(s) and/or sub-goals for solving this problem is/are ____	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Audio Input; Notes; Fill-in-The-Blank; Web Objects (AI-based Feedback)
	List needed knowledge/information (Shahbodin & bt Bakar, 2010)	What information and knowledge are needed to solve this problem? Or To solve this problem, ____ are needed	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Audio Input; Notes; Drag and Drop; Checkbox; Radio Button; Fill-in-the Blank; Web Objects (AI-based Feedback)

Plan

Tzohar-Rozen and Kramarski (2014) defined planning as a skill of choosing strategies and allocating resources. It is applied in problem representation and generating solutions stages. In problem representation stage, planning activities include activating prior knowledge and experience (Huttenlock, 2007; Meijer et al., 2006; Yıldız-Feyzioğlu et al., 2013), setting goals (Bannert & Reimann, 2012; Garofalo & Lester, 1985; Yıldız-Feyzioğlu et al., 2013), and formulating hypothesis (Bogard et al., 2013; Gick, 1986; Meijer et al., 2006). During the solution generation phase, planning activities contain formulating an action plan, identifying needed information, and selecting strategies (Huttenlock, 2007). Planning scaffolds can also be designed using question prompts, such as “Do you have any related experience with this kind of problem?”, “What are possible solutions of this problem?”, and “What are the steps to solve this problem?” (Shahbodin & bt Bakar, 2010). More information on planning scaffolds and examples can be found in Table 3 (in Appendix 1) and Table 4 (in Appendix 2).

Table 4

Revised design of computer-based metacognitive scaffolds in generate solutions stage

Metacognitive strategies	Instructional strategies	Examples		Interactive media type
		Questions	Feedback
Plan (Brown, 1987; Gordon, 1996; King, 1991; Kirsh, 2005; Molenaar & Chiu, 2014; Pintrich, 2002; Shin et al., 2003)	Propose solutions (Shahbodin & bt Bakar, 2010)	What are possible solutions for this problem? (Ge & Land, 2003; Shahbodin & bt Bakar, 2010) Or Solutions for this problem include ___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Audio Input; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Formulate action plan (Huttenlock, 2007; Kapa, 2001; Meijer et al., 2006)	What are the steps to solve this problem? (Hollingworth & McLoughlin, 2001; Shahbodin & bt Bakar, 2010) Or Steps that it takes to solve this problem include___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Audio Input; Text Entry Box; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Select strategy (Huttenlock, 2007)	Which strategy/strategies will you use? (Tzohar-Rozen & Kramarski, 2014) Or Strategy/Strategies that you will use to solve this problem include___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Audio Input; Text Entry Box; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
Plan (Brown, 1987; Gordon, 1996; King, 1991; Kirsh, 2005; Molenaar & Chiu, 2014; Pintrich, 2002; Shin et al., 2003)	Identify needed information (Bulu & Pedersen, 2010; Huttenlock, 2007; Swanson, 1990)	What information do you need to solve this problem? (Bulu & Pedersen, 2010) Or To solve this problem, you will need_____ (Bulu & Pedersen, 2010)	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Audio Input; Text Entry Box; Checkbox; Radio Button; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Find information/data (Bannert & Reimann, 2012)	Where can you find the information? (Bannert & Reimann, 2012) Or You can find information from ___	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Audio Input; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Propose evaluation criteria (Swanson, 1990)	What are the necessary elements of an acceptable problem solution? (Belland, 2013) Or Components that need to be included in the problem solution are__	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Audio Input; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Data/Information generation (Huttenlock, 2007)	Do you find needed information to solve this problem?	If learner answers “Yes”, he/she will be prompted to answer the next question If learner answers “No”, he/she will be given three options, including “Clicks into the Resources tab to study relevant materials”, “Ask the Chabot”, and “Seek help in the Discussion Forum”	True/False; Checkbox; Radio Button; Audio Input; Web Objects (Chatbot; Discussion Forum)
Monitor (Brown, 1987; Efklides, 2014; Gordon, 1996; King, 1991; Kirsh, 2005; Pintrich, 2000; Zimmerman & Campillo, 2003)	Goal checking (Bannert & Reimann, 2012)	Are you reaching your goals? (Huttenlock, 2007)	If learner answers “Yes”, message says “good job!” will appear on the screen If learner answers “No”, message says “Keep working. You will get there!” will appear on the screen	True/False; Checkbox; Radio Button; Audio Input;
	Strategy implementation (Kwang, 2000)	Are you using the strategy that you chose? (Kwang, 2000)	If learner answers “Yes”, he/she will be prompted to the next question If learner answers “No”, he/she will be prompted to discuss other strategies with the Chatbot or his/her peers in the Discussion Forum	True/False; Checkbox; Radio Button; Audio Input; Short Answer; Text Entry Box; Web Objects (Chatbot; Discussion Forum)
	Check progress (Bulu & Pedersen, 2010; Ge & Land, 2003, 2004; King, 1991; Kwang, 2000)	Are you on the right track?	If learner answers “Yes”, message says “Keep the good work!” will appear on the screen If learner answers “No, I am lost/stuck”, he/she will be presented with three options, including “Please review your plans”, “Ask the Chabot”, and “Seek help in the Discussion Forum”	Multiple-Choice; Checkbox; Radio Button; Audio Input; Backward Button; Web Objects (Chatbot; Discussion Forum)

Monitor

Monitor refers to one’s awareness of task comprehension and performance (Schraw, 2001; Tzohar-Rozen & Kramarski, 2014). Quintana et al. (2005) described monitoring activities as identifying tasks, evaluating work progress, and predicting outcomes. Monitoring occurs throughout the problem-solving process, which includes the monitor of cognitive process and learner emotion. Examples of monitoring prompts include “Is this problem easy or hard?” (Tzohar-Rozen & Kramarski, 2014), “Are you reaching your goals?” (Huttenlock, 2007), and “Are you using the strategy that you chose?” (Kwang, 2000). Additionally, expert modeling should be offered when learner expresses difficulties in completing tasks. Expert modeling can be presented in video demonstration or simulation learning environment. More monitoring scaffolds and examples can be found in Table 4 in Appendix 2.

Evaluate

Evaluation strategies are used in the evaluation stage of solving ill-structured problems, which serves the purpose to justify proposed solution and process (Tzohar-Rozen & Kramarski, 2014). Metacognitive scaffolds adopted in this phase are specified as the assessment of supporting information (Bulu & Pedersen, 2010) and defending of a chosen solution (Ge & Land, 2004). However, if the chosen strategy does not meet the evaluation criteria, learner needs to restart the problem-solving process. It is through evaluation that the learner refines their working process and outcomes. Example of question prompts that are used to foster metacognitive evaluation are “What are your reasons and/or arguments for proposing this solution?” (Ge & Land, 2003) and “Have you taken into account the perspectives of different stakeholders?” (Ge & Land, 2004). More strategies and examples are described in Table 5 in Appendix 3.

Table 5

Revised design of computer-based metacognitive scaffolds in present and evaluate solution stages

Metacognitive strategies	Instructional strategies	Examples		Interactive media type
		Questions	Feedback
Evaluate (Bannert & Reimann, 2012; Brown, 1987; Gordon, 1996; Meijer et al., 2006; Molenaar & Chiu, 2014; Schmidt & Ford, 2003; Yıldız-Feyzioğlu et al., 2013)	Justify proposed solution (Bulu & Pedersen, 2010; Ge & Land, 2003, 2004; Kauffman et al., 2008; Meijer et al., 2006)	What are your reasons and/or arguments for proposing this solution? (Ge & Land, 2003) Or You selected this solution because of___ (Bulu & Pedersen, 2010)	Contextualized feedback will be provided by the instructor or AI-based Feedback System	Short Answer; Text Entry Box; Notes; Fill-in-the-Blank; Web Objects (AI-based Feedback)
	Assessment of supporting evidence/information (Bulu & Pedersen, 2010)	Is your evidence appropriate to convince someone of your solution?	If learner answers “Yes”, he/she will be prompted to explain why the evidence supports the proposed solution If learner answers “Not sure” or “Don’t know”, he/she will be prompted to return to the last section to re-plan, re-collect and re-evaluate the evidence and solution If learner chooses “Need Help”, he/she will be prompted to discuss his/her question(s) with the Chatbot or peers in the Discussion Forum	Multiple Choice; Checkbox; Radio Button; Backward Button; Notes; Web Objects (Chatbot; Discussion Forum)
	Check with evaluation criteria (Swanson, 1990)	Did your solution meet the pre-set evaluation criteria?	If learner answers “Yes”, message says “Good job!” will appear on the screen If learner answers “Not sure” or “Don’t know”, message says “Please review the evaluation criteria and refine your solution accordingly. You can also discuss your solution with the Chatbot or peers in the Discussion Forum” will appear on the screen	Multiple Choice; Checkbox; Radio Button; Audio Input; Backward Button; Web Objects (Chatbot; Discussion Forum)
	Consider perspectives from all stakeholders (Ge & Land, 2004)	Have you considered the perspectives of different stakeholders? (Ge & Land, 2004)	If learner answers “Yes”, message says “Great!” will appear on the screen If learner answers “Not sure” or “No”, he/she will be prompted to consider perspectives of different stakeholders	Multiple Choice; Checkbox; Radio Button; Audio Input; Backward Button;
	Identify side effects (Ge & Land, 2004)	Did you point out the side effects and approaches to reduce them when presenting your solution?	If learner answers “Yes”, message says “Fantastic!” will appear on the screen If learner answers “No” or “Not sure”, he or she will be prompted to refine the solution with analyzing side effects or discuss the side effects with the Chatbot or peers in the Discussion Forum	Multiple Choice; Checkbox; Radio Button; Audio Input; Backward Button; Web Objects (Chatbot; Discussion Forum)

Designing metacognitive scaffolds to facilitate learner reflection is the last stage in creating computer-based ill-structured problem-solving instruction. The reflect strategy helps the learner develop an understanding of oneself and related events to inform future actions (Sandars, 2009). Guided reflection and feedback could challenge learners while help them develop a deeper understanding of related content area (Sandars, 2009). The rich interactive features of multimedia-enhanced problem-solving instruction offers new means to support those reflection activities (Gordon, 1996; Liu et al., 2012). Relevant scaffolds are presented in Table 6 in Appendix 4, such as “What did you learn from this problem?”, “Can you think of other settings where those skills could be applied?”, and “What are the pros and cons of the chosen solution?” (Bulu & Pedersen, 2010; Ge & Land, 2004).

Table 6

Revised design of computer-based metacognitive scaffolds in reflect stage

Metacognitive strategies	Instructional strategies	Examples		Interactive media type
		Questions	Feedback
Reflect (Meijer et al., 2006; Molenaar & Chiu, 2014; Moreno & Mayer, 2007; Pintrich, 2000; Zimmerman & Campillo, 2003)	Knowledge acquisition (Bannert & Reimann, 2012)	What did you learn from this problem?	Contextualized feedback will be provided by the instructor or AI-based feedback system. In the meantime, learner will also be prompted to discuss the answer with the Chatbot or peers in the Discussion Forum	Short Answer; Text Entry Box; Audio Input; Notes; Web Objects (Chatbot; Discussion Forum)
	Knowledge transfer (Lin et al., 1994; Liu et al., 2012)	Can you think of other settings where those skills could be applied?	Contextualized feedback will be provided by the instructor or AI-based feedback system. In the meantime, learner will also be prompted to discuss the answer with the Chatbot or peers in the Discussion Forum	Short Answer; Text Entry Box; Audio Input; Notes; Web Objects (Chatbot; Discussion Forum)
	Present alternative solutions (Ge & Land, 2003, 2004)	What are other solutions? Why didn’t you choose them?	Contextualized feedback will be provided by the instructor or AI-based feedback system. In the meantime, learner will also be prompted to discuss the answer with the Chatbot or peers in the Discussion Forum	Short Answer; Text Entry Box; Audio Input; Notes; Web Objects (Chatbot; Discussion Forum)
	Identify pros and cons of proposed solution (Ge & Land, 2004)	What are the pros and cons of the chosen solution? (Bulu & Pedersen, 2010; Ge & Land, 2004)	Contextualized feedback will be provided by the instructor or AI-based feedback system. In the meantime, learner will also be prompted to discuss the answer the Chatbot or peers in the Discussion Forum	Short Answer; Text Entry Box; Audio Input; Notes; Web Objects (Chatbot; Discussion Forum)

Media Attributes and Examples

Interactive features in e-Learning authoring tools can be used effectively to design metacognitive scaffolds. The rich affordability of e-Learning authoring tools allows designers, instructors, and other stakeholders to present a problem with utilizing a wide variety of media attributes, such as text, audio, video, simulation, and animation. Interactive media types presented in this framework were identified from a variety of e-Learning authoring tools, and examples were created using Adobe Captivate (see Fig. 2 as an example). Those media types can be selected from the menu bar of e-Learning authoring tools, which includes Text Box, Short Answer, True/False, Multiple Choice, Checkbox, Radio Button, Hyperlinks, Audio Input, and Animation (Articulate Storyline, 2012; Captivate User Guide, n.d.). The following section highlights utilizing those interactive attributes to create computer-based metacognitive scaffolds.

Fig. 2

Adding slides in captivate

Multiple Choice Questions

Multiple Choice is a commonly used feature in e-Learning authoring tools. It provides learner with pertinent feedback based on the selected answer(s). Figure 3 shows an example of monitoring scaffolds using branching function to direct the learner to different learning paths, such as “Go to the previous slide,” “Go to the next slide,” or “Open URL or file.” Much like Multiple Choice, True/False, Checkbox, and Radio Button can be used to fulfill the same purpose with assigning variables in the Advanced Action section in Adobe Captivate.

Fig. 3

Adding advanced actions in captivate

Short Answer Questions

Short Answer Question is regarded as an open-ended question, which prompts the learner to respond to a question briefly with putting their thoughts into words (Pappas, 2015a). A planning scaffold created using Short Answer Question and Button (e.g., Clear and Submit) in Adobe Captivate is shown in Fig. 4. Short Answer is an ideal question format to train learner’s argumentation and justification skills by asking learner to describe different solution paths and explain why they choose a certain solution.

Fig. 4

Sample image of metacognitive scaffolds in justifying solution stage

Text-to-Speech

Text-to-speech feature in e-Learning authoring tools allows instructors and course designers to create audio files from written texts or scripts (Fair, n.d.). Text-to-speech can be used to present a problem or elaborate on a topic in the audio format. It helps instructors and e-Learning developers who are not native English speakers to produce English audio narration. In addition, Text-to-speech is a great tool to design computer-based interactive content for visually impaired learners as well as learners who prefer audio instruction over plain texts.

Drag-and-Drop

Drag-and-drop function is widely acknowledged for creating engaging and immersive e-Learning experiences (Pappas, 2015b). It allows the learner to drag a target and drop it to a correct spot, which could be used to match, sort, and label items. Figure 5 shows an orienting scaffold created using the drag-and-drop feature in Adobe Captivate to differentiate causes and non-causes of air pollution.

Fig. 5

Problem orienting activity designed using drag-and-drop

Web Objects

Web objects, including AI-based feedback systems, discussion forums, and chatbots, can be added to e-Learning courseware. AI-based feedback systems can grade students’ responses using machine learning methods and provides contextualized feedback immediately (Chen, 2004; Warschauer & Grimes, 2008); Discussion forums can also be added to support peer interaction and reflective thinking (Pedro et al., 2012); The integration of a chatbot could provide timely assistance to the learner. A chatbot is also known as conversational agent (Griol et al., 2014; Kerry et al., 2008), or personal coach (Roda et al., 2001).

In addition to multiple choice, short answer, text-to-speech, drag-and-drop, notes, and web objects, other media types, including hyperlinks and highlight box (Jonassen, 2000), matching (Pappas, 2015c), and hotspot (Brooks, n.d.) are also available across e-Learning authoring tools. More complex ill-structured problem-solving learning materials and environments, such as educational games (Pappas, 2015b), can be created with utilizing and combing a variety of media types. Tables 3, 4, 5 and 6 in Appendices together present a revised framework for developing metacognitive scaffolds using e-Learning authoring tools based on experts’ feedback.

Conclusion

This study makes several contributions to the fields of instructional design, distance education, and problem-based learning. Firstly, it bridges the gap between theory and practice by proposing a theory-based framework to guide instructional design practices. Secondly, it highlights the importance of designing ill-structured problem-solving learning solutions to teach critical metacognitive strategies. Thirdly, it provides detailed instructions for practitioners to create computer-based metacognitive scaffolds using e-Learning authoring tools. Lastly, this framework adopts the type II design and development research method that produces generalizable findings (Richey & Klein, 2007, 2014). Instructors and course designers could follow the guidelines to design computer-based ill-structured problem-solving instruction across disciplines.

Expert feedback is instrumental in refining the initial framework. For instance, the expert reviewers suggested including contextualized feedback in the framework. This indicates that educators and course designers need to dive deep into the media attributes of e-Learning authoring tools and further integrate external web applications to offer contextualized feedback to learners. The external web application includes an AI-based feedback system, discussion forum where the learner can receive feedback and continue discussions with the instructor and their peers, or a chatbot that helps the learner develop self-regulation skills via conversation exchange.

In summary, the interactive features of computer-based learning environments proposed in this framework exemplify potential additional ways to leverage media affordances and prior research in support of metacognitive skill development for ill-structured problem solving.