The Process of Developing a Digital Repository for Online Teaching Using Design-Based Research.

The Purdue Repository for Online Teaching and Learning (PoRTAL) was developed as an Open Educational Resource (OER) for graduate students and faculty in higher education settings to enhance their online teaching skills and strategies. The PoRTAL team used a design-based research approach (DBR; Wang & Hannafin, Educational Technology Research and Development, 53(4), 5-23, 2005). In this study context, we used Van Tiem et al.'s (2012) model to identify problems faced by instructors who struggled with or were new to online teaching from a Human Performance Technology (HPT) standpoint. To address the identified needs, we created resources for online teaching and embedded our research within practical activities to further study our design process. Our efforts resulted in an HPT-OER Model for Designing Digital Repositories. The purpose of this paper is to share the DBR process that we used to develop an OER repository within an HPT model. © Association for Educational Communications & Technology 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Introduction

The Learning Design and Technology field has seen a rise in research related to teaching and designing for online learning, and a specific need for training graduate students and faculty how to teach online not just from a technological but a pedagogical perspective (Ching et al., 2015; Hodges et al., 2020; Hurtado et al., 2012; Richardson & Alsup, 2015). We undertook a design-based research (DBR) project with a focus on the development of open educational resources (OERs) from a Human Performance Technology (HPT) standpoint responding to this need. Additionally, our DBR approach aligns with a call from Reeves and Lin (2020) encouraging research in our field.

… to consider conducting educational design research to address serious problems related

to teaching, learning, and performance, collaborating more closely with teachers,

administrators, and other practitioners in tackling these problems, and always striving to

make a difference in the lives of learners around the world (p. 199).

With their call to a reawakening of the goals of our field we set out to consider further, “what is the problem, how can we solve it, and what new knowledge can be derived from the solution?” (Reeves & Lin, 2020, p. 198). Through a review of the literature, on-campus generated help sessions, and a campus-wide needs analysis we determined the problem (access to and preferences for training resources) and generated ways to attend to and solve the problem. When considering what “new knowledge” could be derived from the solution we realized that it was not only the results of the DBR but also the process we implemented, including stakeholders. Therefore, this paper, the first in a series, focuses on the pragmatic design and research process we implemented.

Background

Digital repositories are knowledge warehouses where knowledge components are cataloged and stored for reuse, and serve as electronic performance support systems, intelligent help, or reference materials (Lin et al., 2020; Yacci, 1999). Further, open access repositories, or open educational resources, allow instructors to organize, classify, and store digital educational resources and their associated metadata in web-based repositories that can be shared and reused by other instructors (Ferguson, 2017; Friesen, 2009; Lane & McAndrew, 2010). The Purdue Repository for Online Teaching and Learning (PoRTAL) was developed as an OER resource for graduate students and faculty in higher education settings to enhance their online teaching skills and strategies.

The development of PoRTAL was structured through a DBR approach. The DBR methodology allows us to take a “systematic but flexible” (Wang & Hannafin, 2005, p. 6) approach to collaboratively refine educational processes through iterative design. In this study context, we used Van Tiem et al.’s (2012) model to identify problems faced by instructors who struggled with or were new to online teaching from a Human Performance Technology (HPT) standpoint. To address the identified needs, we created resources for online teaching and learning and embedded research within practical activities to further study our design process (Wang & Hannafin, 2005). Our efforts resulted in an HPT-OER Model for Designing Digital Repositories; the purpose of this paper is to share the DBR process that we used to develop an OER repository within an HPT model.

Since many graduate students enter academia or other jobs where teaching is required, PoRTAL began as a course offering to support their development of online teaching skills. After a few approaches were implemented (e.g., a short course, a handbook), we decided that a 24/7, just-in-time, self-paced support resource was necessary. Determined to make the OER repository practical to the target audience, we initiated a design-based project to understand why and how PoRTAL works (Barab, 2014).

Design-Based Research

Design-Based Research (DBR) “is a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually-sensitive design principles and theories” (Wang & Hannafin, 2005, pp. 6–7). Also, “the theoretical framework upon which the design is based may be extended and developed; in some cases, a new framework may emerge,” (Wang & Hannafin, 2005, p. 10). DBR is characterized for being 1) pragmatic, 2) grounded, 3) interactive, iterative, and flexible, 4) integrative, and 5) contextual. It is grounded and pragmatic because it looks to inform theory based on real-world settings. DBR also allows researchers to collaborate with participants in the process, allowing enough flexibility to change and revisit steps when necessary. Researchers purposefully integrate multiple methods to answer their questions. Finally, DBR’s findings are based on the context and provide enough details for others to adapt to their contexts (Wang & Hannafin, 2005).

For a systematic and rigorous implementation of DBR, Wang and Hannafin (2005) describe nine essential principles:

“Support design with research from the outset”: DBR needs to ground the process in the literature. For example, choosing a theoretical framework that meets the needs of the context.

“Set practical goals for theory development and develop an initial plan”: Determining attainable goals and boundaries for the project that converge to the theoretical contribution of the project.

“Conduct research in representative real-world settings”: Such settings need to be prototypical contexts that include the complexity of the environment.

“Collaborate closely with participants”: Participants and researchers become co-creators of the design product. Yet, researchers must be aware of potential influences on participants' opinions.

“Implement research methods systematically and purposefully:” To do so, researchers may draw from multiple sources of data (e.g., surveys, document analysis, observations). Careful documentation of methods and data collection employed assures transparency in decision-making.

“Analyze data immediately, continuously, and retrospectively:” Two levels of coded data emerge, one that captures immediate impressions, and a second that retrospectively distills information from the first level.

“Refine designs continually:” The flexible characteristic of DBR allows researchers to modify their design plans informed by the context and the theory.

“Document contextual influences with design principles:” Use design frameworks to inform the design continually, and vice versa, the design framework enlightens theory.

“Validate the generalizability of the design:” Through collaboration with participants, DBR escalates the design to serve others in similar contexts.

Human Performance Technology Model

Our team used a Performance Improvement/Human Performance Technology model (Van Tiem et al., 2012) as a theoretical framework for our design-based research process, which was adapted and modified as we learned more about the context and specific needs of graduate students and faculty. As an outline strategy to conduct design-based research (Wang & Hannafin, 2005), the HPT model guided the design, development, research steps, design setting, research participants, and research methods, and was flexible enough to allow refinements in the design process. The HPT model includes performance analysis of need or opportunity; intervention selection, design, and development; and intervention implementation and maintenance with evaluation integrated into each phase (Dessinger et al., 2012). These five components are framed within the change management process in which change is considered as a normal occurrence in all phases of the intervention. The HPT model was needed as it “represents a unifying process that helps accomplish successful change, create resiliency and sustainability,” and improve workplace processes (Dessinger et al., 2012, p. 10). Therefore, grounded in a DBR approach, the HPT model allowed us to initiate and sustain our change management initiative. Moreover, like the dynamic DBR process, the HPT model is systematic, iterative, and flexible.

Open Educational Resources

Open Educational Resources (OERs) are “technology-enabled, open provision of educational resources for consultation, use and adaptation by a community of users for non-commercial purposes” (UNESCO, 2002). As the PoRTAL repository was intended to be an OER, we ensured that the 5Rs of OERs (Wiley, n.d.): Retain, reuse, revise, remix, and redistribute, were incorporated into the resources following the DBR process within an HPT model. In Open Educational Resources (OERs), “open” refers to the materials being “licensed with copyright licenses that provide permission for everyone to participate in the 5R activities” (Wiley & Hilton, 2018, p.134). Wiley (n.d.) describes the 5Rs as follows:

Retain—make, own, and control a copy of the resource (e.g., download and keep your own copy)

Revise—edit, adapt, and modify your copy of the resource (e.g., translate into another language)

Remix—combine your original or revised copy of the resource with other existing material to create something new (e.g., make a mashup)

Reuse—use your original, revised, or remixed copy of the resource publicly (e.g., on a website, in a presentation, in a class)

Redistribute—share copies of your original, revised, or remixed copy of the resource with others (e.g., post a copy online or give one to a friend)

The incorporation of the OER principles is connected to the intervention implementation and maintenance phase of the HPT model. All the OERs created for PoRTAL were licensed under a Creative Commons BY-NC-ND 4.0 International License. This license requires attribution of the materials to the creators (i.e., BY) only for non-commercial use (i.e., NC), and adaptations of the work are not permitted (i.e., ND) (Creative Commons, n.d.). Ultimately our project resulted in an online digital repository of OERs, freely available to users, with a focus on online teaching strategies, policies, and procedures.

The PoRTAL Project

We began the PoRTAL project by conducting a needs analysis of prospective users—graduate students and faculty in higher education settings who were new to online teaching and learning. To enhance their online teaching skills and to provide practical tools and strategies we proceeded to design the repository, develop the repository, implement it, and evaluate it at various stages. We based these steps on the five characteristics (i.e., pragmatic, grounded, interactive, iterative and flexible, integrative, and contextual) of the DBR process as defined by Wang and Hannafin (2005, pg. 8), and further operationalized these steps through the Human Performance Technology model proposed by Van Tiem et al. (2012). From a DBR standpoint, as instructional designers, we were trying to solve a real-world problem (pragmatic) that is faced by new online instructors by designing a repository of resources to “inform and improve practice.” We defined the problem through the performance analysis of need or opportunity phase of the HPT model, which includes both organizational and environmental analysis, gap analysis, and cause analysis. DBR grounded the process in instructional design principles, instructional design models, and theories and the “design [is] conducted in real-world settings.” To reflect this, we utilized the HPT model to guide us throughout the development and maintenance of the repository. Following the third characteristic of DBR, our study process was interactive, iterative, and flexible. This is aligned with the HPT model as interactions between partners is encouraged in the intervention implementation and maintenance phase of the model, and iteration and flexibility are promoted by connecting the evaluation component to each step of the model. For instance, interactive collaboration among researchers and practitioners drove the design and development process. An “iterative cycle of analysis, design, implementation, and redesign” (Wang & Hannafin, 2005, p. 8) enabled us to revise the resources multiple times following several steps of evaluations to match user needs and to maintain the design uniformity of the resources. The DBR process that is implemented through the HPT model allowed this flexibility to improve the resources in the repository. It was also “integrative” and to “maximize the credibility of ongoing research” data was collected using both qualitative and quantitative methods from multiple sources, another tenet of DBR (Wang & Hannafin, 2005). These multi-faceted data collection procedures also ensure that evaluations occur at multiple levels in light of the HPT model. Finally, we documented the research process, findings, and changes we made during the DBR process, and provided guidance for using the repository making it contextual. Generating the HPT-OER model is an outcome of the process and further strengthens this contextualization in that it provides guidelines to those who aim to create similar repositories. The following seven steps were implemented in the DBR process through the HPT model and outlined an “iterative cycle of analysis, design, implementation, and redesign” (Wang & Hannafin, 2005, p.8):

Needs Analysis – a survey of faculty members and graduate students pertaining to their online teaching experiences and needs; topics were determined for resources

Design & Development—topics were designed and developed as (just-in-time) resources

Internal Evaluation—by the PoRTAL team (qualitative data was collected)

Modifications—based on internal evaluation

Expert Evaluation—external evaluation by experts in instructional design and topics (qualitative and quantitative data was collected)

Modifications—based on external evaluations

Intervention implementation and evaluation- advisory board and user feedback (qualitative and quantitative data was collected)

Throughout our DBR approach, we recorded the steps that led to accomplishing each phase of the design process and kept an audit trail of why we made pertinent decisions which are described in the following sections. Figure 1 shows the alignment between our DBR and HPT processes as well as the integration of the OER characteristics.

Fig. 1

Alignment of DBR, HPT, and OER components for the development of PoRTAL

PoRTAL’s Design-Based Research Process

In lieu of technology-only focused training (i.e., learning management systems), Adnan (2018) recommends offering professional development opportunities focused on online teaching strategies and pedagogies to support instructors in the creation of dynamic and discipline-specific learning environments. Further, Adnan (2018) explains that training instructors in online pedagogies creates opportunities for mentoring and coaching that eventually benefit higher education institutions and produce a sense of institutional belonging.

In 2016, our university supported the development of a “Facilitation Guidelines” booklet for online teaching and learning. The initiative produced a comprehensive document to guide instructors on principles of teaching online, characteristics of the online learning community, and online discussions. Although the guide was a starting point, partners across campus discussed the need for a more inclusive resource; one that could reach a broader audience including faculty members and graduate students across the university campuses to enhance online teaching across disciplines. This identified need was met by the launch of the PoRTAL (Purdue Repository for Online Teaching and Learning) digital repository project in 2019.

To ensure instructors’ successful transition from traditional to online environments, involvement and cooperation from multiple stakeholders like administrators, staff, faculty peers, and instructional design teams is essential (Bennett & Lockyer, 2004; Covington et al., 2005; Frey & Donehue, 2002; Hoffmann & Dudjak, 2012). Therefore, a diverse team fashioned from across the university campus joined forces to identify areas where instructors needed the most support. The PoRTAL team was comprised of faculty (n = 3), graduate students (n = 7), and staff (n = 6) from the academic program Learning Design and Technology (LDT), Teaching and Learning Technologies (TLT), the Center for Instructional Excellence (CIE), Digital Education (DE), and the Libraries.

Step 1. Needs Analysis

Hinson and LaPrairie (2005) suggest using examples of other institutions as guidance to foresee and address faculty needs when moving from traditional teaching to online teaching. Therefore, the project began with the gathering of potential topics based on the literature and team members’ own experiences. Next, a review of other available repositories was conducted and a gap was determined; most repositories provided limited or outdated resources, some focused on the tools or training for particular platforms, while others lacked quality content, resources, and methods. Several repositories for online teaching and learning were found to be of high quality and are referenced within the PoRTAL resources.

Subsequent to our review of current repositories, the needs analysis was conducted at the university’s main campus with both faculty and graduate teaching assistants. An online survey was distributed as part of the needs assessment and included (1) background information related to instructors’ experiences (e.g., Have you previously taught a fully online course?) and anticipated duties related to teaching online (e.g., What challenges have you experienced or think you may experience in teaching online?), (2) level of interest in specific topics (e.g., Are you ready to teach online? – Readiness to teach online), and (3) level of interest in specific tools (e.g., Video-making tools (e.g., Camtasia). A total of 174 faculty members and 435 graduate students responded to the survey from across campus. The survey was sent through the official university communication system to all faculty and graduate students regardless of their online teaching experience. We believed that input from experienced, novice, and prospective online instructors was necessary to identify the community’s needs.

Through the “Performance Analysis of Need or Opportunity” part of the HPT model (Van Tiem et al., 2012), we identified a gap between the desired and actual performance; that is, instructors did not apply evidence-based practices to their online teaching. This gap analysis led us to a cause analysis, where the following core causes were identified: inadequate tools and resources to support online teaching, lack of awareness of existing resources, limited skills, and knowledge to effectively design, develop, and conduct online courses, and misconceptions about online teaching and workload.

Survey respondents identified three types of relevant support: just-in-time resources with strategies that instructors could access at any time, self-paced training for instructors who sought further strategy development, and face-to-face training to supplement PoRTAL and connect instructors with on-campus support. A digital repository was determined to be the most suitable channel for providing ongoing support to instructors across campus. One might infer that just-in-time resources were the best option because of their high flexibility and instructors’ busy schedules. The results align with Taylor and McQuiggan’s (2008) observations of Penn State’s World Campus, where faculty members reported online and self-paced resources as more effective than face-to-face options. Further, Adnan et al. (2017) found that strict deadlines were a drawback when implementing a faculty development program for online teaching. Given that hard deadlines increased instructors’ anxiety and contributed to dropout rates (Adnan et al., 2017), the use of a self-paced format of instruction was appropriate.

The survey included a starting pool of 29 topics grouped into five categories to determine areas for continued development. Topics were selected based on the PoRTAL team’s areas of expertise and anecdotal evidence of the most common questions from instructors during their transition to online environments. A final set of topics were selected for inclusion in the initial listing of PoRTAL resources (see Table 1) based on level of interest indicated by survey responses. Topics were narrowed down based on the survey responders’ highest level of interest.

Table 1

Initial Listing of the PoRTAL Resources (OERs) n = 28

General Category	Resource Name
Instructor Strategies	Discussion Board Facilitation
	Instructor Role Adjustment
	Online Teaching Persona
	Posting with Intentionality
	Principles for Online Teaching
	Readiness to Teach Online
	Social Presence
	Getting Students Ready for Peer Reviews*
	Use of Technology to Enhance Language Teaching*
	Journaling with Students*
	Case Based Learning*
	Authentic Learning*
Course Design	CDD Online Course Evaluation Rubric
	Community of Inquiry
	Cultural Competence
	Student-Centered Teaching
	Universal Design for Learning (UDL)
	Interactive Synchronous Sessions in Online Courses*
Course Enhancement	21st Century Skills and Online Learning
	Gamification
	Information Literacy
	Open Educational Resources
	Teamwork
Policies & Procedures	Accessibility
	Copyright
	Course Management Strategies
	Strategies for Online Academic Integrity
	Syllabus for Online Teaching

* Indicates Open Educational Resources that were not part of the initial pool of resources but were added after the need for a topic was identified

The results indicated that two levels of resources should be developed (Tier 1 and Tier 2); thus, the creation of PoRTAL began. Tier 1 resources (currently n = 28) serve as just-in-time resources or a gateway for instructors interested in enhancing their current online practices, offering resources, examples, and key principles. The initial pool of resources started with 22 OERs, and the number increased to 28 after new needs for topics were identified (e.g., Interactive Synchronous Sessions in Online Courses during COVID lockdowns). Tier 2 resources (in process) are a set of self-paced tutorials and training modules that offer in-depth information, activities, and opportunities to gain professional development credits.

Step 2. Design and Development of Intervention (OERs)

The Tier 1 (just-in-time) resources are job aids categorized by users’ needs (instructor strategies, course design, course enhancement, and policies and procedures). Content experts among the research team were employed to develop the topics into OERs. Standards for Tier 1 resources were established to best meet users' needs. For instance, Tier 1 resources were intended as succinct, informative, and convenient documents with a maximum length of two pages. Since the content was written for a general audience of instructors, concise definitions were prioritized. The team used EDUCAUSE’s (n.d.) “7 Things You Should Know About” publication as the design concept due to its efficient format, familiarity to the audience, and customizability to meet institutional branding standards. The following headings were provided to guarantee consistency across Tier 1 resources:

What is it?- Provides definitions

Why is it important?- Background information and related research highlights

How to do it?- Steps to implement with a tips and tricks subsection

Tools- Recommendations for multiple tools that can be useful for applying the topic in online courses.

Additional Resources- Additional readings or websites that provide practical information for those interested in learning more about a topic.

References- Citations from within the resource.

The team prioritized the availability and value of Tier 1 resources not only to the home university, but to broader, outside audiences. As such, efforts were made to ensure that free and easily accessible tools, readings, and other resources were provided for those who may not possess a university affiliation.

Steps 3 and 4. Internal Evaluation and Modifications

In DBR, continuous collaboration among researchers and participants helps efficiently resolve emerging issues and participant concerns throughout iterative cycles of design, implementation, analysis, and redesign (Wang & Hannafin, 2005). Throughout the process, “All participants are immersed in the setting and work as collaborators or co-constructors of the design”, which requires close collaboration among the designers/researchers, end-users, and evaluators (Wang & Hannafin, 2005, p. 17). To confirm that the OERs served the purpose of empowering new online instructors, they were initially evaluated by the PoRTAL team comprised of experts in online learning and instructional design (n = 3), end-users who included faculty and administrators (n = 4), and graduate teaching assistants with expertise in either online learning or instructional design (n = 5). Project members were selected based on their years of experience with online teaching, instructional design, and/or their experiences with faculty or graduate student professional development.

Multiple sources of data were collected and integrated to “increase objectivity, validity, and applicability of the ongoing research” (Wang & Hannafin, 2005, p. 10). First, a 15-point checklist (see Appendix A) was created to guarantee that all documents adhered to a particular format. Our team used the checklist to confirm format coherence and provide feedback during iterations within the phase. Formative evaluation in this phase focused on the structure of the document, readability, quality of resources (links checker and resources outside of the university), accessibility standards verification, and appropriate APA references.

Evaluation data also included documentation of initial content and image review to confirm that content was readable for a wider audience beyond the field and university. Furthermore, all related documents (within and across the Tier 1 resources) were reviewed for cross-referencing (hyper-linking) across the various topics. Multiple rounds of internal content and format reviews were conducted by the PoRTAL team members. At first, three graduate students focused on revising content and noting documents that required further work; comments could be as simple as incongruous format or as complex as complete resource revisions. Minor details such as spelling or punctuation were performed without further notice. The internal evaluation group moved between content and format throughout the modification process.

Steps 5 and 6. Expert Evaluations and Modifications

A pool of 10 external experts from outside the PoRTAL team and/or outside the university reviewed the Tier 1 resources. Experts were selected based on publications and years of experience in their area of expertise; four were experts in online learning in general while the other six were experts in a particular topic such as accessibility, copyright, synchronous sessions, and journaling with students. Using guiding evaluation questions provided by the research team, experts reviewed each document for content accuracy and provided feedback in the terms of areas requiring revision, information expansion, and additional resources to be included. Guiding questions focused on the content of each resource across domains of depth, breadth, clarity, and timeliness of information. To identify potential overlaps and gaps across the project topics, reviewers were also asked to view each resource as one part of a whole in the digital repository. Lastly, the format of resources was reviewed with specific regard to readability and citation accuracy.

The external reviewers’ diverse areas of expertise and vast experience resulted in a significant improvement to the Tier 1 resources. For instance, the experts’ input allowed us to amend resources that we previously believed were ready for distribution. Furthermore, documents that required substantial change were identified and rewritten. Despite the fact that feedback was requested for specific resources, the external evaluators’ annotations were also beneficial for the PoRTAL project as a whole. The experts highlighted the need for “how-to” steps that demonstrate the execution of the strategies, and in some cases provided examples of those steps. For instance, one of the experts emphasized the need for hands-on experience and real-life examples for the resources to achieve their full potential:

The ideas presented were good, they are the ones that I would hope you would have included, but it would really expand the value of this job aid if each of these suggested “How to” strategies or an actual example or a video of someone explaining how it could be created/implemented. An example here would be critical. (Expert 1, Comment to Tier 1 resource “21st Century Skills and Online Learning”)

Expert 1’s concern about the need for examples and hands-on experience offers concrete and substantive ideas for the continued development of Tier 2 resources. Tier 2 resources in the form of tutorials and training modules were designed to integrate the knowledge from Tier 1 resources into realistic scenarios for instructors.

Expert 2 helped us realize that some content might be too specific for the audience and therefore lend itself to misunderstandings. For instance, Expert 2 advised us to clarify the reasoning behind our mention of “verbal immediacy behaviors” to promote Social Presence within the Community of Inquiry framework:

This is a really good idea but verbal immediacy behaviors include affective (self-disclosure, values, etc.), cohesive (vocatives, salutations, self-reference, etc.) and interactive (acknowledgement, approval, etc.) behaviors – you could just take out the stuff in parentheses but I am not sure your average reader would understand “verbal immediacy behaviors. (Expert 2 Comment to Tier 1 resource “Community of Inquiry Framework”)

Although the PoRTAL team subject matter experts focused on explaining specific concepts in language understandable to new online instructors, it was necessary to be critical and selective about the amount of jargon used in the documents. Such observations were then used as a guideline for the final evaluative iteration of the OERs. Finally, several external experts were so involved in the feedback process that they were included as authors of the resources that changed significantly following their feedback.

After all experts’ comments were addressed, our team implemented necessary changes. After the resources were approved, they were converted to accessible documents, and the team created tags for each resource as descriptors to guide users through the OER content. The sample tags included course design, discussion boards, getting started, and policies and procedures, among others. Additionally, we linked different OERs using hyperlinks to better reflect the interdependence of topics. For example, the resource “Posting with Intentionality” provides a hyperlink to the “Discussion Board Facilitation” resource.

Step 7. Intervention Implementation and Evaluation

Implementation

We launched the PoRTAL digital repository within our university’s Innovative Learning website dedicated to supporting teaching and learning under the “Supporting Instruction” section (i.e., https://www.purdue.edu/innovativelearning/supporting-instruction/portal/). The PoRTAL website has four sections. First, introductory text describes PoRTAL’s purpose and navigation. In lieu of a menu, users will find four categories (i.e., strategies for online teaching and learning, getting started…thinking about your course design, upping your game, policies and procedures) and tags that categorize the resources. Each OER title shows the tags associated with it as well as a short description of the resource. When users click on the title of the OER, a new tab opens with the two-page document ready for users to download. Finally, at the bottom of the PoRTAL website, users can find the details of PoRTAL team members and contact information.

Considering that PoRTAL is an OER, we will never know its full reach. However, we have participated in several dissemination activities as part of the intervention implementation and maintenance phases of the HPT model, where a marketing strategy targeted at the potential users was developed and implemented. First, we announced the project via a campus-based website to advertise the PoRTAL project designed with faculty members in mind (https://www.purdue.edu/innovativelearning/supporting-instruction/portal/). Next, digital flyers were created to reach the 9000 + on-campus graduate students. We also submitted information to invite users to PoRTAL via an international educational research association repository and presented our project at several professional conferences. Practitioners who were not included in our original target population (e.g., instructional designers) were also reached through a professional newsletter and professional listservs.

The COVID-19 pandemic catapulted PoRTAL’s dissemination and validated its value. A number of professional organizations, higher education institutions, and K-12 institutions included PoRTAL as a resource for online teaching in their own repositories or on websites related to their organizations. For instance, our OER “Syllabus for Online Teaching” was referenced in the “Delivering High-Quality Instruction Online in Response to COVID-19” (O’Keefe et al., 2020) sponsored by the Association of Public and Land Grant Universities, the Online Learning Consortium, and the Bill & Melinda Gates Foundation. Likewise, the Association for Educational Communications and Technology (AECT) recommended PoRTAL in their publication, “The Higher Education Crowdsourced Expert Resources with Commentaries” and referenced PoRTAL as a general resource for online teaching, and also cited specific PoRTAL resources (i.e., Principles for Online Teaching, Discussion Board Facilitation, Instructor Course Postings). The STEM Pedagogy website of the Hispanic Serving Institutions STEM Hub also recommended PoRTAL to instructors who suddenly moved their instruction online due to the COVID-19 pandemic. These references demonstrate the widespread reach of PoRTAL and endorse PoRTAL as a high-quality repository that provides useful and practical advice for experienced instructors and those who are completely new to the topic.

Higher education institutions (e.g., Cornell University, University of Notre Dame, Texas Tech University, University of North Carolina at Charlotte, The Mississippi University for Women, Anatolia College, College of Marin, Wittenberg University, The University of Texas Health Science Center at San Antonio, Shawnee State University) also recommended PoRTAL on their websites to support teaching and learning at their institutions. At the university where this study was conducted, The Office of Engagement has included 10 of the PoRTAL resources in a self-paced tutorial titled: Service-Learning Instructors’ Resource Site. This site is an ever-evolving repository of support resources for faculty and staff as they engage in service-learning pedagogy. Although most institutions referenced PoRTAL as a general resource, some OERs were directly linked within the institution’s website, which indicates the high quality of the resource and the critical need for such topics. Importantly, it achieves the goal of open content.

Although its target audience is higher education instructors, PoRTAL is also referenced as a source for K-12 instructors. For instance, the Indiana GEAR UP project that prepares K-12 students to succeed in higher education references PoRTAL as a general online resource section in their “Guide to e-Learning and Online Resources for Math, Science, Social Studies, English, and World Languages” (Reed, n.d.). However, some sources directly used PoRTAL content without providing references; therefore, proper implementation of the Creative Commons license remains a challenge.

Advisory Board

Following the deployment of the PoRTAL website, a subset of needs analysis survey participants was solicited for feedback in an advisory capacity; the subset of participants had previously indicated their willingness to serve on the Advisory Board via the needs analysis survey. Thirteen instructors and five graduate students joined the final advisory board, and all had experience teaching three or more online courses. Their experience in various disciplines (i.e., science, veterinary, engineering, liberal arts, business) contributed to our purpose of ensuring the content would be comprehensible and self-explanatory for users new to online learning and teaching. Their feedback was collected via Qualtrics and focused on the perceived usefulness and design of PoRTAL. Questions included in the survey included respondents' experience teaching/learning online, their affiliation with the institution (e.g., graduate student, continuing lecturer, associate professor), and the primary college they reside (e.g., agriculture, education, engineering). The feedback questions were open-ended and focused on the application, improvement, and navigation of the PoRTAL website. Sample items included: What challenges do you think you might face while applying the content and teaching strategies from PoRTAL in your online teaching? and Did you face any challenges while navigating through PoRTAL? (Please describe). Respondents were given examples in each question. When answering about what recommendations they had for improving the PoRTAL resources, we saw examples like “additional features within the PoRTAL documents” or “language was too complex.”

One instructor reported that the OERs can support online teaching practices by “strengthening [her] online teaching persona [and] help to utilize the discussion board more effectively.” Another instructor mentioned that the OERs were “Helpful when I am also mentoring others who are teaching online courses with me.” We are optimistic that PoRTAL is a solid first step for offering comprehensive and timely support to new online instructors. Graduate students also perceived PoRTAL to be useful.

“I think these are great just-in-time job aids. The explanation of its use and importance along with the succinct directives, such as those for the rubric and syllabus are really helpful. I especially like the Readiness to Teach Online content as it explains the difference in the role of the instructor and learners. I liked the Faculty Online Teaching Readiness Survey as well. I also felt the UDL page was really helpful.” - Anonymous

The advisory board feedback revealed a need for more practical content like exemplars or reviews of learning platforms. Furthermore, they reported broken links that required updates and sought clarity on some of the technical language. Although we are continuously updating Tier 1 OERs, feedback from faculty and graduate students has supported our efforts and allowed us to recognize our blind spots as a team of online teaching and learning experts.

Repository Visitor User Feedback

Surveys and emails were used to obtain visitor feedback. A brief feedback survey was embedded in the PoRTAL website to collect user feedback. A set of six questions gathered information regarding the perceived usefulness for instructors new to online teaching and learning (e.g., Will you recommend PoRTAL to a friend or colleague?). Visitors to the PoRTAL website can report on content, user experience questions (i.e., Was navigation through the PoRTAL website intuitive?), institutional affiliation (i.e., are you affiliated with [https://www.purdue.edu/innovativelearning/supporting-instruction/portal/] University), and also have the option to leave written comments (e.g., Comments? Feedback?). To date, visitors who responded to the online feedback survey (n = 39) indicated that they would recommend PoRTAL to a friend or colleague and found the resources useful, yet a few also mentioned a desire for further training. For instance:

“I look forward to teaching online and I find this resource page extremely helpful! Thank you! I really appreciate the PDFs that you provided, and I'm wondering if there are any workshops (either f2f or online) that students can attend?” - Anonymous

This comment affirmed our need to continue developing Tier 2 of the project.

User feedback also revealed that visitors outside of academia can benefit from PoRTAL. For instance, a user acknowledged the value of PoRTAL in her corporate work, I’m not a teacher, but I found this info helpful to put into practice within my corporate job. Page bookmarked! Thank you!” (Anonymous).

A PoRTAL project email is also provided to allow for additional feedback on the website and/or resources; our aim is to establish an open communication channel with visitors who have specific requests and recommendations. Although user feedback allows us to improve and redesign our OERs, it is still necessary to measure to what extent visiting the resources translates into application of online teaching and learning.

Feedback from the advisory board, repository visitors, and project email refer to the summative evaluation domain of the HPT model. As an initial flexible and exploratory framework, the HPT model guided the analysis, design, development, implementation, maintenance, and evaluation steps, all of which aligned with the DBR methodology.

General User Analytics

General user analytics are also being collected regularly through the Google Analytics platform. Initial insights show that the PoRTAL project has crossed institutional borders by reaching international audiences. For instance, PoRTAL has reached Colombia, Bangladesh, Australia, Brazil, Costa Rica, Sweden, Armenia, Argentina, Chile, Spain, Honduras, Turkey, New Zealand, South Africa, India, the United Arab Emirates, Canada, Singapore, and the United Kingdom among others. As one user wrote:

“[E]verything is good and makes for a great resource that can be shared with others - even outside of the University.” - Anonymous.

Through July 2021, PoRTAL resources had 9,682 page views, where 8,906 of them were unique views. Although we cannot verify implementation, we obtained the top five most downloaded resources to shed light on what users are most interested in: (1) Discussion Board Facilitation, (2) Online Teaching Persona, (3) Principles for Online Teaching, (4) Course Design Development Rubric, and (5) 21st Century Skills and Online Learning. One can infer that these five topics are among online instructors’ greatest concerns for online teaching, and thus should be further developed.

Discussion

HPT-OER Model for Designing Digital Repositories

We identified a human performance improvement issue and aimed to fill the gap by helping new or struggling online instructors with access to just-in-time resources. We utilized the HPT model (Van Tiem et al., 2012) that was adapted and modified according to the context and gap analysis. As an outline strategy, the HPT model guided the design, development, research steps, design setting, research participants, research methods, and was flexible enough to allow for refinements in the design process (Wang & Hannafin, 2005).

The adapted HPT model for the development of OERs (see Fig. 2) bears implications for practitioners and institutional stakeholders. For example, they can apply this model to develop collaborative solutions to a problem. Apart from focusing on problems, the HPT model also allows practitioners and institutional stakeholders to identify new opportunities through its Need Analysis and Opportunities component to improve current practices. These opportunities may be unknown prior to a needs analysis.

Fig. 2

Performance Improvement/HPT model for the development of PoRTAL

It is the flexibility of the original HPT model that has allowed our adaptation to reach practitioners and stakeholders in other institutions. Unlike a linear model, the HPT model has offered us freedom throughout each iteration of the PoRTAL project to include ongoing analysis and evaluation – a recursive process that has fostered opportunities for change management within our institution. Specifically, change management considers interventions to be anything that will “change the world, workplace, work, and worker” (Dessinger et al., 2012). In our case, PoRTAL was designed to change the campus environment, the workplace for faculty and graduate teaching assistants, their online teaching work, and themselves as online instructors. For instance, it has helped us build partnerships across university units that facilitate faculty work and obtain continuous funding for faculty professional development in online learning. It was the comprehensive nature of the model that allowed us to recognize the potential of PoRTAL for supporting faculty. Overall, we concluded that the HPT model (Van Tiem et al., 2012) is a suitable model to guide a DBR initiative. Its adaptability makes it practical for guiding interventions from gap analysis until evaluation, and, therefore, we posit that multiple, intentionally designed adaptations of the HPT model can attend to the individualized needs of diverse contexts.

Conclusions and Lessons Learned

This study took place in a real world setting where the OERs were developed, reviewed by online instructors to identify areas for improvement, and implemented through the university website on a freely accessible platform. Active collaboration and OER evaluation efforts between experts in instructional design, end users, and the PoRTAL team led to further improvement and refinement of the repository. This ongoing collaboration among the researchers and participants addressed emergent issues and participant concerns as they arose throughout each iterative cycle of the design, implementation, analysis, and redesign phases of the DBR process. Since our central objective was to improve human performance in online teaching and learning, we used the HPT (Van Tiem et al., 2012) model to guide our design and development steps and adapted/modified the model according to our DBR study context and gap analysis results.

The process of developing PoRTAL involved seven major steps. We began with a needs analysis to identify the needs of faculty members and graduate students in terms of online teaching and learning. Then, we designed and developed resources based on the needs and went through several rounds of evaluations to improve resource effectiveness based on feedback from subject matter experts. We also chose the most accessible platform to host and disseminate these resources to the target audience. Finally, we ensured that all resources available on PoRTAL adhered to the 5Rs of OERs. Digital repositories serve as OERs that allow users to Retain, Reuse, Revise, Remix, and Redistribute (Wiley & Hilton, 2018) the resources by implementing the Creative Commons license BY-NC-ND 4.0. We acknowledge the need to better guide users on the correct citation of PoRTAL using the Creative Commons BY-NC-ND 4.0 International License. For example, providing a preferred citation at the top of each OER could be a potential solution to avoid incorrect citations or unintentional plagiarism. The PoRTAL development process that followed the DBR approach helped us to determine best practices for creating OERs that can serve other institutions facing similar projects.

Moreover, we were able to leverage the subject matter expertise and specializations of different university units within the university through an institution-wide collaboration. In addition to helping us validate the OERs, collaboration across university units allowed the PoRTAL team to consider additional ways to enhance online teaching and learning through the incorporation of newly recommended OERs. The collaboration also increased awareness about the existence of these OERs and promoted them across the university.

Despite our best efforts, some users declared that the menu and resource tags were not intuitive. Feedback from users indicated the need for expounded tutorials and resources on new topics, which we are continuously working to improve. For example, our team has designed and developed a Tier 2 set of in-depth, self-paced online modules on some of the topics as well as added new OERs in response to user feedback. Our efforts were made possible by the DBR approach and adaptation of the HPT model to improve human performance in online teaching and learning.

Being a systematic process, DBR enabled our team to collaborate with practitioners in real-world settings to design the OERs. Furthermore, the iterative nature of DBR provided the flexibility to improve at each stage of the HPT model while conducting the needs analysis, and while designing, developing, and implementing the OERs. In addition, the multiple levels of evaluation ensured that the OERs served their intended purpose. We hope that this detailing of our DBR process will guide and support future design and development projects that involve the creation of digital repositories.

We also have come away with recommendations for others interested in following this or a similar process. For the success of a DBR project it is very important to have a strong committed team that is focused on the same goals. The team needs to be flexible and ready to address feedback from the data when revising a digital repository. Revisiting the needs analysis throughout the project to serve as a control for the project scope. Being extremely cautious about plagiarism (intentional and unintentional) via citations while balancing a less academic tone to reach a wider audience is also critical. Finally, understanding the DBR methodology and DBR principles (see nine essential principles under Design Based Research section) assists in the design and development process of digital repositories. The second principle is: “Set practical goals for theory development and develop an initial plan.” Hence, it is important to determine attainable goals and boundaries for the project to make theoretical contributions. In this case, we adapted and modified the HPT model/framework to develop a digital repository of resources that match the context and specific needs of graduate students and faculty. While the seven steps we followed to implement the DBR process through the HPT model lens were effective several of our substeps could have been better clarified earlier on in the process. For example, in future DBR studies we will work to improve our collaboration and communication with technical personnel (i.e., web designer), as applicable, to better reflect our ideas in the end design. Next, the division of work between authors (i.e., subject matter experts) and designers could have been streamlined. Finally, securing buy-in from collaborators by better communicating the purpose of the project earlier on and clarifying that we did not intend to duplicate other departments’ or units initiatives.