Iterative Design and Integration of a Microlearning Mobile App for Performance Improvement and Support for NATO Employees.

This case study details microlearning content development, testing, and implementation of the NATO Headquarters Supreme Allied Command Transformation (HQ SACT) e-Learning Networking App (NeNA) conducted by an AECT microlearning consulting team. NeNA was designed to provide microlearning "just-in-time" content, to create socialization (affinity spaces), channel organizational content, and increase employee motivation. The purpose of the AECT and HQ SACT partnership was to test the NeNA app as a viable way to develop and deliver microlearning content that personalizes self-directed learning, reduces cognitive load, meets the changing needs of the younger workforce, and captures implicit knowledge of employees prior to the end of their military duties. Primary lessons learned surrounded the importance of practitioner awareness of organizational and contextual challenges, including usability of solutions and essential steps to onboarding (training and integration), when adopting mobile learning solutions to provide performance support and improvement to employees. © Association for Educational Communications & Technology 2022, Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Introduction

Given the fast paced, technology driven nature of our society, there are many ways to provide learning opportunities that are flexible, short, engaging, and available when the learner needs them. These approaches entail designing the activities that make use of the microcontent that can be consumed in small segments in multiple settings. Since the early 2000’s, terms such as mobile learning (m-learning), microcontent, and microlearning have grown in popularity. These concepts were initially studied in technology-enhanced learning (Hug, 2005). However, microcontent, microlearning, and m-learning have expanded over the years to many other areas, such as education and professional development (Brion, 2019; Hug, 2010, 2015; Jahnke et al., 2020; Kukulska-Hulme & Viberg, 2018; Pimmer & Pachler, 2014; Torgerson, 2021). This case study illustrates how microlearning was being tested for military professional development and job support.

In today’s workplace, it is critical to find effective, efficient, and immediate ways to get workers the necessary information or skills to complete their tasks with minimum interruptions or to correct their work performance (ATD, 2021; Jahnke et al., 2020; Kohnke, 2021). The COVID-19 pandemic has magnified how critical the world’s need is for accessing technology and knowledge to complete jobs, not to mention how dependent the world is on technology and the knowledge that is transmitted every second of the day (Auxier, 2020; Kohnke, 2021; C. Kumsal, personal communication, September 13, 2021). We have yet to see how the pandemic has permanently changed the three primary business sectors (private, public, nonprofit), and we do know that there are mixed results on generational differences in the way learners learn (Borg et al., 2020; Cretu et al., 2020; Dolasinski & Reynolds, 2020; Lai & Hong, 2015). The learning and development needs of the military are similar to other work environments, especially with how information is transmitted to ensure employees can successfully complete their duties with minimal disruption.

Our workforce will continue to be a mixed generational workforce, where younger employees are more tech-savvy, having been raised with internet accessibility and technology throughout their lives (Anderson & Jiang, 2018; Dimock, 2019; Zhang & West, 2020). This younger group of workers expects to access their knowledge in engaging, short bursts when they need it (Borg et al., 2020; Kakodkar, 2018; Katambur, 2020). To meet the changing needs of the younger workforce and the changes from the pandemic, more workplaces will benefit from implementing mobile learning and supporting their training information with the use of tailored company mobile apps, such as the app that NATO Headquarters Supreme Allied Command Transformation (HQ SACT) tested and implemented in 2021–2022, the NATO e-Learning Networking App (NeNA). Mobile learning can be used in many ways, and microlearning content was an ideal complement for mobile devices and mobile learning (Pimmer & Pachler, 2014) for the NeNA app.

Problem Statement

The high turnover rate for HQ SACT causes a tremendous amount of implicit knowledge to be lost. The lost information, if captured in microcontent, could be a resource for new soldiers who would then experience less of a learning curve in their new roles. The hope is that with access to what previous employees learned in each job, new employees will be able to move the organization forward, rather than the start, stop, start-over pattern that HQ SACT often encounters currently (C. Kumsal, personal communication, April 5, 2021).

It was this turnover pattern and renewed learning curve that HQ SACT staff saw that made them recognize a need for performance improvement. HQ SACT staff felt that they could provide the necessary new employee support through materials that would limit cognitive load while capturing the implicit knowledge of seasoned employees prior to their military duties ending. The overall goals were to personalize individuals’ self-directed learning while decreasing time away from duties and to limit loss of knowledge, as NATO operations are carried out by soldiers who have limited tours of duty (an average of 24 months). One possible solution that arose was mobile microlearning.

Mobile Microlearning

Pimmer and Pachler (2014) discussed how valuable mobile learning technology could be for work-based education. In the human performance field, some might argue that this type of design and instruction is known as “just-in-time-training” or “on-the-job training” (Reiser & Dempsey, 2018; Van Tiem et al., 2012). Because of the flexibility and accessibility of mobile devices, HQ SACT knew that employees could easily retrieve essential information, which drove their decision to make mobile learning part of their performance improvement solution.

Torgerson (2021) explains how mobile devices are important to microlearning for several reasons. The first is that the content on phones is typically microcontent. Second, attention spans are relatively short when using mobile phones. Third, the screen sizes are smaller than other types of electronic devices. Fourth, mobile phones provide multiple opportunities to design content that can be integrated into formal and informal learning processes.

Though there is no common definition for microlearning, many organizations are using microlearning, such as NATO (C. Kumsal, personal communication, February 26, 2021), Berkshire Hathaway Media Group (High, n.d.), Walmart (High, n.d.), and Region’s bank (L. McNeil, personal communication, July 27, 2021). In addition to the large businesses that have implemented mobile learning and microlearning as training methods, the Association for Talent Development (ATD) offers a two-day training certificate for microlearning (ATD, 2021). Badrul Khan, a well-known figure in eLearning and microlearning, was a keynote speaker for NATO’s 2020 Training Technology Conference. He posited that microlearning was “perfectly suited for today’s fast-moving world” (Corbeil et al., 2021, p. 10). Khan’s presentation further influenced NATO’s training division, ACT, to develop and implement the app design from AECT’s 2020 Design and Development Competition.

Based on HQ SACT’s needs for a mobile app for training purposes both in and out of the office, as well as an understanding of microlearning, HQ SACT’s NeNA team chose the below definition and explanation of microlearning for the organization.

Microlearning is a way of teaching and delivering content to learners in small, very specific bursts to drive job performance and personal development. The learners are in control of what and when they are learning. It utilizes quick, informal, self-explanatory presentations, often in the form of videos, but may also include short articles or infographics. While eLearning is delivered on a computer or mobile device, microlearning content is most often accessed with a personal mobile device that is with users at all times. Microlearning provides just-in-time (JIT) content that fills a specific knowledge gap quickly. It is appropriate when the learner needs help doing something specific or reference a snippet of content. This contrasts traditional learning where broader goals are the focus. The bottom line is that microlearning is uniquely effective at promoting knowledge sharing, developing job specific skills and strengthening existing understanding in small pieces. (C. Kumsal, personal communication, July 30, 2022)

Over the years, mobile microlearning research has explored various forms of effectiveness. For example, the following is a sampling of what has been in the literature since 2018: microlearning content (Hogle, 2021; Javorcik & Polasek, 2018; Kohnke, 2021; Khong & Kabilan, 2020; Shail, 2019), designs (Hogle, 2021; Jahnke et al., 2020; Kohnke, 2021; Khong & Kabilan, 2020; Major & Calandrino, 2018; Shail, 2019), models (Dolasinski & Reynolds, 2020; Hanshaw & Hanson, 2019), implementation (Hogle, 2021; Javorcik & Polasek, 2018; Jahnke et al., 2020; Kohnke, 2021; Shail, 2019), and learners’ perceptions (Brion, 2019; Borg et al., 2020; Javorcik & Polasek, 2018; Khong & Kabilan, 2020; Nikou & Economides, 2018). In addition, many authors have concluded that microlearning can effectively prepare learners for complex training (Cretu et al., 2020; Hogle, 2021; Khong & Kabilan, 2020) and reinforcement of learning (Hogle, 2021; Shail, 2019).

Proposed Solution

After discussing the potential benefits, HQ SACT concluded that mobile microlearning was the most appropriate solution for helping reinforce previous training, providing support to learners in a way that was most convenient for them and easily implemented (C. Kumsal, personal communication, April 5, 2021). NeNA’s employee support function would provide a way for learners to access needed information when they did not have time to complete a long course, when they were mobile and dispersed with difficulty reaching a computer, or if there was a need to enhance on-the-job performance. Therefore, HQ SACT announced NeNA as a platform and began building the NeNA team, bringing in AECT consultants to help conduct pilot testing. HQ SACT contracted with a third-party vendor to develop NeNA specifically as a place to store as well as create microlearning content for employees to access anytime, anywhere, even when offline due to being in the field.

Instructional designers are tasked with ensuring that the way learners are presented with new information does not increase the risk of cognitive overload. Therefore, the NeNA team wanted to increase the application of newly learned knowledge, skills, and abilities and ensure not to overload learners’ processing by adhering to cognitive load theory (Sweller et al., 2019). Designers need to make sure that instructional materials and delivery modes adhere to Clark and Mayer’s (2016) cognitive theory of multimedia learning. To ensure a successful pilot test of the NeNA app, HQ SACT staff partnered with members of the Association for Educational Communications and Technology (AECT) to serve as consultants, as they were knowledgeable about these core theories.

The AECT microlearning consulting team consisted of professionals whose knowledge of microlearning design was directed by the three theories identified by Khong and Kabilan (2020): cognitive load theory, cognitive theory of multimedia learning, and self-determination theory. One of the key design features of content is to avoid overloading the cognitive processes (Kohnke, 2021; Major & Calandrino, 2018; Jahnke et al., 2020). Microlearning is ideal because the content is to be simple, focused on one concept only, and delivered in a short timeframe, which is supported by Miller’s (1956) concept of the “magic number plus or minus two,” in which he shared ways to increase the amount of information learners could process by chunking the information. If done properly, the content eliminates extraneous processing, focuses on the essential processing, and ideally will allow for generative or germane processing (Ayres, 2020; Clark & Mayer, 2016; Mavilidi & Zhong, 2019; Sweller et al., 2019; Tufan, 2021). The AECT consultants’ knowledge meant that they could assist HQ SACT staff in designing, creating, implementing, and evaluating instructional content as an alternative or supplement to the lengthier courses available on NATO’s traditional learning platform. Likewise, the expertise within the team using user centered design and the ADDIE model (Branson et al., 1975) empowered the NeNA team to iterate on the app design and the approach to integration and implementation into the HQ SACT training ecosystem.

A primary goal for implementing a mobile microlearning solution was for NeNA to increase learner collaboration and sharing of knowledge by allowing users to create content. Content designed to be microlearning easily aligns with research on how learners are best able to process new information, which will increase the likelihood that employees will apply what is being taught. Research in this area asserts that employees process information using visual and audio channels (Ayres, 2020; Clark & Mayer, 2016; Mavilidi & Zhong, 2019; Sweller et al., 2019). One of the three assumptions Mayer and Moreno (1998) discuss regarding Mayer's cognitive theory of multimedia learning is a learner’s limited capacity. Research about a learner’s limited capacity indicates that they can process from four pieces of information (Cowan, 2001) to seven, plus or minus two (Miller, 1956). Regardless of the actual number of an employee’s information processing capacity, both Miller (1956) and Cowan (2001) stress the importance of chunking new information to prevent cognitive overload.

Design Process

The NeNA app was specifically designed for microlearning content, and it was modeled on the United States military’s adaptive learning app, PERLS. However, the NeNA app is unique in that it has been designed to enable users to create microlearning content to share with other NeNA app users (A. Dolowitz, personal communication, July 24, 2021; C. Kumsal, personal communication, July 26, 2021). The three main components of the app design were socialization (affinity spaces), channeled organization, and motivation.

An affinity space is a community of practice for individuals to learn and interact, thereby leading to a sense of belonging. Affinity spaces begin with some form of content, but participant interactions organize and transform it. Therefore, learning is social and situated, occurring through both interactions and activity, with knowledge and learning distributed among participants (Honeychurch & Patrick, 2018). The advancement of technology has allowed NATO to move communities of practice beyond the walls of traditional workplace interactions. NATO staff shared knowledge within this microlearning affinity space, resulting in socialized learning, personal growth, and increased job performance. The channeled organization allows users to easily find, review, and contribute instructional content. Users select the NATO organization that they are interested in to get content that is related to their needs. The channeled organization prohibits users from receiving all uploaded content, much of which may be irrelevant to their duties. They also have the opportunity to change this selection after using the app for a while. Data collected during the testing phase showed that the users’ number of selected organizations always went down after use, indicating a preference to focus on the content they need to know and related to their job. Users’ homepages contained content based on affinity group member activity and channels they belonged to. Motivational elements encouraged staff members to return for continued participation in the microlearning modules, user-generated content, and affinity groups. Digital badges were provided to serve as validation of accomplishments within the app. The NATO team hoped these digital tokens would bridge the gap between formal and informal learning, shifting the focus from attendance-based certifications to criteria-based accomplishments as seen in real-world performance (North Atlantic Treaty Organization, n.d.).

The NeNA team conducted this iterative process by engaging new users with the app and tracking their use, questions, and comments. The first set of new users were the members of the AECT microlearning consulting team, who went through the initial onboarding process. Based on their experience, the microlearning consulting team recommended changes to make onboarding easier for the ultimate end-users. These changes included managing font sizes, adding descriptive text to useful tools (e.g., text defining the use of tags, titles, and topics), mindfulness of colors for accessibility, and adding a feature for the content creator to view the content in the way the learner would. In addition, the microlearning consulting team created training modules to teach users about the operation of the application to understand the concepts of microlearning. The training content was delivered in presentations at the annual NATO Training Technology Conference (NTTC), including information on the guidelines for creating NeNA content, a checklist that tested the guidelines, and microlearning modules in the NeNA app. Finally, the team gathered feedback in informal expert reviews, surveys, and a field study through the NTTC. Feedback from this conference was used to further improve the design. It was frequently noted that users tried to upload existing instructional or informational content as microlearning content, even when it did not fit the criteria. Changes included clarifying user roles in the system and more clearly expressing the guidelines for effective microlearning.

App Content Development

HQ SACT assigned us the role of “Content Managers” in the NeNA app. This role allowed us to implement our microlearning instructional designer skillset. Microlearning needs to be interactive and engaging, with a user interface design, delivered in short durations either as stand-alone micromodules or part of a whole as just-in-time support to boost retention via device-independent technologies. Furthermore, the evaluation of learning must be outcomes focused. Content managers needed specific instructional design skills to create their own and review user-created microlearning content. These skills include creating compact and user-friendly content, creating guidelines, reviewing user-created content, encouraging social learning and interaction, and understanding and implementing gamification techniques.

The project leader for NeNA was clear about the need to chunk the microlearning content so it could be both stand-alone and be part of a series of training. Those needs align with the instructional importance of chunking, for new information to be broken up into smaller, bite-sized chunks focused on what is relevant to learners (Clark & Mayer, 2016; Sweller, 1994). Additionally, the NeNA project followed the ADDIE model and integrated some gamification elements (Branson et al., 1975; Norsanto & Rosmansyah, 2018). While ADDIE is a linear design process, we used a more user-centered approach to allow for an iterative design and integration process for NeNA, including working with HQ SACT staff and the app developer on how to design the most effective mobile microlearning solution. The AECT consultants began by exploring the initial iteration of the application to understand the current app experience in terms of standard usability heuristics, such as learnability, error prevention, minimalist design, efficiency, and ease of use (Nielson & Molich, 1990), then moved into usability engineering strategies such as usability goal setting, iterative design standards, and user feedback (Mayhew, 1999).

After exploring the alpha version of the project, the team created microlearning content based on existing HQ SACT training materials, then worked with the app developer to improve the user experience. This process elucidated the need for extensive training to prepare end-users on how to develop and upload appropriate content. The next step for the team was to create microlearning content to teach end-users about the NeNA app itself and instructional content on the concept of microlearning.

Content Evaluation

User content was evaluated using a set of guidelines developed by the AECT microlearning consulting team, who served as content managers. Users were also directed to the guidelines as a set of heuristics for the course content development. The creation of the guidelines was grounded in the microlearning literature and recommendations from SACT HQ leadership. The guidelines included the criteria of: courses that are ideally five minutes or less to complete; have one clear learning objective; provide useful on-demand content; utilize multimedia, e.g., spoken text, music, images, animations, or GIFs; are mobile friendly; avoid information overload; are accessible for learners with disabilities; and are intuitively related to the content provided. Content managers tested the guidelines by evaluating the existing classes they had previously created. The results led the team to update the content manager guidelines to align with the learning objective and assessment in the update. The content managers also used this set of guidelines in the live and computer-based training to onboard users to the platform.

NeNA Project Evaluation

In addition to evaluating user content, the AECT consultants were tasked with evaluating the NeNA project to help provide insight on whether the NeNA app should be added permanently as a training resource for NATO staff. The decision was made by viewing analytics within the app, which provided data on the 108 users and their activities (e.g., login frequency, number of courses completed).

The team designed and distributed two surveys. The first survey was a quick gauge of satisfaction with and understanding of the app. The survey was distributed to participants of a five-day NeNA challenge designed by the AECT consultants for NATO’s annual training conference attendees. Although only a few users completed the survey (n = 10), 70% of users reported the app was easy to use, and 80% of users reported the app was easy to navigate. Further, all users stated they would make using NeNA at least a moderate priority after the five-day challenge was complete.

The second survey was distributed to all NeNA users at the end of the pilot program. The participation rate for this survey was higher than the previous attempt (n = 29). This survey aimed to evaluate the usability and relevance of the NeNA microlearning application and discover areas for improvement. Qualitative data revealed that barriers to use were largely related to a lack of content relevant to staff members’ current jobs. Barriers were expected since the app was new, and many divisions had not yet submitted content to be housed within the app. However, 27.6% of users selected maybe and 72.4% of users selected yes when asked if they would use the NeNA app regularly if challenges or barriers were resolved. When asked if they would recommend the NeNA app to their colleagues, 82.8% of users selected yes, 13.8% selected maybe, and only 3.5% selected no. When asked if the NeNA app should permanently become a resource available to NATO staff, one participant failed to respond. Of the remaining participants (n = 28), 85.7% selected yes, 14.3% selected maybe, and 0.0% selected no.

Based on these results, the NATO team determined that the pilot project was successful and worth adopting and adapting as part of HQ SACT’s learning and development efforts. “We based our decision solely on the survey because this was an educational research project on testing the microlearning. The success for us was the adaptation rate of the users instead of the success rate of microlearning in general” (C. Kumsal, personal communication, August 3, 2022).

Discussion (Lessons Learned)

NATO staff have resolved that personnel will benefit from being able to align relevant microlearning content to the tasks at hand without spending more time away from performing other required duties. In addition, it was determined that NATO staff will benefit from taking charge and overseeing their own learning. However, personnel can also share their knowledge with peers and their supervisors, creating a collaborative learning environment.

The NeNA team’s experience reflects Kohnke’s (2021) assertion that mobile microlearning enables the learner to obtain information they need, when they need it, without being overloaded or taking a lengthy refresher course. The challenges and opportunities that were specific to NATO are applicable to other non-governmental agencies. For instance, security clearance will be an issue for companies or organizations that have trade secrets or different levels of security clearances for personnel. All organizations will want to ensure that the information that employees will be sharing (implicit knowledge) is designed to be congruent with a microlearning app. The challenge here will be that a content manager will need the support of their superiors to ensure that regardless of the employee sharing the information, it remains uniform in its design. A content manager will want to have samples of diverse types of microlearning content, such as infographics, podcasts, or videos and guidelines to share with employees to help begin the design process centered on microlearning principles.

Conclusion

As organization-wide mobile apps become more prevalent and another tool that learning and development teams use to deliver informal or formal learning, there will be a need for increased research on how these apps impact human performance. For example, are the learning goals or objectives for delivering learning this way being met? Do instructional designers see more companies do what NATO has sought to do, using an app to capture, organize, and store the employee’s valuable implicit knowledge before it is lost to new assignments, retirement, or turnover? What role will practitioners take in ensuring that companies can capitalize on the informal learning that is often missed because it is informal? These critical questions will be ones that instructional designers need to be prepared to answer for their organizations or clients. Practitioners’ awareness of organizational and contextual challenges will help ensure increased success when adopting mobile learning solutions for employee performance support and improvement.