Augmented Reality Application to Develop a Learning Tool for Students: Transforming Cellphones into Flashcards.

OBJECTIVE: Flashcards are one of the most popular and optimized ways to learn factual knowledge and improve memory performance. Students of modern age, who use smart technology and mobile devices in their daily lives, often lack the time and motivation to create and use flashcards effectively. We aim to use the inseparable relationship between university students and their smartphones to create new options for higher education, converting their cellphones into flashcards. We have used this new technology to develop a simple application (app) to convert the smart mobile devices of students into flashcards.
METHODS: We have developed an augmented reality (AR) flashcard application using Unity3D, which requires the user to identify a target image. Once the target image is identified, it can be replaced by any other digital output, i.e., 2D image, 3D models, or videos. We used images of histological sections of oral mucosa, which dentistry students study as a part of an oral biology course.
RESULTS: The AR flashcard application worked on both iOS and Android systems. It was able to detect the target image and replace it with the output image on the device screen.
CONCLUSION: Using this application, students will be able to independently learn and self-test their learning at their own convenience. Instructors can use the application to provide additional study aids for the students. Our application, which is being developed as a pilot project, will be expanded and applied as a learning tool for students studying dentistry at the University of Alberta.

I. Introduction

Flashcards are one of the most popular and effective methods of teaching and learning. Using flashcards to study and retain factual knowledge has proven to be extremely helpful for motivated learners as it promotes active recall, metacognition, and confidence-based repetition [1]. By using flashcards, students can self-pace and self-test their learning and perform confidence-based repetition, a process proven to optimize memory performance [2,3]. Despite the existence of ubiquitous learning technologies, several studies and surveys have shown that flashcards are still frequently used to self-test and monitor learning progress [4]. Two major factors that affect the effectiveness of learning and self-testing using flashcards are the amount of practice and the timing of practice. For this reason, the flash card technique is beneficial only for motivated learners, who are willing to spend time making flashcards according to the learning objective and practicing frequently to strengthen their memory and understanding [1,5-7].

The Educause Center for Applied Research (ECAR) survey on mobile information technology (IT) in higher education showed that students embrace mobile computing devices, such as smartphones, iPads, and tablets, in higher education. Sixty-seven percent of surveyed students who regularly use their mobile devices for academic activities consider that these devices are important to their academic success [8]. The inseparable relationship between postsecondary students and their smartphones has the potential to create new options for higher education, where smartphones and mobility can be used as the basis of an instructional strategy.

Augmented reality (AR) is rapidly gaining popularity as entertainment and an educational tool. This technology is defined as the incorporation of digital information, such as 3D models, images, video, and audio, into a real-world space [9]. Studies show that using an AR interface in education, significantly enhances the learning experience and assists students in improving their understanding of the material [10-12].

The use of AR is supported by Mayer’s cognitive theory of multimedia learning, which states that two or more of each of the modalities of delivery, presentation, and sensory systems will support effective learning [13]. The application of AR as ubiquitous learning tools helps to create authentic environments for the learners. Given that even learners at the postsecondary level have difficulty with 3D thinking, AR may be particularly useful in assisting learners to establish spatial relationships and understand abstract concepts. Early familiarization with spatial learning is particularly important for dentistry and dental hygiene students, who are required to be adept with spatial reasoning.

One of the major foundational science courses for the students studying dentistry at the University of Alberta is on oral biology. In this course, students are required to identify histological slides from various parts of the oral mucosa and to recognize various cell types and cell layers, salivary glands, and duct systems from histological sections of the mouth. During didactic lectures students are presented sample histological slides shown in PowerPoint presentations. However, the few examples that can be presented during limited class time are often not enough to ensure that students have a firm understanding of the structures. Additional materials, including sample questions and practice examples allow students to learn and test their knowledge in their own time. To provide students with additional exemplars and improve their ability to recognize histological sections, we developed a simple smart phone application that integrates AR as a “pilot project” with a limited artifact database.

II. Case Description

1. Methods

Our AR flashcard application was developed on a personal laptop and is usable on iPhones, iPads, and Android mobile devices. The software applications and software development kits (SDK) used include Unity3D (version 2017.4.34f1), Vuforia 7, and Xcode 10.3 for building and testing on an iOS device.

The first step of developing our AR flashcard application was the preparation of the target images. Unity3D requires identification of an image or object as a target image. Once the target image is identified, it can then be replaced by any other digital output, i.e., 2D image, 3D models, animations, or videos. For the histology section of our AR application, photographs taken from microscopic slides from our laboratory microscopes were used as target images.

Next, the target images were incorporated in a Vuforia database and were tested for their augment ability. A developer account was created in the Vuforia Developer Portal to generate the target image database. The images of the histological slides were uploaded in JPG format in the target image database of Vuforia, where these images were automatically scanned and assigned a feature-tracking rating on a scale of 1 to 5. This score is indicative of how well the application will be able to detect the target image. A higher score indicates higher augment ability, i.e., better ability to recognize the target image. Images with the highest scores were used in Unity3D as a target image (Figure 1). The target image size was predetermined as 20 cm, which is equivalent to 0.2 unit in Unity3D.

Figure 1

Preparation of a target image for the augmented reality (AR) application. (A) Image of a histological slide from oral mucosa that was used as a target image for our AR flashcard application. The image of the slide was taken using a light microscope and cellphone camera. (B) The target image was uploaded to the database of Vuforia 7. Yellow cross-signs on the image represent feature tracking by Vuforia. (C) According to the tracking score, the augment ability was calculated and represented as 5 stars for this target image. The image was also assigned a unique target ID.

The third step of the AR application development was to incorporate the AR output image. For this step, we assigned a quadrangle on the target image using Unity3D, rotated it by 90°, and aligned it with our target image. For our AR flashcard application, the output was a labelled image of the cell layers. This output image was then uploaded on the quadrangle that was created on top of the target image (Figure 2).

Figure 2

Application window of Unity3D (version 2017.4.34f1), showing the target image superimposed with the output image (labelled cell layers). The axis of the augmented reality camera is shown on top of the image. The target image was imported from the database created in Vuforia 7. The output image was created in PowerPoint by labelling the layers of the target image. The finalized output was saved in JPG format on a personal computer. The image was then exported to Unity3D as a resource and prepared for Unity3D recognition (indicated by arrows, respectively). A quadrangle was imposed on the target image, rotated by 90°, and aligned with the target image.

The final step was to build the application for an iOS device using Xcode 10.3 (https://developer.apple.com). After the application was built and tested on the laptop, Xcode project files were generated to make the application usable on an iPad and any other iOS device. Once the application was successfully built, it was tested further for proper coding and usability. Approaches that are similar to Unity3D and Vuforia have been used previously and were explained in detail by Lee et al. [14].

2. Results

After development, the AR flashcard application was run on an iPad to assess it. Once the AR flashcard application was successfully built on an iPad, the application was given access to the camera of the device. When the camera is moved on the target image (which is a histological slide in this example), it automatically detects the image and replaces it with our pre-defined “output image” on the iPad screen, which in this case, is the labelled cell layers of the histological section (Figure 3).

Figure 3

Running and testing the augmented reality (AR) application on an iOS device. (A) Original output file created by labelling the cell layers on the target image in PowerPoint. (B, C) The AR flashcard application was able to recognize the target image and instantly replace it with the output on the iPad screen. The application was tested both on printouts (B) and images from the computer screen (C).

We also tested the application with target images from both printouts and computer screen (Figure 3) and both formats were able to produce the AR images equally well.

III. Discussion

AR is being applied in teaching neurobiology, cardiology, and structural biochemistry [9,15-17]. Most of the recent application of AR, virtual reality (VR), mixed, and extended reality aim to provide an immersive and interactive experience, which involves 3D modeling. Developing 3D models with an ability to interact with the user in real time, requires high-throughput computing, programing, and designing skills, which is often lacking in educators who have backgrounds in biology and foundational sciences. The technology of AR detects any real-life image or object as a target image or target object and replaces that target object or image with another digital image, 3D model, animation, or video clip, which the viewer will observe through the camera options of his or her mobile device. We have used this simple basic idea: we use a 2D image as the target image and replaced that target image with another digital image or animated video.

This simple application can be tailored according to any teaching needs. Instructors will be able to provide students with sets of practice questions or concepts that are needed for a specific course and its learning outcomes. This mobile AR application is equally suitable for video and animation files. Instructors will be able to incorporate animations and video clips according to their own needs and objectives. Having the AR flashcard application in their phones and iPads, students will be able to study during their free time, while traveling or on their way to school at their own pace and without any internet connection. Students will be able to self-test their knowledge on a given topic, perform confidence-based repetition to memorize and practice, watch incorporated videos and animations to learn, and self-test physiological or any other cellular mechanisms.

In the School of Dentistry at the University of Alberta, we are planning to develop a database of target images for our AR flashcard application that can be used in oral biology, physiology, and other foundational science courses. The application will be further expanded with more targets and outputs for all iOS and Android devices. We are also aiming to collaborate with several publishers to gain full license to use their images in our target database, so students can have the freedom to choose and study materials from a massive database of resources. To evaluate the effectiveness of our application, questionnaires will be distributed to the students and instructors following a semester. Student summative assessments will be compiled to examine the effectiveness of the tool to improve student performance. Focus groups and/or individual interviews will be conducted to gather further insights from the participants regarding the usefulness and usability of the application and to gather opinions to guide its further improvement.