Lifelog Retrieval From Daily Digital Data: Narrative Review.

BACKGROUND: Over the past decade, the wide availability and small size of different types of sensors, together with the decrease in pricing, have allowed the acquisition of a substantial amount of data about a person's life in real time. These sensors can be incorporated into personal electronic devices available at a reasonable cost, such as smartphones and small wearable devices. They allow the acquisition of images, audio, location, physical activity, and physiological signals among other data. With these data, usually denoted as lifelog data, we can then analyze and understand personal experiences and behaviors. This process is called lifelogging.
OBJECTIVE: The objective of this paper was to present a narrative review of the existing literature about lifelogging over the past decade. To achieve this goal, we analyzed lifelogging applications used to retrieve relevant information from daily digital data, some of them with the purpose of monitoring and assisting people with memory issues and others designed for memory augmentation. We aimed for this review to be used by researchers to obtain a broad idea of the type of data used, methodologies, and applications available in this research field.
METHODS: We followed a narrative review methodology to conduct a comprehensive search for relevant publications in Google Scholar and Scopus databases using lifelog topic-related keywords. A total of 411 publications were retrieved and screened. Of these 411 publications, 114 (27.7%) publications were fully reviewed. In addition, 30 publications were manually included based on our bibliographical knowledge of this research field.
RESULTS: From the 144 reviewed publications, a total of 113 (78.5%) were selected and included in this narrative review based on content analysis. The findings of this narrative review suggest that lifelogs are prone to become powerful tools to retrieve memories or increase knowledge about an individual's experiences or behaviors. Several computational tools are already available for a considerable range of applications. These tools use multimodal data of different natures, with visual lifelogs being one of the most used and rich sources of information. Different approaches and algorithms to process these data are currently in use, as this review will unravel. Moreover, we identified several open questions and possible lines of investigation in lifelogging.
CONCLUSIONS: The use of personal lifelogs can be beneficial to improve the quality of our life, as they can serve as tools for memory augmentation or for providing support to people with memory issues. Through the acquisition and analysis of lifelog data, lifelogging systems can create digital memories that can be potentially used as surrogate memory. Through this narrative review, we understand that contextual information can be extracted from lifelogs, which provides an understanding of the daily life of a person based on events, experiences, and behaviors. ©Ricardo Ribeiro, Alina Trifan, António J R Neves. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 02.05.2022.

Introduction

Background

With the expansive use of technology by humans, such as smartphones and wearable devices accessible worldwide, the acquisition of data about a person’s activity is changing dramatically, allowing the acquisition of a huge amount of different types of data every day in the form of images, locations, and physiological signals. With the rapid development of Internet of Things solutions, these personal data can be applied in a wide range of applications. One such application is lifelogging.

Lifelogging is defined as a form of pervasive computing, consisting of a unified digital record of the totality of an individual’s experiences, which is usually called a lifelogger, captured multimodally through digital sensors and stored permanently as a personal multimedia archive. In a simple way, lifelogging is the process of tracking and recording personal data created through our activities and behavior [1,2].

The idea of storing knowledge and information to provide an auxiliary memory to support people was envisioned by Vannevar Bush [3]. At the end of the Second World War in 1945, Vannevar Bush presented the Memex concept to the world. Memex represented a device in which an individual stores knowledge and information, such as his books, records, and communications, based on association, similar to the brain, and exposes it as a memory aid. Bush also envisioned 2 other devices that have come to life: the minicamera worn on the forehead that would allow users to take photographs from their point of view and a device that would record voice in text format. Remarkably, the use of these 3 devices together would enable what could be considered as the starting point of lifelogging.

With the evolution of digital technologies over the years, solutions to record, store, and organize a lifetime of information and knowledge have become possible, as envisioned by Vannevar Bush. Bush’s vision remains an inspiration for many information retrieval and lifelogging systems. However, the amount of information available to be stored and processed today is difficult to analyze and retrieve. To overcome this problem, a wide range of research fields can be explored, such as image and information retrieval, knowledge extraction, image understanding, sentiment analysis, and data mining just to name a few, which provide solutions to organize, process, and retrieve personal data. These personal data are also named as lifelogs and can be used as surrogate memory within a lifelogging system capable of organizing and managing these lifelogs [2]. Therefore, the extraction of relevant information from personal lifelogs can be used to improve the quality of everyday life for people with memory problems or even used as a digital diary.

The practice of lifelogging has become an important resource of contextual data. Projects such as Digital Eye Glass [4-11], MyLifeBits [12-16], and SenseCam [17,18] were the most relevant in the past. The amount of lifelog data (volume), the different types of data obtained from several sources (variety), and the agility to process the lifelogs and generate the necessary information (velocity) make lifelogging an interesting and challenging big data application [2,19]. For example, Gurrin et al [20] started to analyze the large visual lifelogs that were captured during a period of more than a year. Therefore, it is not surprising that the complexity and interdisciplinary challenges are increasing the attention on the lifelogging subject from the research community.

Objectives

Memory is often compared with a computer as it constitutes an information processing system. Both systems have basic functions such as encoding (input and processing of information), storage (retention of information), and retrieval (obtaining information from the storage) [21]. The loss of information from memory, also known as forgetting, occurs when a failure in encoding occurs owing to interference or other memory errors. Encoding failures can be circumvented through lifelogging. Lifelogs, particularly visual lifelogs, provide context cues that can help recall and recognition [21]. As a result, lifelogging has the potential for supporting memory augmentation, which can be applied to aid memory retrieval not only for people with dementia but also for healthy people.

The world’s population above the age of 60 years has been increasing since 1950, and it is estimated to reach approximately 2.1 billion by 2050 [22]. Consequently, the World Health Organization recognized dementia as a public health priority and proposed a global action plan with several action areas, which includes the development, implementation, and improvement of surveillance and monitoring systems, to improve the functional trajectories of people with dementia, their careers, and families [23]. Considering everything mentioned above, patients with dementia could benefit from a lifelogging application that would work as a digital everyday life journal or as a personal historical record [24].

Lifelogging technologies give us the opportunity to create human digital memories, allowing us to represent and understand every moment of our lives and store this information for further use. However, each memory has specific cues, which can be captured from multiple sources based on our surroundings, such as visual cues, verbal and environmental sounds, locations, and actions, thus providing a large amount of contextual information that requires an interactive software tool to retrieve and explore the memory space. In this narrative review, we have discussed about the several types of personal lifelogs and lifelogging applications used to retrieve these lifelogs.

Methods

Search Strategy

This narrative review [25,26] explored a broad perspective of lifelogging approaches and technologies with the aim of synthesizing and understanding the literature on this research topic. Google Scholar and Scopus databases were used to conduct an iterative search based on a combination of search terms or keywords and appropriated Boolean operators to identify relevant publications.

The following search terms were explored: (lifelog OR lifelogging) AND (visual OR audio OR location OR physical activity OR physiological signal OR dementia). A search period was included for searching the publications within the period of 2008 to 2020. However, to explore a historical view of the research topic, relevant publications before 2008 were manually identified and included. This additional inclusion of potential manuscripts of interest was based on our knowledge of this research topic and the association of authors and references of the publications included previously. Only publications in English were considered.

Inclusion and Exclusion Criteria

A total of 411 search results were screened based on the relevance of their title and abstract. Of these 411 publications, 114 (27.7%) publications were selected for full-text analysis. Of the 114 publications, 31 (27.2%) publications were excluded based on their content, and finally, 113 publications were included in our narrative review after including several other publications through citation searching.

Figure 1 shows a flow diagram with the search strategy that led to the included citations, following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines [27]. Initially, our search resulted in a total of 14,614 articles by searching the keywords lifelog and lifelogging. However, as the number of resulting publications was high, we chose to combine keywords, such as visual, audio, location, physical activity, physiological signs, and dementia. Several duplicate articles were excluded, and we selected 2.81% (411/14,614) of the publications. To further restrict our article selection, several articles were excluded based on the relevance of their title and abstract, number of citations, relevance, and approaches or methods. As a result of this search, 27.7% (114/411) of the publications were selected and fully reviewed. Moreover, 27.2% (31/114) of these publications were excluded based on their content. Finally, based on our knowledge of this research topic and by exploring the publication records of the authors of the selected papers, we included 30 more articles to conclude our manuscript collection process with 113 publications.

Figure 1

Flow diagram of the literature selection process for this narrative review.

Results

Types of Lifelog Data

Overview

Recent technological advances have introduced new types of sensors and devices that allow the compilation of vast archives of personal data. According to our research, the review of the literature proposes that the most prominent data explored in the lifelogging research are images, videos, locations, physical activities, and physiological signals, as presented graphically in Figure 2. In visual lifelogs, data are captured by cameras in the form of images or videos. Although audio is not widely used, the voice of the users or sounds in the environment can be useful data that can be integrated into lifelogging systems. The locations can be understood in 2 different ways, such as GPS locations (longitude and latitude) or physical locations (University of Aveiro, home, work, etc). Currently, devices such as smartwatches, which are wearable devices that incorporate sensors such as accelerometers, gyroscopes, force sensors, and pressure sensors, are frequently used by many people. They enable the extraction of information to monitor physical activities. However, these types of wearable devices also incorporate other sensors capable of recording physiological signals such as heart rate and body temperature.

Figure 2

Main types of lifelog data used in lifelogging identified from our review of the literature.

Table 1 summarizes the types of data used in the selected studies on lifelogging. Description of the several approaches is presented in the following subsections. As seen in Table 1, visual data are the most used owing to its richness and the advances in image processing algorithms that allow the extraction of relevant information from images or video. However, several studies have already been reported on the use of other types of data and multimodal solutions.

Table 1

Studies and types of data used.

Study	Visual	Audio	Location	Physical activity	Physiological signals
Piasek et al [24]	✓ (SenseCam)	—a	—	—	—
Hodges et al [17]	✓ (SenseCam)	—	—	—	—
Doherty et al [28,29]	✓ (SenseCam)	—	—	—	—
Gurrin et al [30]	✓ (smartphone)	—	—	—	—
Pauly-Takacs et al [31]	✓ (SenseCam)	—	—	—	—
Wang et al [32,33]	✓ (SenseCam)	—	—	—	—
Song et al [34]	✓ (Google Glass)	—	—	—	—
Li et al [35]	✓ (SenseCam)	—	—	—	—
Bolanos et al [36]	✓ (Narrative Clip)	—	—	—	—
Talavera et al [37]	✓ (Narrative Clip and SenseCam)	—	—	—	—
Dimiccoli et al [38]	✓ (data sets)	—	—	—	—
Gupta and Gurrin [39]	✓ (OMG Autographer)	—	—	—	—
Fan et al [40]	✓ (Narrative Clip)	—	—	—	—
Garcia del Molino et al [41]	✓ (data sets)	—	—	—	—
Furnari et al [42]	✓ (Looxcie LX2)	—	✓ (semantic)	—	—
Oliveira-Barra et al [43]	✓ (data sets)	—	—	—	—
Ellis and Lee [44]	—	✓	—	—	—
Shaikh et al [45]	—	✓	—	—	—
Shah et al [46]	—	✓	✓ (GPS)	—	—
Yamano and Itou [47]	—	✓	✓ (GPS)	—	—
Ziaei et al [48]	—	✓		—	—
Li et al [49]	—	—	✓ (GPS)	—	—
Tanaka et al [50]	—	—	✓ (GPS)	—	—
Aizawa et al [51], Hori et al [52], and Datchakorn et al [53]	✓	✓	✓ (GPS and semantic)	✓ (multiple sensors)	—
Doherty et al [54]	✓ (SenseCam)	—	—	✓ (smartphone)	—
Hurvitz et al [55]	—	—	✓ (GPS and semantic)	✓ (multiple sensors)	—
Yang et al [56,57]	—	—	—	✓ (multiple sensors)	—
Dobbins et al [58]	—	—	—	✓ (data sets)	✓ (data sets)
Ni et al [59]	—	—	✓ (GPS)	✓ (smartphone)	—
Kim et al [60]	—	—	—	—	✓ (smartwatch)
Choi et al [61]	—	—	—	—	✓ (multiple sensors)
Dobbins and Fairclough [62]	—	—	—	—	✓ (multiple sensors)

aThe study does not use this type of data.

Visual

We can observe from Table 1 that several studies on lifelogs have explored the use of visual data. Visual lifelogs are generally collected in the form of photographic or videographic records to trigger memories. Photographs are the preferable representation of autobiographical memories [63,64]. In recent years, wearable devices capable of capturing images or videos continuously from a personal perspective are increasingly used. Examples of these wearable devices are SenseCam, OMG Autographer, Narrative Clip, Google Glass, and GoPro. In addition to these wearable devices, smartphones with high-quality cameras and other sensors are also an important tool for lifelogging. Gurrin et al [30] explored the use of smartphones as an alternative solution to wearable devices such as SenseCam and described several advantages of using smartphones as behavior monitoring devices.

The ability of devices, such as SenseCam, to improve autobiographical memory was studied on a patient with amnesia [17]. This case study indicated that short-term recall improved owing to the use of SenseCam. Furthermore, the use of SenseCam also enhanced long-term memory of autobiographical events. In contrast to the SenseCam application, the written diary helped to recall events in the short term, but not in the long term. The main challenge that this type of devices, and consequently visual lifelogs, face is the processing of such tremendous amounts of data [18]. It is essential to develop techniques that are able to automatically label, segment, and present relevant images in a meaningful sequence.

Pauly-Takacs et al [31] used the images captured by SenseCam during a walk to assist a boy aged 13 years, with profound episodic memory difficulties in remembering those moments. The result of this experiment shows the ability of the images to support the formation of personal semantic memories and memory rehabilitation. In addition to helping in the improvement of retrospective memory, SenseCam can also be applied to patients with dementia, but as a cognitive stimulation therapy. A case study was conducted with the aim of mentally stimulating the patient and encouraging factual and opinionated communication [24].

It is obvious that visual lifelogs are essential as memory reminders to reconstruct previous life experiences, but these lifelogs can be used in other use cases, such as general lifestyle analysis. Doherty et al [28,29] proposed a method to automatically classify visual lifelogs into different lifestyle traits using images collected by SenseCam. The camera captures details of the individual’s everyday activities, in an approach to build a memory of the past. Moreover, Doherty et al [54] used SenseCam images to complement accelerometry measures to identify behavior type and context information across a range of activity episodes.

It is essential to develop techniques that are capable of summarizing the large number of images collected through visual lifelogging. Similarly, Wang and Smeaton [32] proposed a technique for identifying everyday activities captured using SenseCam. It is worth noting that these findings are consistent with previous literature [2,21]. In terms of daily human activities, a very wide range of semantic concepts can be identified in visual lifelogs. For the same activity, a variety of semantic concepts can be observed across individuals. Wang et al [33] characterized everyday activities and behaviors of individuals based on the detection of semantic concepts that appear in visual lifelogs obtained from events that have been automatically segmented based on the technique introduced in the study by Lee et al [65].

In another study conducted using SenseCam, a day of a user was recorded by taking a photo every 30 seconds [35]. Following the lifelogging process, the user reviewed the collected data and classified the day into 12 events to create a ground truth. This method has the potential to retrieve autobiographical events, enabled by the creation of visual lifelogs. Therefore, the use of a wearable camera along with the methods mentioned in this paper constitutes a promising approach to help people retrieve their memories.

In the study by Song et al [34], several egocentric videos were recorded using Google Glass, which captured the diversity and complexity of different daily human activities from a first-person perspective. These videos were collected from 10 different individuals and contained 13 categories of activities relevant to lifelogging applications. Song et al [34] performed several experiments through which they accurately recognized these activities by adopting the dense trajectory approach.

Bolanos et al [36] proposed a method for creation of visual summaries of a set of egocentric images captured by a wearable camera, the Narrative Clip. This summarization aims to support people with neuronal degradation. Other similar studies have been proposed based on the same methodology of clustering-based event segmentation [37] and summarization using contextual and semantic information [38].

Recently, methods based on deep learning to extract visual concepts from images have grown rapidly, making it possible to automatically extract and annotate visual lifelogs accurately. Gupta and Gurrin [39] proposed event segmentation of visual lifelogs based on 2 different approaches for visual concept extraction and image classification, such as objects and activities. The visual lifelogs were collected using a wearable camera, OMG Autographer.

Fan et al [40] proposed the compilation of a journal using the captions of photo streams acquired through camera-based lifelogs. This type of lifelogging collects a large number of images, which in turn are of low quality, noisy, and ambiguous, as they are taken automatically. In this study, 2 authors used Narrative Clip cameras for 5 months to create a data set.

Most studies that used visual lifelogs collected images or videos and created data sets that often contain very limited data, which results in insufficient data to train machine and deep learning algorithms efficiently. In the study by Garcia del Molino et al [41], a large-scale data set with a first-person perspective was created with >1.5 million images captured by 57 users using a wearable camera to train a visual context predictor. This approach can be used to model daily activities and learn the associations between different scenes.

Furnari et al [42] presented a method for temporal segmentation based on personal locations. This study is very promising because it achieves results that are as accurate as those of other methods in the literature. Oliveira-Barra et al [43] proposed a comprehensive methodology for egocentric photo stream analysis. They performed a summary of autobiographical episodes and a semantic key-frame selection and, finally, implemented text-based inverted index retrieval techniques. The episode temporal segmentation was based on semantic regularized–clustering [38]. This model was applied to a data set, and the results suggest that this system stimulates the memory of patients with mild cognitive impairment; for example, patients with dementia.

Audio

As stated in Table 1, a lifelogging application can also use audio lifelogs, generally captured by wearable audio recorders, smartphones, or video cameras that can record audio for several hours or days using a microphone. In the MyLifeBits project [12,13], Gordon Bell used a wearable microphone to record audio clips and stored them in his personal lifelogs. Ellis and Lee [44] described several practical advantages of using audio lifelogs and conducted experiments with different equipment and techniques. Totally, there are 3 major advantages of using audio lifelogs [44]: audio devices, such as microphones, are less sensitive to positioning or motion than cameras; audio data are smaller in file size than videos or image sequences; and audio archives can provide a wide range of useful information, such as location, activities, people, and words.

Audio lifelogs can provide useful information to lifelogging systems, and human activities are reflected in a rich variety of acoustic events and environmental sound cues. Shaikh et al [45] proposed a method to detect and classify activities of daily living, such as laughing, talking, cooking, and so on, and location of the person, such as inside a train, at home, at school, and so on, from the environmental sound cues. Shah et al [46] proposed a lifelogging system using audio records that included speech, music, and environmental sounds. In large audio lifelogs, manual browsing and searching for events or specific audio clips is time-consuming. Therefore, to deal with several types of audio and build an easy, intuitive, and efficient lifelogging application, a generalized and more complex approach was presented in the study by Shah et al [46].

Other studies have used audio lifelogs to segment and classify them according to several characteristics. For example, Yamano and Itou [47] recorded audio lifelogs using wearable microphones and conducted several experiments that enabled browsing these lifelogs. The audio lifelogs were segmented and clustered into events to classify them according to place, speaker, and time. Ziaei et al [48] proposed an analysis system, which automatically estimates the number of unique people and environments using personal audio records.

Location

Lifelogs based on locations can be recognized in 2 different ways: GPS coordinates, such as longitude and latitude, and physical or semantic locations characterized by the place or environment, such as home, office, or more specific locations such as the University of Aveiro. Literature indicates that GPS tracking devices and wearable devices improve the users’ self-esteem when evaluating the effects on the quality of life [66,67]. It is important to note that in this case, the data from GPS tracking devices were not intended to retrieve memories. Nevertheless, the location information may complement visual lifelogs by identifying where the images were taken. This information is usually expressed as coordinates. Moreover, lifelogs offer the option to register relevant locations under intuitive names such as my son’s house [42,49]. When the user checks her lifelog, both the image and the location are displayed. Thus, the user may recall the corresponding memory more easily, even if no spatial cues are visible in the image.

Li et al [49] proposed a method for relating user activities to their location. The authors used spatial and temporal constraints to infer where the user worked or studied. Although this method does not correctly identify all the activities, the results are promising. Furthermore, the proposed method points to the possible automatic compilation of a journal with the places and activities of everyday life by just using a smartphone, which, in turn, can aid memory retrieval.

In the study by Tanaka et al [50], a method for daily context recognition by recording lifelogs based on GPS location from a smartphone was proposed. The proposed method recognizes the lifelogger’s location and activity as contexts. It can also recognize several contexts at the same location; for example, in a shopping mall, the method can distinguish between shopping, eating a meal, or watching a movie at the cinema. By using a smartphone, the lifeloggers can track their activities over time and observe their daily life in more detail.

Physical Activity

Physical activity is fundamental for human beings and is associated with better general health status and improved quality of life. Accelerometers, gyroscopes, goniometers, force sensors, and pressure sensors enable the collection of diverse information. When strategically placed on the user, these sensors can assess the gait and detect falls [68]. Moreover, these sensors are often incorporated into smartwatches or smart bands to monitor physical activity [40]. In addition to counting steps and estimating walked distance, smartwatches and wristbands can record the heart rate and detect stair climbing, arousal, stress, and excitement through electrodermal activity [21].

Doherty et al [54], following their previous study on event-based segmentation [28] and recognition of human activities [29], proposed the use of accelerometers combined with images from wearable cameras to identify certain physical activity behaviors. In this approach, the accelerometer data determined the event boundaries, and the authors could identify sedentary and light, moderate, and vigorous intensity physical activities.

With the easy accessibility of sensors such as accelerometers, which measure the acceleration forces acting on an object or person to determine the object’s position in space and monitor the movement, Hurvitz et al [55] proposed methods to measure and analyze activity behaviors using data, such as location, activity, and environment, collected from the combination of accelerometers, GPS data, and travel diaries. The authors also provided an interface tool to structure and visualize location and physical activity data simultaneously.

Yang et al [56,57] studied several existing lifelogging physical activity measurement devices and identified some measurement uncertainties in an Internet of Things environment that impact the efficiency and accuracy of lifelogging and health applications.

Several diseases such as obesity, hypertension, and cardiovascular diseases are correlated with insufficient physical activity. Dobbins et al [58] proposed an approach to collect and process data from triaxial accelerometers and a heart rate monitor to classify physical activities, such as lying, sitting, running, working on computer, and walking, into different activity levels. In addition to this classification, a visual interface was provided to display the classification of daily physical activities of the user on a smartwatch.

Recently, Ni et al [59] explored a 2-stage hybrid model to predict human physical activity status from lifelogging data collected by wearable sensors. Their goal was to provide health care analytics to support individual decisions in real-time monitoring and statistical analysis, provide personalized advice to individuals, and ultimately, encourage positive attitudes toward healthy lifestyles.

Physiological Signals

Physiological data are inevitably related to the health care service area. These data have been increasingly used in lifelogs over the years, which can be explained by the expansion of the fitness industry [21,60]. The main physiological data are presented in Figure 3. The most relevant data are heart rate, blood pressure, electroencephalogram, electromyogram, electrocardiogram, blood oxygen saturation, blood glucose, body temperature, and breathing rate [61,68,69]. However, the sensors needed to collect most of these data still have to be incorporated into more practical devices before they become prominent in lifelogging applications.

Figure 3

Summary of the main physiological data. ECG: electrocardiogram; EEG: electroencephalogram; EMG: electromyogram.

Heart rate is related to user activity; therefore, it plays a relevant role; for example, when the intention is to identify user activities from visual lifelogs. In the study by Dobbins et al [58], the use of heart rate information was combined with an accelerometer to detect physical activity and support people with diseases such as obesity. Another relevant biological signal is blood pressure, and similar to heart rate, the respective sensors can be incorporated into wearable devices, particularly smartwatches [69].

Dobbins and Fairclough [62] collected lifelogging data from multiple sources including physiological signals, such as ECC and photoplethysmogram data, and driving data, such as the speed of the vehicle, location, and first-person environment images, to develop several classifiers for detecting stress in real-world driving.

Challenges and Data Sets

Over the past years, the term lifelogging has received significant attention from both research and commercial communities. The events that introduced the lifelogging concept to the academic community were the Association for Computing Machinery Continuous Archiving of Personal Experiences workshops from 2004 to 2006 [70-72]. These workshops were essential for those who previously designed and developed solutions independently [6,44,73,74], share tools and experiences, and lead lifelogging as an emerging research area.

Table 2 presents the most relevant data sets existing in the literature for lifelog research.

Table 2

Data sets.

Data sets	Visual	Audio	Location	Physical activity	Physiological signals
NTCIRa-12 lifelog [75]	✓ (OMG Autographer)	—b	✓ (semantic)	✓ (smartphone)	—
NTCIR-13 lifelog [76]	✓ (Narrative Clip)	✓ (music listened)	✓ (semantic)	✓ (smartphone)	✓ (multiple sensors)
NTCIR-14 lifelog [77]	✓ (OMG Autographer)	✓ (music listened)	✓ (semantic)	✓ (smartphone)	✓ (multiple sensors)
EDUBc [78]	✓ (Narrative Clip)	—	—	—	—

aNTCIR: National Institute of Informatics Testbeds and Community for Information Access Research.

bThe data set does not contain this type of data.

cEDUB: Egocentric Dataset of the University of Barcelona.

In 2016, the first test collection for personal lifelog data was introduced [75], which has been used for the National Institute of Informatics Testbeds and Community for Information Access Research (NTCIR)-12–Lifelog task [79]. It promoted a comparative evaluation of information access and retrieval systems operating over personal lifelogs. The lifelogs in this test collection consisted of images from 3 lifeloggers using the wearable camera, OMG Autographer. It also contained several semantic locations, such as home, work, and so on, and physical activities, such as walking, transport, and running. The data set was anonymized to ensure the privacy of both lifeloggers and individuals by removing identifiable content, such as recognizable faces and absolute GPS locations. The test collection also included a set of topic descriptions, which represent the retrieval and reflection reason of the lifelogger for accessing memories [75].

Consequently, challenges regarding lifelogging started emerging. The First Workshop on Lifelogging Tools and Applications in 2016 [80] aimed to discuss approaches to lifelog data capture, analytics, and applications, thus identifying opportunities and challenges for researchers in this new and challenging area. In 2017, the Second Workshop on Lifelogging Tools and Applications was organized simultaneously with the lifelog evaluation tasks, NTCIR-13 Lifelog-2 Task [76] and ImageCLEFlifelog 2017 Task [81].

The ImageCLEFlifelog 2017 challenge was introduced with the NTCIR-12–Lifelog data set [75], but different subtasks were proposed to the participants. However, in the NTCIR-13 Lifelog-2 Task, the organizers created a new data set based on the requirements of the first test collection for personal lifelog data. In addition to the tasks of NTCIR-12–Lifelog, they addressed 2 different challenges for lifelog data organization and retrieval [76].

Since then, workshops and tasks have been organized to advance research on some of the key challenges: ImageCLEFlifelog challenges [82-84]; Lifelog Search Challenge [85-87], which aims to encourage the development of efficient interactive lifelog retrieval systems; and NTCIR Lifelog Tasks [77]. Over the years, these challenges have focused on creating a comparative benchmark activity for lifelogging applications, and data sets used in each of them are very similar or even the same. These data sets start with the first test collection for personal lifelog data [75], which they extend or improve.

In addition to the data sets used in these challenges, several other data sets containing egocentric data are available [34,42,78,88]. However, most of these data sets focus on different and smaller amounts of data for specific use case applications and not on capturing all the daily activities and behaviors of a lifelogger. An example of these data sets is the Egocentric Dataset of the University of Barcelona (EDUB) [78], which is divided into different sub–data sets depending on the data annotations, such as EDUB-Obj data set for object localization or segmentation [89], EDUB-Seg data set for egocentric event segmentation [37,38], and EDUB-SegDesc data set that can be used either for egocentric event segmentation or for egocentric sequence description [90].

Lifelog Retrieval Software Tools

Throughout the referred challenges and workshops, several applications were presented. The Lifelog Search Challenge has been one of the challenges in which several lifelogging systems have been presented with several utilities for real-world use, unlike other challenges, such as ImageCLEFlifelog challenges, which present very specific tasks.

A retrieval and exploration lifelogging system, called lifeXplore, which allows to search and browse features that have been optimized for lifelog data, was presented by Münzer et al [91]. It was based on a video search system, diveXplore [92-94], previously developed for video retrieval competitions. Besides efficient presentation and summarization of lifelog data, it includes different methods of retrieving and visualizing content, such as feature map, day inspector, lifelog filter, sketch search, and similarity search. Over time, the lifeXplore system was improved by including location-based filtering, automatic feature map browsing, and optical character recognition. Moreover, uniform sampling was used as an alternative method for segmenting videos [95,96].

Other tools obtained from video retrieval competitions are the VIRET tool [97-100], which is an updated version of the SIRET interactive video retrieval tool [101] addressing specific properties of visual lifelogs, and vitrivr [102,103], which was developed for video retrieval [104] and later adapted to support multimodal data [105], such as lifelogs.

Zhou et al [106] proposed an iterative lifelog search engine called LIFER, which is queried based on several different forms of lifelog data, such as visual concepts, activities, locations, time, and so on. Despite some limitations of LIFER, this application allows users to retrieve the moments from their personal life archives in a reliable and efficient manner. Enhanced versions of LIFER, such as LIFER 2.0 [107] and LIFER 3.0 [108], were proposed with additional visual features to solve several tasks of ImageCLEFlifelog 2019 and 2020, respectively. It should be noted that many other applications have been proposed in the challenges and workshops mentioned previously [109-114].

In addition to the mentioned applications, other applications have been incorporated into the context of health care. Health lifelogs focus on medical and clinical perspectives. In this case, lifelogs exploit other sensors to gather information. Physical activity, heart rate, blood pressure, and body temperature are examples of measurements that may be valuable from a clinical perspective [61,115,116]. Lifelogs can be used to create platforms that provide a collection of digital memories in a structured and searchable manner, similar to the DigMem system [117]. Another example of an application is the compilation of a diary based on information extracted from the lifelogs [40].

A recent study introduced the use of lifelog monitoring for the early detection of complications in pregnancy [116]. These lifelogs feature physiological data and self-reported information. The authors aimed to detect physiological changes and, together with the multiomics data, try to understand the mechanisms responsible for pregnancy-related diseases. Kim et al [118] proposed the development of a ubiquitous health care system based on biological and lifelog data. This system was designed to assist the care of patients with chronic medical conditions. A Japanese study discussed the viability of a platform (PeOPLe) containing self-recorded lifelogs and medical records to support health care applications [115]. Each patient should provide lifelogs to the platform to assist the health management of the patients who are old and request physician support based on automatic predictions. Similar to PeOPLe, the study presented by Choi et al [61] identified machine learning and mobile learning as helpful tools to examine big data resulting from lifelogs.

In addition to developing diagnostic and health care systems, as illustrated by the examples mentioned previously, lifelogging can assist the change of lifestyle and behaviors [119]. The awareness provided by self-monitoring encourages users to make healthy choices, and if the progression is noticeable, they feel motivated to continue. This applies to nutrition, physical activity, sports, active travel, and psychological well-being [2,40,64,115,120].

Applications

Lifelogs comprise data of different natures, and consequently, they present an extensive range of possible applications within different use cases, as presented in Table 3. It is noteworthy that work or other procedures may be recorded through lifelogging. An example is the visual lifelogging of a workday by health care professionals [2]. Despite the popularity of wearable lifelogging devices, other sensors can be strategically placed to monitor user activity. These sensors can be used for older people with assisted living needs, and the data acquired by them can be recorded as a lifelog.

Table 3

Applications of the research presented in the selected publications considering 5 major areas.

Study	Daily activities	Event segmentation	Health care	Summarization	Retrieval
Piasek et al [24]	—a	—	✓	—	—
Hodges et al [17]	—	—	✓	—	—
Doherty et al [29]	✓	—	—	—	—
Gurrin et al [30]	✓	—	—	—	—
Pauly-Takacs et al [31]	—	—	✓	—	—
Wang et al [32,33]	✓	—	—	—	—
Song et al [34]	✓	—	—	—	—
Li et al [35]	—	✓	—	—	—
Bolanos et al [36]	—	✓	—	✓	—
Talavera et al [37]	—	✓	—	—	—
Dimiccoli et al [38]	—	✓	—	—	—
Gupta and Gurrin [39]	—	✓	—	—	—
Fan et al [40]	—	—	—	✓	—
Garcia del Molino et al [41]	—	✓	—	—	—
Furnari et al [42]	—	✓	—	—	—
Oliveira-Barra et al [43]	—	—	—	✓	✓
Ellis and Lee [44]	—	✓	—	—	—
Shaikh et al [45]	✓	—	—	—	—
Shah et al [46]	—	—	—	—	✓
Yamano and Itou [47]	—	✓	—	—	—
Ziaei et al [48]	—	✓	—	—	—
Li et al [49]	✓	—	—	—	—
Tanaka et al [50]	✓	—	—	—	—
Doherty et al [54]	✓	—	—	✓	—
Hurvitz et al [55]	✓	—	—	—	—
Yang et al [56,57]	✓	—	✓	—	—
Dobbins et al [58]	✓	—	✓	—	—
Ni et al [59]	✓	—	✓	—	—
Kim et al [60]	—	—	—	—	✓
Choi et al [61]	—	—	✓	—	✓
Dobbins and Fairclough [62]	—	—	✓	—	—
Leibetseder and Schoeffmann [96]	—	—	—	✓	✓
Kovalčík et al [100]	—	—	—	—	✓
Gasser et al [105]	—	—	—	—	✓
Le et al [108]	—	—	—	—	✓
Le et al [110]	—	—	—	—	✓
Ribeiro et al [109]	—	—	—	—	✓
Mai-Nguyen et al [111]	—	✓	—	—	✓
Tran et al [112]	—	✓	—	—	✓
Rossetto et al [113]	—	—	—	—	✓
Khan et al [114]	—	—	—	—	✓
Dobbins et al [117]	—	—	—	—	✓
Karako et al [115]	—	—	✓	—	—
Sugawara et al [116]	—	—	✓	—	—
Kim et al [118]	—	—	✓	—	—
Dobbins and Fairclough [64]	—	—	✓	—	—

aThe computational tool does not focus on this application.

In summary, besides memory assistance, monitoring is the main application of lifelogging in health care. This is specifically relevant for the older population, but not exclusively. In addition, monitoring prompts self-reflection by the user, resulting in the motivation for self-improvement.

Privacy and Concerns

One of the most evident challenges associated with lifelogging is infringement of privacy [2,108]. The nonconsensual logging of bystanders and even the logging of aware friends and family exposes them. A possible solution for visual lifelogs is to blur faces [21,121]. However, when visual lifelogs are used as a memory aid, blurring the faces may hinder this function. Moreover, lifelogs may pose a privacy threat to the surrounding people. Lifeloggers are also susceptible to privacy issues, as lifelogs may constitute valuable information for corporations, including advertisers, which reinforces the necessity of the General Data Protection Regulation. Nevertheless, unobtrusive recording of audio or capturing of images without the explicit consent of everyone involved is prohibited by law.

The use of smartwatches by lifeloggers can be advantageous for recording health data. Kim et al [60] proposed a method to collect data from smartwatches while preserving the user’s privacy. This study is of interest as it attempts to circumvent privacy issues regarding the use of smartwatches. These principles can serve as inspiration for similar approaches for other devices.

Lifelogs may affect our perception of reality; for example, memories may seem more recent than they actually are [119]. Furthermore, despite all the efforts, lifelogs can only capture a small fraction of reality, and as such, only concrete information about subjective experiences can be recorded. Consequently, lifelogs cannot be considered as the ground truth, as there may be failures that prevent full documentation [21].

Another result of our analysis was the permanent character inherent in lifelogs. Although this is advantageous for applications such as memory retrieval, it may become problematic. For example, people with mental illnesses may be obsessed with some memories and dwell on them [7]. Furthermore, even for healthy people, this permanent record may put them under the impression that they are not allowed to change [119]. Therefore, it has been proposed in the literature that lifelogs should try to mimic human memory and implement a forgetting functionality [2,21,64].

Another pertinent concern regarding lifelogging is the possibility that people may rely excessively on lifelogs to remember [119]. This is specifically relevant for future research, as the goal is to enhance the memory of healthy people or improve the memory of people with dementia.

With the popularization of lifelogs and adherence by most of the population, surveillance may become an issue. On one hand, law enforcement may consider lifelogs as a viable method to investigate criminals, which may result in intrusion of the privacy of innocent people [21]. On the other hand, lifelogs may be admitted as proof of innocence. In addition, lifelogs can also potentially empower surveillance by authorities. A legitimate ethical question that emerges from this surveillance is whether illegal behaviors perpetrated by bystanders should be reported by lifeloggers [2].

Discussion

Principal Findings

In lifelogging, devices should be ubiquitous, and data capture should occur without requiring any action on the part of the wearer. Currently, everything and everybody with network connectivity can be turned into sensors that continuously generate data. Mobile and wearable devices have been integrated into everyday activities in a seamless and ubiquitous manner. It has become increasingly possible to remotely monitor behaviors using our smartphones or wearable devices.

Lifelogs are personal data created through life experiences and behaviors of individuals during their daily life, such as images, videos, audio, biometric data, or locations, that are collected by physical sensors. Lifelogs are prone to become a powerful tool to retrieve memories or increase the knowledge about an individual’s experiences or behaviors. However, regarding human digital memories (or personal digital memories), different viewpoints arise. Although some refer to human digital memories interchangeably with lifelogs, it is valid to argue that human digital memories are the result of the processing and organization of lifelogs [2,122,123].

Visual lifelogs are one of the most used data in lifelogging approaches and applications. These lifelogs provide important visual information such as environment, objects, activity, and behavior, which are performed and visualized by the lifelogger. As human beings, we can distinguish this visual information and interpret it to reconstruct a memory that was previously experienced. However, for machines such as our computers, this information is only pixels or numbers, which requires the development of algorithms and methods for the interpretation and analysis of these data to retrieve a specific memory efficiently. One of the main advantages of visual lifelogging is the resulting feeling of security. The users are not worried about remembering because they know that everything is being documented [21]. It should be noted that visual lifelogs are usually accompanied by supplementary information, as illustrated by the examples analyzed in the previous sections. These data can help in memory retrieval, because the richer the lifelogs, the more likely they are to hold relevant cues.

Audio lifelogs are less used in lifelogging applications than visual lifelogs because of the additional challenges that they bring to the application. They can be uncomfortable for the lifelogger. However, audio lifelogs may contain important information for lifelogging applications, such as conversations, speeches, music, or several environmental sounds. Moreover, visual entry lifelogs can take advantage of sound records, as illustrated in the cases mentioned in the Results section. Although audio devices are mainly used as reminder devices, voice records can be used to document important events as the user is experiencing them or shortly thereafter. However, there is a lack of studies on the use of audio lifelogs and their relevance in lifelogging applications for people with dementia.

Location-based lifelogs allow people to retrieve information about the environment and activities that may occur in that location. Regarding memory retrieval, the locations complemented by other information, such as visual lifelogs or temporal features, facilitate the search for these data and make a lifelogging system more accurate [21]. For example, people with dementia tend to lose their ability to recognize familiar places or locations or become lost and confused about their location. Such information can be retrieved together with visual lifelogs and, therefore, stimulate the memory of these people.

Extracting physical activities only from images is a complex process and sometimes inaccurate, because certain objects or scenes can be associated with a wide range of activities. However, lifelog data such as heart rate and accelerometer data can be used to recognize activities of the lifelogger. By using semantic concepts extracted from the images and locations, the classification of these activities can improve significantly. Human physiological signals have several potential benefits in lifelogging applications, such as for health care and daily life monitoring. However, to use a wide range of these data, several sensors are necessary, and most existing lifelogging technologies do not incorporate all these sensors. For example, multiple devices are required to collect these signals from an individual in real time, which becomes challenging for data synchronization and filtering [64].

Physiological data are rarely used in isolation, and generally, these data alone rarely show cues to retrieve memories, particularly in patients with dementia, as their memories are triggered mainly by visual information. The main utility of physiological data in lifelogging is for medical records and physical activity. However, they may also be used to detect emotions, and similar to visual lifelogs, they can form a more complete digital memory [117].

Regarding privacy and concerns, lifeloggers must have access to their data and opportunities to rectify, remove, and control the data that is collected. In addition, lifeloggers should be aware of how their data are stored and used, who owns the lifelogs, and who owns the information obtained from their lifelogs [119]. Gurrin et al [2] assume that the data gatherer owns the lifelogs, which raises the question, “What happens to lifelogs when the correspondent lifelogger dies?” On one hand, lifelogs contain a lifetime of personal information. However, if they are stored in databases, it can help to improve research approaches. Thus, it is necessary to establish regulations on how to approach these concerning issues.

Conclusions

The integration of lifelogging into people’s lives can be beneficial to improve the quality of their life, either by serving as a tool for memory augmentation or by providing support when having memory issues. Lifelogging systems can create relevant digital memories. Through this narrative review, we understand that contextual information can be extracted from lifelogs, which provides an understanding of a person’s daily activities based on events, experiences, and behaviors.

Initially, the scientific community in the lifelogging research field focused their attention on the design and development of solutions or devices capable of acquiring and storing data without interfering with one’s daily life. However, with the increase in wearable devices available for personal data acquisition and the large amount of data to be stored and retrieved, new challenges and issues arose regarding the storage, processing, organization, and retrieval of lifelogs.

An important conclusion of this research exercise is that visual lifelogs are most prevalent when the goal is to create digital memories as surrogate memories. Nevertheless, there is a tendency to associate visual lifelogs with other lifelog data such as audio, location, physical activities, and physiological signals. Audio lifelogs can provide relevant information, such as speeches or environmental sounds, which encode information about locations, activities, and overall context. Along with these personal data, location-based lifelogs can provide additional information. Physical activity and physiological lifelog data are often associated with health care and quality of life. The several sensors that can be incorporated in wearable and easy-to-use devices provide useful information for the recognition and classification of the activities and behaviors of a user. These data used in isolation have some benefits for health care and personal monitoring. Nevertheless, when combined with other lifelogs, they potentially provide important cues to retrieve and form more complete personal digital memories. In addition to creating human digital memories, the acquisition and processing of these lifelogs can be used for monitoring daily life and self-improvement. As they comprise data of different natures, they present an extensive range of possible applications within different use cases. In addition to their relevance in health care, several other applications have been explored such as daily activity analysis, event segmentation, summarization, and information retrieval.

The practice of lifelogging requires tracking and recording of lifelogs in everyday life, for which it is necessary to capture personal data over long periods or even the lifelogger’s entire life. These lifelogs can be combined to develop methods to recognize several contextual data to provide a broader understanding of the lifelogger’s life, such as events, experiences, behaviors, and moments. However, the lifelogs must be synchronized with each other, which can be achieved through time features recorded at the time of lifelog acquisition.

Nevertheless, when these lifelogs are introduced into a lifelogging application, some of them are not relevant or do not contain useful information for further processing and visualization. Therefore, preprocessing methods can be applied to select only relevant lifelogs and remove or correct those that may introduce errors and noise into the system. To retrieve and visualize the previously selected lifelogs, the lifelogging system must be able to interpret these lifelogs in a way similar to that of the lifelogger. Therefore, it is important to annotate, organize, and store the lifelogs with semantic concepts that provide more information about the environment and activities of the lifelogger. These semantic concepts are useful to understand the lifelogger’s behavior and define events and specific moments, which may be required and visualized in the future as surrogate memories.

This narrative review shows that there is a considerable number of published studies on lifelogging. However, we identified several open questions through the analysis and possible lines of investigation in this currently important topic.