Comparative analysis of actigraphy performance in healthy young subjects.

Sleep-related health disorders are increasing worldwide; diagnosis and treatment of such sleep diseases are commonly invasive and sometimes unpractical or expensive. Actigraphy has been recently introduced as a tool for the study of sleep and circadian disorders; however, there are several devices that claim to be useful for research and have not been thoroughly tested. This comparative study provides activity, sleep and temperature information regarding several of the most commonly used actigraphers: Micro-Mini Motion Logger; Act Trust; Misfit Flash; Fitbit Flex & Thermochron. Twenty-two healthy young subjects were assessed with five different commercial actigraphs (Micro-Mini Motionlogger Watch, Condor Act Trust, MisFit Flash and Fitbit Flex) and a temperature recorder (Thermochron), and also completed a sleep diary for a week. There were not significant differences in the analysis of rest-activity pattern between devices. Temperature rhythm comparison between the Act Trust and the Thermochron showed significant differences in rhythm percentage (p<0.05) and mesor (p<0.0563) but not in amplitude or acrophase. Although data accessibility and ease of use was very different for the diverse devices, there were no significant differences for sleep onset, total sleep time and sleep efficiency recordings, where applicable. In conclusion, depending on the type of study and analysis desired (as well as cost and compliance of use), we propose some relative advantages for the different actigraphy/temperature recording devices.

Introduction

Sleep disorder diagnosis is usually based on Electroencephalography (EEG) and Polysomnography (PSG) recordings that involve unpractical and invasive sleep assessment, high costs, and usually demand a great effort from the patient [1], [2], [3], [4]. An easier and cheaper alternative to assess circadian sleep disorders is the Daily Sleep Diary in which the person performs a subjective record about his/her sleep habits, including sleep onset and offset. While the Diary is practical to identify general sleep patterns and weekday/weekend differences, it is not a reliable instrument because personal sleep assessment is clearly affected by subjective recollections [1], [5], [6].

In recent years, with the spread of mobile telephony, the ease to connect to the internet wirelessly and the development of online applications, different companies have started to offer portable devices for the monitoring and recording of sleep. While these devices were initially used for investigation purposes, this type of portable technology started to be commercialized so that athletes could personally monitor their activity patterns, including distance and speed [7], [8], [9], [10].

There are currently more than 200 different portable activity trackers, which can be easily purchased. The main objective here is to offer comfortable tools for health measurements (biometrics) and easier data management (information-surveyed records).

Activity or Sleep (activity/sleep trackers) monitors are portable, non-invasive and relatively cheap devices developed to track and quantify different movement patterns through accelerometers that measures the force of inertia generated when a mass is affected by a change in velocity. Actigraphy is based on the monitoring of movements during sleep or activity for long periods of time and, thanks to the use of specific computer algorithms, small devices can offer information regarding individual sleep patterns. Many software applications are able to record, analyze and score physical activity and, in turn, infer sleep/wake characteristics which allow to diagnose and even prevent sleep-related disease [2], [3], [11], [12], [13], [14], [15], [16], [17], [18].

In addition to movement, many of these devices include sensors that can record other variables such as temperature, heart rate, light exposure, oxygen saturation, perspiration, and respiration. Indeed, actigraphy is being increasingly established as the technology of choice for clinicians and circadian/sleep researchers. A recent survey estimated that about one fourth of such studies use actigraphy rather than polysomnography.8.

However, the use of most commercially available actigraphy devices has not been fully validated for clinical or research applications and, in addition, some of the most commonly used motion trackers claim to record sleep accurately, a statement that remains to be tested.

The objective of the present study is to compare different actigraphy and temperature portable devices in the general population. While some devices like Micro-Mini Motion Logger and Thermochron have been commonly used in sleep or circadian research, others (Condor Act Trust, Fitbit Flex or Misfit Shine) have been introduced more recently and have not necessarily been thoroughly tested. Two of these actigraphy monitors (Micro Mini-Motion Logger and Condor Act Trust) have been developed specifically for the assessment of circadian rhythms, while the Fitbit and Misfit devices are motion loggers whose main application is to trace physical activity programs. Our aim was to perform an initial comparison of these technical resources and their relative advantages and weaknesses for sleep and circadian research.

Materials and methods

Participants

A total of 22 (11 males, 11 females) healthy volunteers (21–34 years old) participated in the study. All of them were notified of the study description and objective, and signed an informed consent form. A written instructive portfolio with a detailed description of the study and the devices was given to all participants.

Subjects were asked to complete a demographic questionnaire and a general health questionnaire to discard any physical or psychological disease that might interfere with the study. The Munich Chronotype Questionnaire (MCQ) [19], was used to identify potentially extreme larks or owls.

On the first day of the recordings, the devices were placed and started by one of the investigators, who recovered all apparatus one week later.

Actigraphy and temperature recorders

We initially tested nine commercially available devices, based on diverse recording principles, as follows:

Act Trust (CAT; Condor Instruments, São Paulo, Brazil)

Beddit, (Beddit, Espoo, Greater Helsinki, Finland)

Fitbit Flex, (FF; Fitbit, San Francisco, California, USA)

Hexoskin Smart (Hexoskin, Montréal, Quebec, Canadá)

Micro-Mini Motionlogger Watch (MM; Ambulatory Monitoring Inc., Ardsley, New York, USA)

Misfit Flash (MF; Misfit, San Francisco, California, USA)

Thermochron (TH; Maxim Integrated, San José, California, USA)

Withings Activité Pop (Withings, Issy-les-Moulineaux,Île-de-France, France)

Withings Pulse OX, Withings (Withings, Issy-les-Moulineaux,Île-de-France, France)

Table 1 Summarizes the main features of the nine appliances of our initial screening.

Table 1.

Main features of the nine appliances of our initial screening.

#	Brand	Model	Style	App	Software	Sync	Waterproof	Battery	On display info	Alarm
1	Ambulatory monitoring	Micro-mini Motionlogger actigraph	Watch	No	Yes	IR to USB interface	Yes	Watch Battery 1 DL2430 (125 days)	No	No
2	Beddit	Beddit	Sensor placed on the mattress	Android/ i OS	No	Bluetooth	No	Has to be plugged	No	No
3	Condor instruments	Act trust	Watch	No	Yes	USB using a dock as interface	Yes	USB charge (up to 3 months)	No	No
4	Fitbit	Flex	Wristband	Android/ i OS	Yes	Bluetooth & Cable	No. Only splash.	USB charge	Lights	Yes
5	Hexoskin	Hexoskin Smart	Shirt	Android/ i OS	Yes	Bluetooth	Yes	USB charge	No	No
6	Maxim integrated	Thermochron DS1921G	iButton	No	Yes	Simple Serial Port Interfaces to Most Microcontrollers for rapid data transfer	No	Disposable device	No	No
7	Misfit	Misfit flash	Watch/ Multiwear	Android/ i OS	No	Bluetooth	Yes	Watch battery	Lights	No
8	Withings	Pulse OX	Watch/ Multiwear	Android/ i OS	No	Bluetooth	No	USB charge	On display	Yes
9	Withings	Activité Pop	Watch	Android/ i OS	No	Bluetooth	Yes	Watch battery	On display	Yes

After the pilot testing, five devices were chosen for further research with test subjects. They were chosen based upon the ease of data accessibility, comfort and compliance of the subjects, previous research experience and manufacturers” claim of their validity in sleep and circadian assessment. We also included the Thermochron, an additional temperature recording device which is increasingly used in circadian research [20].

Recordings

Sleep diary

Volunteers were provided with a Daily Sleep Diary where they reported sleep habits including sleep onset and offset starting from the day after the actigraphy recordings were started.

Actigraphy and temperature recordings

Participants were asked to simultaneously use different actigraphs and temperature devices to measure activity and sleep patterns for seven days. They had instructions to remove trackers during activities involving water like taking a bath, washing dishes or swimming, as well as during the practice of specifics sports or during medical studies.

Volunteers were asked to use Both CAT and TH in the same wrist (non-dominant). The TH was placed over the radial artery while CAT was placed on the dorsal surface of wrist. Placement of the other actigraphs was randomly assigned in relation to the wrist. Indeed, although there are changes in overall activity when assessed from actigraphs placed in either wrist, sleep or circadian parameters do not appear to be significantly different [21].

Actigraphic recorders were programmed to collect data in PIM Mode (Proportional Integrating Measure), a measure of activity level or vigor of motion, which considers the area under the curve, and adds that size for each time period.

Individual data collection of each device was as follows:

Micro-Mini Motionlogger Watch (MM; Ambulatory Monitoring Inc., Ardsley, New York, USA): data was downloaded and translated into 1-min sleep–wake epochs using a validated scoring algorithm with Action-W software, version 2.6.9905 software from the same manufacturer.

Act Trust (CAT; Condor Instruments, São Paulo, Brazil). This wristwatch records activity (PIM, TAT, ZCM); sleep (Bed Time, Get Up Time, Time in Bed, Total Sleep Time -h-, Onset Latency -min-, Sleep Efficiency %-, wake after sleep onset -WASO-, number of awakenings); environmental light and skin temperature. Information is downloaded with the Act Studio software (Condor Instruments, São Paulo, Brazil), which allows extracting, visualizing and exporting collected data.

Fitbit Flex, (FF; Fitbit, San Francisco, California, USA). This wristband monitors steps, distance, calories and sleep quality (including bed time. get up time, total sleep time, time awake, light sleep and deep sleep). The subject can log food or water consumption and the device has a silent alarm. Data can be loaded with the Fitbit app for Smartphones via Bluetooth.

Misfit Flash (MF; Misfit, San Francisco, California, USA). This wristwatch measures: activity, steps, distance, calories and sleep (including sleep cycles and sleep quality indicators). Data can be loaded with the Misfit app for Smartphones via Bluetooth.

Thermochron (TH, Maxim Integrated, San José, California, USA). The iButton is a self-sufficient system that measures temperature in the physiological range and stores data that can be downloaded with the OneWireViewer from the same manufacturer.

Table 2 Summarizes the recording characteristics of the five devices under study.

Table 2.

Recording characteristics of the five devices under study.

Brand	Model	Activity	Sampling intervals	Sleep	Auto sleep detection	Light sensor	Temperature	Distance	Calories	Food log
Ambulatory monitoring	Micro-mini Motionlogger Actigraph	PIM, TAT and ZCM	1 min	Yes	Yes	No. Separated.	33°F to 120°F	No	No	No
Condor instruments	Act Trust	PIM, TAT and ZCM	Settable from 1 s to 86,400 s	Yes	Yes	Yes. Red, Green, Blue and Infrared	0–70 °C	No	No	No
Fitbit	Flex	Yes. Steps. Minutes of activities.	15 min	Yes	Yes (From Dec. 2015)	No	No	Yes	Yes	Yes
Maxim integrated	Thermochron DS1921G	No	User-Programmable Intervals from 1 min to 255 min	No	No	No	Yes. From −30 °C to+70 °C	No	No	No
Misfit	Misfit Flash	Yes. Steps. Swim Cycle.	30 min	Yes	Yes	No	No	Yes	Yes	Yes

Statistical analysis

Activity and temperature rhythms were plotted with ActogramJ (v. 0.8) and activity onsets were calculated using the “activity on-and offset” tool [22]. Rhythmic percentage, mesor, amplitude and acrophase were calculated using the Chronos-Fit (v. 1.0) program. For mean comparison One-way ANOVA (with Tukey's multiple comparison test as post test) or T Test was performed. Two-way ANOVA was used to asses if there was any difference between the devices in activity onset during weekdays and weekend days. Finally, to study sleep parameters, the correlation between the devices was calculated using Pearson's correlation and the limits of agreement between measures was asses by Blant – Altman plots. All statistical analysis was performed using SPSS (v. 23).

Results

With the aim of testing some of the most popular actigraph devices available in the market, we asked each participant to simultaneously wear the five devices under study for 7 days. Activity and sleep rhythms were analyzed with the MM, CAT, MF and FF; temperature was assessed using CAT and TH. Due to technical difficulties the number of participants for each device was variable (a total of 15 subjects for MM and CAT, 14 for MF and 4 for FF). Two subjects forgot to log the sleep diary and were excluded for the activity onset analysis.

Activity rhythms

There were no significant differences in the rest-activity pattern recorded by the different devices, as observed in the representative actograms shown in Fig. 1.

Fig. 1

Representative actograms for 5 days (3 weekdays and 2 weekend days, i.e. Wed-Fri., Sat-Sun.) for activity counts extracted from Micro-Mini Motionlogger (MM), Act Trust (CAT), Misfit Flash (MF) and Fitbit Flex (FF) devices. Triangles represent temperature acrophases assessed from CAT and Thermochron TH devices.

The analysis of the activity onset (Fig. 2) in weekdays or weekend days, commonly used for the determination of social jetlag, indicated clear differences for the day of recording (two-way ANOVA p<0.0001) but not between devices, with no interactions between the two factors. To further analyze the activity onset recorded by each device, daily activity data was plotted relative to the activity onset log from the sleep diary. One-way ANOVA (p=0.0128) showed significant differences between the mean relative onset recorded by MM and MF and CAT and MF (p<0.05). A complete analysis of activity rhythms is shown in Table 3, including onset analysis and Cosinor analysis.

Fig. 2

Activity onset calculation. Left panel: Average activity onset for weekdays and weekend from Micro-Mini Motionlogger (MM), Act Trust (CAT), Misfit Flash (MF) and Fitbit Flex (FF) devices. Two-way ANOVA shows only significant differences between weekdays and weekends (***p<0.001). Right panel: Relative activity onset (activity onset for each device – activity onset of the sleep diary) analysis shows significant differences for MF compared with MM and CAT. One-way ANOVA (p=0.0128) followed by Tukey's multiple comparison test (*p<0.05).

Table 3.

Activity rhythms parameters and statistics (mean±SD) for each device.

	Activity
	%Rhythm	Mesor	Amplitude	Acrophase	Onset weekdays	Onset weekend	Relative onset
MM	16.8±5.1	6885±1876	4610±914.5	17.7±1.2	504.9±36.6	740.8±100.3	17.7±8.2
CAT	13.7±3.2	2398±516.7	1575±383.7	17.8±1.4	504.5±37.6	738.5±100.3	18.6±10.1
MF	11.4±5.9	19.4±6.3	18.6±6.5	16.7±1.5	487.5±5	705.5±95.9	38.7±22.3
FF	9.3±4.3	99.3±24.7	81.6±40.6	17.5±0.8	502.8±53.8	748.8±91.7	29.9±3.2
ANOVA p	0,0291*	<0.0001***	<0.0001***	0,2889	0.8922	0.8675	0.0128*
Tukey’s post test
MM vs CAT	ns	<0.0001***	<0.0001***	ns	ns	ns	ns
MM vs Misfit	ns	<0.0001***	<0.0001***	ns	ns	ns	<0.005*
MM vs Fitbit	ns	<0.0001***	<0.0001***	ns	ns	ns	ns
CAT vs Misfit	ns	<0.0001***	<0.0001***	ns	ns	ns	<0.005*
CAT vs Fitbit	ns	<0.001**	<0.0001***	ns	ns	ns	ns
Misfit vs Fitbit	ns	ns	ns	ns	ns	ns	ns

Temperature rhythms

Temperature rhythms were recorded with CAT and TH. The analysis shows significant differences in rhythmic percentage (p<0.05) and mesor (p<0.0563) but not in the amplitude or acrophase of the rhythms (Table 4).

Table 4.

Temperature rhythms parameters and statistics (mean±SD) for each device.

	Temperature
	%Rhythm	Mesor	Amplitude	Acrophase
Th	34.3±2.7	33.6±0.2	1.0±0.09	5.9±0.3
CAT	25.4±1.9	31.8±0.1	0.8±0.08	6.4±0.5
T Test p	0.0111*	<0.0001***	0.0563	0.2573

Sleep rhythms

Sleep onset (MM, CAT and Misfit), total sleep time (MM, CAT and MF) and sleep efficiency (MM and CAT) were analyzed. We found a good correlation (Table 5) between the parameters assessed by each device (Fig. 3). Bland–Altman plots (Fig. 4), which are useful to assess the limits of agreement between these variables, indicate a very good agreement between these devices. There were no significant differences between the devices in the values for average sleep onset, total sleep time and sleep efficiencies (One-way ANOVA p=0.0697; One-way ANOVA p=0.2446 and T test p=0.6433 respectively) (Table 5) (Fig. 5).

Table 5.

Sleep rhythms parameters and statistics (mean±SD) for each device.

	Sleep
	Onset	Total sleep time	Efficiency (%)
MM	1:58±2:36	452.6±111.5	95.70±0.4555
CAT	1:48±2:36	462.9±128.8	95.28±0.7973
MF	1:03±2:36	486.6±167.9
T Test p			0,6433
ANOVA p; F	0.0697	0.2446
Pearson correlation
MM vs CAT (r; p)	0.902;<0.0001	0.607;<0.0001	0.301; 0.003
MM vs Misfit (r; p)	0.469;<0.0001	0.106; 0.3211
CAT vs Misfit (r; p)	0.488;<0.0001	0.351;<0.001

Fig. 3

Sleep onset, total sleep time and sleep efficiency analysis. Top panel: Correlation curves for Micro-Mini Motionlogger (MM), Act Trust (CAT) and Misfit Flash(MF) devices. Sleep onset exhibited very good correlation for all the devices. Middle panel: Correlation curves for Micro-Mini Motionlogger (MM), Act Trust (CAT) and Misfit Flash (MF) devices for total sleep time evidenced very good correlation for all the devices. Bottom panel: Correlation curve for Micro-Mini Motionlogger (MM) and Act Trust (CAT) devices. Sleep efficiency estimates indicated a low correlation between the different devices. Graph insets indicate Pearson's r correlation values and p bilateral significance values (r and p respectively).

Fig. 4

Bland – Altman assessing the limits of agreement between sleep onset (top panels), total sleep time (TST, middle panel) and sleep efficiency (SE, bottom panel). Solid lines represent the mean difference, while dashed lines represent mean difference +/– 2 standard deviations.

Fig. 5

Sleep onset, total sleep time and sleep efficiency analysis. Mean sleep onset (top panel) and total sleep time (middle panel) from Micro-Mini Motionlogger (MM), Act Trust (CAT) and Misfit Flash (MF) analysis indicated no significant differences between the devices (ANOVA (p=0.0697 and p=0.2446 respectively). Lower panel: there were no significant differences for sleep efficiency between Micro-Mini Motionlogger (MM) and Act Trust (CAT) devices (n.s., Student's t test), although ANOVA indicated significant differences in their variance (p<0.0001).

Even though the data recorded by the different devices appear to be reliable and no important significant differences were found for circadian and sleep parameters, the ease of use, participant's compliance and accessibility of the data was certainly very variable, a fact that accounts for the potential choice in research and clinical applications (see Discussion).

Discussion

Some works have studied and compared different actigraphs [8], [10], [11], [12], [23], [24] but, to our knowledge, this is the first to examine and compare five different actigraphs weared simultaneously (two commonly used in sleep and temperature researches: Micro-Mini Motion Logger and Thermochron; two commercial actigraphs: Misfit and Fitbit Flex and a newer one, Act Trust) and their validity for the measurement of activity, sleep and temperature patterns.

It should be stated that a potential limitation of the study is that subjects used the devices for only seven days. This is the maximum period of time that subjects could use the five devices without the need to charge their batteries; indeed, future studies (with less devices) could take advantage of longer periods of data acquisition. Moreover, for a possible clinical application, it is likely that similar durations will be used. In addition, in this study the devices were tested only in healthy people, since our main aim was to compare the different actigraphs features. Indeed, the outcome might be different in specific patients undergoing sleep or circadian disruption.

We found some minor discrepancies between devices. For the physical activity assessment we found significant differences in the estimation of activity onset for MF vs MM and MF vs CAT. This could be attributed to a relatively weak stability of onset estimation with the MF device. Cosinor analysis indicated no differences in acrophase. As expected, the quantitation of activity, assessed by computing mesor and amplitude of circadian rhythms, showed an important difference between the actigraphs, since each of them computes different parameters for measuring activity. While for MM and CAT activity data can be obtained as Proportional Integrating Measure (PIM), Time-Above-Threshold (TAT) or Zero Crossing Mode (ZCM); MF can be only read as “points obtained” and FF data are recorded as “steps”.

Three of the devices (MM, Cat and MF) include algorithms for Sleep Rhythm analysis, no significant differences were found for sleep onset or total sleep time. FF was excluded from the sleep analysis because at the time of the investigation, this device did not include an Auto Sleep Detection function and volunteers usually forgot to set this parameter.

We detected a significant difference between CAT and MM in sleep efficiency estimates. Indeed, this can be attributed to the disparity in the software used for such estimation. On the other hand, MF Misfit do not estimate sleep efficiency.

We also analyzed skin temperature rhythms. As expected, this diurnal variation is in opposite phase to that of central temperature. Analysis of thermal rhythms with CAT or TH indicated relatively minor differences for mesor and amplitude, possible due to differences regarding the placing of each device (i.e. TH is placed in direct contact with the forearm skin while CAT is a clock-like device which can be slightly loose and not fully attached to the skin). No differences were found in amplitude or acrophase, supporting the idea that both devices are equally useful for temperature rhythm research. The peripheral temperature rhythm assessed in the wrist differs from the core temperature rhythm, mainly in phase, for this reason the acrophase values were find at night [25], [26].

In summary, both “professional” (MM, CAT) and “popular” (MF, FF) devices have clear values for potential circadian or sleep research. The two first apparatus are certainly more complete in terms of both recordings and analytical capabilities. However, there is more to be compared than just the analysis of the data. Indeed, during the course of the study, we encountered several difficulties related to data collection. When using FF, the volunteers tended to forget to recharge the devices or to log sleep onset at bedtime. In the MF case, we found that the auto sleep function fails to work when the user retired late at night.

In addition, we encountered major difficulties associated to the format of the data offered by the different software or smartphone applications. Commercially available actigraphs (FF and MF) return activity data as points or steps; indeed, the lack of detail in movement information makes the interpretation of data quite vague and too general. In other words, these devices might represent a good choice as early indicators of circadian or sleep disruption, but for a further and complete analysis of possible disorders, more specific data collection and analysis is needed in order to elaborate strategies and develop tools for self-control [27].

While activity assessment by MM and CAT resulted remarkably similar, the latter offers detailed information about movements (PIM, TAT and ZCM) as activity/sleep indicators, as well as peripheral temperature recordings.

As already stated, several studies have performed comparisons of traditional sleep assessment with either subjective recordings (e.g., sleep diaries) or actigraphic monitoring [1], [2], [3], [4], [5], [6], [7], [8]. The continuous availability of new devices, including commercially available apparatus for the general public, supports the need for their specific analysis in circadian and sleep research. Such devices might also serve educational purposes and provide easy and cheap alternatives for autorhythmometry recordings. In all cases, a complete comparative inquiry into not only the validity of the data, but also their pricing, ease of use, data format and subject compliance, is essential for the correct evaluation of their applicability and significance.