How accurate are the wrist-based heart rate monitors during walking and running activities? Are they accurate enough?

BACKGROUND: Heart rate (HR) monitors are valuable devices for fitness-orientated individuals. There has been a vast influx of optical sensing blood flow monitors claiming to provide accurate HR during physical activities. These monitors are worn on the arm and wrist to detect HR with photoplethysmography (PPG) techniques. Little is known about the validity of these wearable activity trackers. AIM: Validate the Scosche Rhythm (SR), Mio Alpha (MA), Fitbit Charge HR (FH), Basis Peak (BP), Microsoft Band (MB), and TomTom Runner Cardio (TT) wireless HR monitors.
METHODS: 50 volunteers (males: n=32, age 19-43 years; females: n=18, age 19-38 years) participated. All monitors were worn simultaneously in a randomised configuration. The Polar RS400 HR chest strap was the criterion measure. A treadmill protocol of one 30 min bout of continuous walking and running at 3.2, 4.8, 6.4, 8.0, and 9.6 km/h (5 min at each protocol speed) with HR manually recorded every minute was completed.
RESULTS: For group comparisons, the mean absolute percentage error values were: 3.3%, 3.6%, 4.0%, 4.6%, 4.8% and 6.2% for TT, BP, RH, MA, MB and FH, respectively. Pearson product-moment correlation coefficient (r) was observed: r=0.959 (TT), r=0.956 (MB), r=0.954 (BP), r=0.933 (FH), r=0.930 (RH) and r=0.929 (MA). Results from 95% equivalency testing showed monitors were found to be equivalent to those of the criterion HR (±10% equivalence zone: 98.15-119.96).
CONCLUSIONS: The results demonstrate that the wearable activity trackers provide an accurate measurement of HR during walking and running activities.

Criterion-related validity found between all monitors and the criterion measure.

Wearable activity trackers utilising built-in photoplethysmography (PPG) heart rate (HR) sensors have potential to advance the science and practice of physical activity assessment.

The correct placement of the sensor is important to obtain accurate HR from PPG HR sensors.

Introduction

It was in 1982 that the world's first wireless heart rate (HR) monitor consisting of a chest strap transmitter with a wrist-worn receiver was introduced by Polar Electro to give athletes ‘real time’ feedback during exercise.1 During the late 1980s and 1990s, HR monitoring during physical activity (PA) continued to grow in popularity, and by the early 2000s, there were as many as 26 different models available to the recreational athlete.2 In the past 5 years, HR monitoring devices have been combined with other activity monitors, such as pedometers, accelerometers, and global positioning systems (GPS) to provide individuals more accurate estimates of activity intensity and energy expenditure.3–6

Recently, wearable activity trackers have utilised optical blood flow sensing using photoplethysmography (PPG) techniques to measure HR. PPG is a non-invasive method for the detection of HR and is connected with the optical properties of vascular tissue using a probe, usually LEDs. PPG sensors use the probe (eg, LED lights) to shine directly into the skin and interact with changes in the blood volume to configure a HR. HR is determined based on the theory that blood flow through the artery is inversely related to the amount of light refracted.7 PPG techniques using optical LED blood flow sensors have allowed HR monitoring devices to become increasingly popular, with many new models entering the market each year.

By 2012, consumers spent over $800 million on watches, bands, and bracelets7 to monitor HR using a method that has largely not been validated and published scientifically. Given the huge influx, interest, and money spent on these small, non-invasive, and easy-to-use activity monitors, validated research is needed to ensure the activity monitors accurately project HR under resting, light, moderate, and vigorous intensity conditions.3 The six newly developed wearable activity trackers that are currently popular in the market and provide continuous HR utilising optical blood flow HR monitoring capabilities include: Scosche Rhythm (SR), Mio Alpha (MA), Fitbit Charge HR (FH), TomTom Runner Cardio (TT), Microsoft Band (MB) and Basis Peak (BP). The purpose of this study was to evaluate the accuracy of these wearable activity trackers with HR monitoring capabilities during rest and a controlled treadmill protocol.

Methods

Participants

Fifty participants were recruited primarily from the University of Nebraska at Omaha campus. In accordance with the inclusion criteria, participants were between the ages of 19 and 45 years, and engaged in running activities at least three times per week. Volunteers also passed a blood pressure test (below 140/80 mm Hg) preceding the treadmill protocol. All participants filled out a Physical Activity Readiness Questionnaire (PAR-Q) and general health history screening prior to data collection. All participants were given an overview of procedures, potential risk and benefits of the research, and signed the Institutional Review Board approved informed consent document.

Instruments

Criterion measure

The Polar RS400 Heart Rate Monitor Watch (Polar Electro, Kempele, Finland) (PL) was the criterion measure. The PL is geared towards the active endurance athlete who desires to easily and accurately measure HR during exercise. The monitor comes with a WearLink fabric chest transmitter that is easily paired with a wrist receiver. A previous study validated the PL with the ECG.8

Wearable activity trackers with PPG technique HR monitors

With the exception of the SR (worn on the forearm with no screen readout but pairs via Bluetooth or ANT+), all the wearable activity trackers are worn on the wrist. Yellow and/or green LED optical sensors are used to measure the amount of light refracted in the blood vessels utilising the PPG technique. An algorithm is then applied to translate the data from the refracted light into a continuous measure of HR. Description of proper placement and any previous research conducted with each activity tracker is provided below.

Scosche Rhythm (Scosche Industries, Oxnard, California, USA)

Proper placement is on the forearm. A small treadmill study found the performance of the Rhythm to be satisfactory during exercise when compared with the ECG.9

Mio Alpha (Mio Technology, Santa Clara, California, USA)

Proper fit is above the wrist bone and preferably higher when utilising the HR monitor, especially with a small wrist. A small treadmill study found that the MA performed satisfactorily during exercise when compared with the ECG.9

Fitbit Charge HR (Fitbit, Inc, San Francisco, California, USA)

Proper fit is above the wrist bone, and when exercising, at least two fingers width above. No research has been published on the FH.

Basis Peak (BASIS Science, Inc, San Francisco, USA)

Proper fit of the BP is above the wrist bone. No research has been published on the BP.

Microsoft Band (Microsoft Corp., Redmond, Washington, USA)

The MB HR sensor is located on the back of the clasp and can be worn with the face either on the inside or on top of the wrist. No research has been published on the MB.

TomTom Runner Cardio (TomTom International BV, Amsterdam, the Netherlands)

The TT uses the same technology as the MA. Proper fit of the TT is on the wrist area, away from the wrist bone. No research has been published on the TT.

Procedures

Participants had their body height and weight measured via a calibrated physicians scale (SECA, Chino, California, USA) and stadiometer (SECA, Chino, California, USA). Body mass index (BMI) was calculated and blood pressure (while seated) was measured before and after testing procedures using a portable Omron 10 Series Upper Arm Blood Pressure Monitor (OMRON Healthcare, Inc, Lake Forest, Illinois, USA).

The participants were simultaneously fitted with the six wrist-worn wearable activity trackers in a counterbalanced configuration. A visual inspection of the wrist and forearm was conducted, and watches were correctly fitted according to the manual specifications. The PL chest strap was also fitted. The PL wrist receiver was placed within 1 ft of the participant so that HR could be manually recorded. All devices were reset with the participants' age, gender, height, and weight status prior to the start of the treadmill protocol. Participants then had seated resting HR recorded each minute for 3 min. After finishing the resting phase, participants immediately started the treadmill protocol.

During the treadmill protocol, HR was manually recorded every minute. Participants were asked to read the digital HR display of the MA, BP, MB, and TT to the researcher (PL, SR and FH had HR readouts displayed on an iPod controlled by the researcher). Participants were asked in rotating randomised order to read the display of the watches, for example, left arm with proximal to distal readings, then right arm with distal to proximal readings and vice versa. The Polar RS400 wrist receiver was attached to the handrails of the treadmill so that the researcher could read the HR display. Participants walked and ran on the treadmill at 3.2, 4.8, 6.4, 8.0, and 9.6 km·h−1 for 5 min at each protocol speed. Participants then cooled down at 4.8 km/h−1 for 5 min. On completion of the treadmill protocol, the participant again had seated resting HR recorded every minute for 3 min.

Data analysis

Means and SDs were calculated for age, height, weight, blood pressure, and BMI as well as for all the HR readings from the monitors. Pearson product-moment correlation was conducted to examine overall HR monitor associations. The mean absolute percentage error (MAPE) was calculated to provide a gauge of general measurement error of the HR monitors. Bland-Altman plots with the corresponding intercept and slope were utilised to examine the agreement of the recorded HR derived from all activity trackers. HR values were compared between the different monitors with a series of analysis of variance (ANOVA) with the Tukey honest significant difference (HSD) post hoc test. A two-way ANOVA was conducted to examine the effect of gender and method on measured HR. Finally, to examine the measurement agreements between the wearable HR monitors and the PL, 95% equivalence testing (ie, an α of 5%) for resting, walking, and running phases were performed.

Results

The participant physical characteristics are presented in table 1.

Table 1

Physical characteristics of male (n=32) and female (n=18) participants

Variables	Male		Female
	Mean±SD	Range	Mean±SD	Range
Age (years)	27.47±6.1	19.0–43.0	24.17±4.93	19.0–38.0
Height (cm)	180.22±4.7	173.0–195.0	164.38±8.7	138.0–177.8
Weight (kg)	82.66±9.5	62.0–100.7	63.18±10.9	46.0–89.3
BMI (kg, m2)	25.44±2.6	19.7–30.7	23.36±3.5	17.7–31.9
SBP (mm Hg)	126.63±10.5	102.0–143.0	108.33±10.9	94.0–130.0
DBP (mm Hg)	78.31±7.1	63.0–93.0	72.56±7.2	60.0–96.0

BMI, body mass index; DBP, diastolic blood pressure; SBP, systolic blood pressure.

A two-way ANOVA was conducted to examine the effect of gender and method on measured HR. Analysis found that there was no significant interaction between the effects of gender and method on measured HR: F (11.716)=1.511, p=0.170.

Presented in table 2 are the descriptive statistics, including the mean and SD of the HR for all six activity monitors compared with the measured value of the criterion measure. The criterion measure had a mean of 109.06±29.3 bpm, and the compiled means from the studied activity monitors ranged from a low of 105.00±30.6 bpm for the FH to a high of 111.13±30.9 bpm for the TT.

Table 2

Descriptive statistics on heart rates (HR) for all monitors

	N	Mean±SD (bpm)	Minimum	Maximum
Polar RS400	1794	109.06±29.3	55	194
Scosche Rhythm	1779	108.22±29.9	33	194
Mio Alpha	1794	110.54±30.3	45	194
TomTom	1110	111.13±30.9	55	204
Microsoft Band	1758	109.38±28.8	52	199
Basis Peak	1714	109.27±28.7	53	192
Fitbit Charge HR	1781	105.00±30.6	53	193

Table 3 shows the Pearson product-moment correlation coefficient (r) for all monitors.

Table 3

Correlation matrix for all monitors

	Polar RS400	Scosche Rhythm	Mio Alpha	TomTom	Microsoft Band	Basis Peak	Fitbit Charge HR
Polar RS400	1	0.930*	0.929*	0.959*	0.956*	0.954*	0.933*
Scosche Rhythm		1	0.865*	0.906*	0.891*	0.893*	0.870*
Mio Alpha			1	0.899*	0.904*	0.899*	0.886*
TomTom				1	0.941*	0.903*	0.904*
Microsoft Band					1	0.929*	0.905*
Basis Peak						1	0.902*
Fitbit Charge HR							1

*Correlation is significant at the 0.01 level (2-tailed).

HR, heart rate.

The TT and MB had the strongest correlation with the criterion measure (r=0.959 and r=0.956, respectively). All activity monitors had a significant correlation at the 0.01 level (2-tailed). An analysis that calculated correlation by intensity level was also conducted and it found that the correlation for all the activity trackers decreased from rest to 3.2 km/h, but all had mostly rebounded by the 6.4 km/h phase.

The mean±SD and MAPE calculated for the entire protocol is presented in table 4. Overall, the TT (3.28%), BP (3.61%), and SR (3.98%) were within 3%. During the 3.2 km/h walking phase, the MAPE rose for all activity monitors, most notably for the MA (15.97%).

Table 4

Mean±SD and MAPE for all monitors based on intensity

Intensity	Monitors	Polar RS400	TomTom	Basis Peak	Scosche Rhythm	Mio Alpha	Microsoft Band	Fitbit Charge HR
Rest	Mean±SD (bpm)	72.4±9.8	72.5±10.9	71.6±10.3	72.7±10.8	71.7±10.1	71.1±10.3	70.1±9.9
	MAPE±SD (%)		4.10±3.86	2.80±2.92	3.03±4.06	3.67±3.96	4.54±4.21	5.41±5.79
3.2 km/h	Mean±SD (bpm)	84.5±10.9	84.6±13.4	87.8±12.7	87.0±12.5	90.5±25.7	89.5±11.8	82.3±9.9
	MAPE±SD (%)		5.71±8.41	5.36±11.16	4.98±9.82	15.97±26.44	8.39±9.59	6.77±7.13
4.8 km/h	Mean±SD (bpm)	92.5±11.7	93.9±11.9	93.4±12.1	92.6±17.0	95.5±15.7	94.2±11.5	86.0±12.9
	MAPE±SD (%)		2.82±3.18	3.31±5.36	6.67±13.99	6.09±15.98	4.98±5.38	9.99±11.46
6.4 km/h	Mean±SD (bpm)	109.5±15.3	110.8±16.7	109.4±15.1	108.2±19.5	109.5±15.3	107.4±13.9	98.8±19.4
	MAPE±SD (%)		2.63±4.87	2.12±3.09	4.55±10.11	2.04±3.23	4.68±6.06	10.06±12.96
8.0 km/h	Mean±SD (bpm)	135.6±18.4	139.3±19.3	136.2±17.9	133.4±22.5	136.4±18.6	136.9±18.3	134.0±19.0
	MAPE±SD (%)		1.60±2.67	4.84±9.02	3.17±9.06	1.47±3.65	4.22±6.37	2.46±4.78
9.6 km/h	Mean±SD (bpm)	151.4±19.2	153.9±19.1	149.1±17.0	148.3±23.5	151.7±19.3	150.7±17.9	149.4±19.1
	MAPE±SD (%)		0.97±1.15	3.28±5.95	2.65±9.22	0.82±1.27	3.06±3.78	1.73±2.80
Cool down	Mean±SD (bpm)	115.8±19.2	120.7±23.5	116.0±19.2	114.5±21.5	116.6±19.4	115.4±19.5	113.2±22.7
	MAPE±SD (%)		5.80±19.81	2.55±5.20	3.28±8.18	2.48±5.62	4.29±10.17	5.83±9.28
Rest	Mean±SD (bpm)	89.1±14.3	91.3±14.3	88.5±14.1	88.8±13.7	88.9±13.9	86.8±13.3	85.5±14.5
	MAPE±SD (%)		2.54±2.41	3.15±4.85	2.22±3.55	2.72±4.48	3.81±3.78	7.73±8.79
Total	Mean±SD (bpm)	109.1±29.3	111.1±30.9	109.3±28.7	108.2±29.9	110.54±30.3	109.4±28.8	105.0±30.6
	MAPE±SD (%)		3.28±8.66	3.61±6.84	3.98±9.57	4.56±12.92	4.82±6.99	6.22±9.11

HR, heart rate; MAPE, mean absolute percentage error.

The highest percentage of error occurred in the FH in the 3 (9.99%) and 6.4 km/h (10.06%) walking phase, respectively. During the 8 km/h jogging phase, all percentage errors dropped except for the BP. The FH had the largest drop in percentage error, from 10.06% to 2.46%. In the final running stage of 9.6 km/h, the lowest percentage errors were seen. The MA (0.82%) and TT (0.97%) had <1% error.

Bland-Altman plot10 examination provided the distribution of error. Inspecting for proportional systematic bias in the recorded HR, figure 1 compared the means of the PL to each device, and found the MB (slope=0.02, difference=31.1) and FH (slope=−0.05, difference=43.4) had greater bias; while the bias for the MB is spread throughout the lower and higher mean HR of the PL, the FH had more bias at a lower mean HR.

Figure 1

Bland-Altman plots for all monitors. HR, heart rate.

Results of post hoc Tukey analysis indicated that there were no significant differences between the HR recorded for the PL and five monitors (p>0.531), except for the FH (p=0.001). However, through the utilisation of equivalence testing, which is considered innovative and still evolving, all six activity monitors tested were found to be equivalent to the capture of HR from the criterion measure in the resting, walking and running phases.

Table 5 shows the calculated 90% CIs for the recorded HR from the monitors when compared with the computed equivalence zone for the PL.

Table 5

Results from 95% equivalence testing for agreement in measured heart rate (HR) between the criterion measure and all monitors

Activity	Monitor	Mean±SE	Equivalent lower limit	Lower 90% CI	Upper 90% CI	Equivalent upper limit
Resting	Scosche Rhythm *	80.6±1.6	72.7	77.9	83.3	88.8
	Mio Alpha*	80.3±1.5	72.7	77.8	82.9	88.8
	TomTom*	81.8±2.0	72.7	78.4	85.2	88.8
	Microsoft Band*	78.9±1.4	72.7	76.4	81.4	88.8
	Basis Peak*	79.9±1.5	72.7	77.3	82.6	88.8
	Fitbit Charge HR*	77.6±1.4	72.7	75.2	80.1	88.8
Walking	Scosche Rhythm *	100.5±1.8	90.5	97.4	103.6	110.6
	Mio Alpha*	103.0±1.8	90.5	99.9	106.2	110.6
	TomTom*	102.5±2.3	90.5	98.5	106.5	110.6
	Microsoft Band*	101.6±1.5	90.5	99.0	104.3	110.6
	Basis Peak*	101.5±1.7	90.5	98.6	104.4	110.6
	Fitbit Charge HR*	95.1±1.6	90.5	92.3	97.8	110.6
Running	Scosche Rhythm*	141.3±2.9	129.1	136.4	146.3	157.7
	Mio Alpha*	144.3±2.6	129.1	140.0	148.7	157.7
	TomTom*	147.0±3.4	129.1	141.3	152.8	157.7
	Microsoft Band*	143.7±2.3	129.1	139.8	147.6	157.7
	Basis Peak*	142.3±2.4	129.1	138.3	146.3	157.7
	Fitbit Charge HR*	141.7±2.5	129.1	137.4	145.9	157.7

*Within the equivalent zone.

Discussion

The current study investigated the accuracy of six newly released wearable activity trackers that continually measure HR with PPG techniques through the arm and wrist area during rest (3 min seated before and after experiment), and specific treadmill speeds. The criterion measure was a Polar RS-series chest strap with wrist receiver, which in earlier studies was found to have good criterion-related validity with the ECG, and was well suited for measuring HR during PA and exercise training.8 11

Only a few studies have evaluated the accuracy of HR monitors. In a 2002 study using traditional chest strap HR monitors, investigators found that correlation with the ECG decreased with a higher speed of 9.6 km/h and the investigators attributed this to increased upper body movement.2 A similar study conducted in 2011 involving the Smarthealth watch, an activity monitor that relies on two points of contact to measure the heart's electrical impulse, had comparable results. The researchers validated the HR for the Smarthealth watch at rest and during treadmill activities, but reported that at higher speeds of 7.2 and 9.6 km/h the watch had reduced ability to detect HR (a decrease of 6% and 13.9%, respectively).12 Again, the investigators attributed this reduced ability to increased upper body movement.

Conversely, in the present study, the accuracy of the optical sensing HR activity monitors had the least MAPE during the highest speed tested, 9.6 km/h. During this phase, the greatest MAPE observed was with the BP (3.28%) and MB (3.06%). These results mirror those found in a recent, small study that evaluated the performance of the MA and SR using an ECG as the criterion measure.9 The investigators reported the MAPE of the MA for walking and running was 5.60% and 2.37%, respectively. In the present study, the MAPE of the MA was 8.02% and 1.15%, respectively. In the past study, the MAPE of the SR for walking and running was 10.49% and 3.81%, respectively. For the present study, the MAPE of SR was 5.40% and 2.91%, respectively. Both studies showed a reduction in MAPE with increased speed. One possible explanation is that with increased intensity there is improved perfusion, which could decrease the error rate.

Overall, strong correlations were observed between the activity monitors and the criterion measure, ranging from r 0.87 to 0.96, and the measured HR from all six monitors were significantly equivalent to the measured HR from the criterion measure in resting, walking and running conditions. This suggests that all the activity monitors would provide comparable accuracy to the more established HR monitor. This is an important finding since it informs the existing literature on HR monitoring devices and also supports the utility of these new devices for everyday personal use as well as for research application.

While conducting the experiment, challenges with correct fit and placement were observed. While great care was taken to ensure watches were placed properly, the experiment was conducted in semifree-living conditions which resulted in realistic issues arising. A few of the participants had either larger or smaller wrists and forearms that made proper fitting of the activity monitors a challenge. However, all watches were fitted according to the manual specifications with maximum effort focused on placement control of the watches. In this study, when some participants tried to hold the treadmill railing, HR readings sometimes became irregular, and in two incidents, the BP and FH did not provide a HR reading. The MA also was observed to fluctuate between a high and low HR during this time. Once the participant began walking naturally, with arms swinging, HR readings more closely reflected the criterion measure. Similarly, as soon as the participant started jogging, the arms bent at the elbows and became perpendicular to the body. During the 6 mph jogging phase, the MA and TT had <1% MAPE while the FH was observed with its lowest MAPE for all the protocol intensities. It is speculated that the lesser MAPE is likely attributable to the arms being in a bent, stabilised position combined with the increased HR from exertion. Perhaps a higher and stronger HR can be ‘read’ more easily by the LED lights.

The strengths of this study included a reasonable sample size, examination of a variety of wearable activity HR trackers that are currently available in the market, and utilisation of a mixture of various walking and running intensities. In addition, proper fit and constant supervision provided the best opportunity for activity tracking as each tracker was functioning within its intended capacity. The result of this study add to the existing literature on HR monitoring and is one of the first to undertake validation of new PPG optical sensing HR activity trackers. However, it does have some limitations. The sample population included only healthy, younger individuals (19–45 years) who engaged in regular aerobic exercise and were within the normal range of body weight and body fat. Generalisations cannot be made for youth and/or older adult age groups or for individuals of other body sizes. This study included only walking and running activities; it could be possible that during intermittent or high-intensity interval training results could have been different. The study was also conducted using a controlled treadmill protocol and transfer of results to free-living conditions should be made with caution.

In conclusion, the present study results showed favourable outcomes for the six PPG optically sensing HR wearable activity trackers that were tested at rest, and during treadmill walking and running in a healthy sample population. Good criterion-related validity was found between all monitors and the Polar HR monitor. In addition, the wearable activity trackers were deemed accurate for the recreational athlete and for research purposes. Furthermore, wearable activity trackers utilising built-in PPG HR sensors have the potential to overcome the limitations of the traditional chest strap, and to advance the science and practice of PA assessment. Further tests utilising a fixed floor, such as a track, and various indoor/outdoor environments and high-intensity exercises (including weight lifting and bicycling) could confirm the usability of these wearable trackers in expanded exercise settings. Future studies should include different populations and health concerns, such as young and older adults and individuals afflicted with obesity (ie, epidermal thickness) and diabetes (ie, poor blood circulation).