The validity and reliability of an open source biosensing board to quantify heart rate variability.

BACKGROUND: Heart rate variability (HRV) is a popular tool to quantify autonomic function. However, this typically requires an expensive 3-12 lead electrocardiogram (ECG) and BioAmp system. This investigation sought to determine the validity and reliability of an OpenBCI cyton biosensing board (open source) for accurately quantifying HRV. NEW
METHOD: A cyton board with a 3-lead ECG was employed to acquire heart rate waveform data, which was processed to obtain HRV within both time- and frequency-domains. The concurrent validity was compared to a simultaneous recording from an industry-standard 3-lead ECG (ADInstruments) (n = 15). The reliability of the cyton board was compared between three days within a 7-day timespan (n = 10). Upright quiet-stance short-term HRV metrics were quantified in time- and frequency-domains.
RESULTS: The two devices displayed excellent limits of agreements (all log mean differences ±0.4) and very high between-device variable associations (all r 2 > 0.98). Between the three time points in the same subjects, no differences were noted within time- (all p > 0.71) or frequency-domains (all p > 0.88) across testing points. Finally, all HRV metrics exhibited excellent levels of reliability through high Cronbach's Alpha (all ≥0.916) and intraclass correlation coefficients (all ≥0.930); and small standard error of the measurement (all ≤0.7) and typical error of the measurement (all ≤0.1) metrics. COMPARISON WITH EXISTING
METHODS: The cyton board with 3-lead ECG was compared with an industry-standard ADInstruments ECG during HRV assessments. There were no significant differences between devices with respect to time- and frequency-domains. The cyton board displayed high-levels of between-day reliability and provided values harmonious to previous ECG literature highlighting the applicability for longitudinal studies.
CONCLUSION: With proper background knowledge regarding ECG principles and a small degree of set-up complexity, an open source cyton board can be created and employed to perform multimodal HRV assessments at a fraction of the cost (~4%) of an industry-standard ECG setup.

Introduction

Heart rate variability (HRV) refers to time fluctuations between successive ventricle depolarizations (R-R intervals) [1]. This has been utilized as an indirect measurement method for quantifying the complex interplay between the parasympathetic and sympathetic nervous systems as they relate to autonomic function [2]. The information provided from HRV analysis can be used to describe autonomic function within both time- and frequency-domains via assessing the non-linearity (or unpredictability) of adjacent heart beats [3]. This provides an indication of the relative sympathetic and parasympathetic contributions of one's cardiovascular system [3]. Further, increased or altered HRV has been identified as a risk factor for all-cause mortality, highlighting the importance of this tool in both research and clinical settings [4, 5].

In recent years, HRV metrics have been employed as an assessment tool for a plethora of clinical conditions including: concussion [6], diabetes [7], mental disorders [8], stroke [9], and myocardial infarction [10, 11]. The clinical diversity in this breadth of research exemplifies the utility of HRV and its clinical importance; however until recently [12, 13], it has generally been quantified with a 3–12 lead electrocardiography (ECG) system. These ECG devices are typically single-purpose, where only waveform data related to the depolarization and repolarization across the cardiac tissue is collected (i.e., PRQST waveform). While individual physiological assessments can reveal important information, employing a multimodal approach increases the breadth and understanding related to the physiological underpinnings of diseases and disorders [14, 15]. For example, being able to implement a study simultaneously examining cerebrovascular and cardiovascular function could prove paramount in enhancing the mechanistic understanding of clinical brain disorders such as stroke, traumatic brain injuries, Alzheimer's Disease, Parkinson's Disease, and so on. However, a common limitation to setting up multimodal imaging is the cost associated and data syncing requirements with the numerous pieces of equipment required to obtain these measures during a single recording session. Therefore, if certain devices are capable of collecting several domains of data, this would enhance the ability for these measures to be obtained, while also lowering the cost and maximizing equipment utility. Nevertheless, before such devices can be widely adopted, they must first be proven to accurately quantify what they intend to measure (i.e., validity) and are able to consistently measure a variable (i.e., reliability).

In previous years, several studies have been conducted examining the validity and reliability with respect to various commercially available heart rate monitors [16], smartphones [17, 18], and smartwatches [19, 20, 21], to obtain HRV metrics. Within these investigations, the accuracy and consistency of HRV parameters were commonly calculated using intraclass correlation coefficients (ICC), Bland-Altman plots with 95% limits of agreement (LOA), or other reliability approaches such as standard error of the measurement (SEM) metrics. In brief, these studies commonly deemed the aforementioned technological devices as valid and reliable with: 1) ICC point estimate greater ≥0.81; 2) ICC 95% lower limit ≥0.75; 3) mean bias within the Bland-Altman plots ranging from log transformed values of 0.00–1.70; and 4) SEM values ranging from 1.24 – 1.72 [16, 17, 18, 19, 20, 21].

Therefore the purpose of the current investigation was to determine the validity and reliability of an open source Cyton Biosensing Board with three-lead ECG compared to an industry-standard three-lead ECG to collect waveform data related to the electrical activity of cardiac contractions. The validity/reliability outcome measures in this study will be compared to the values within previous literature that concluded their commercially available technological devices (i.e., smartwatches, heart rate monitors, etc.) were acceptable to derive HRV metrics [16, 17, 18, 19, 20, 21]. As the cyton board is commonly employed to measure cerebral electrical activity through electroencephalography (EEG), this would enhance the ability for future research groups to simultaneously obtain both neural and autonomic activity [22]. Moreover, it would enable forthcoming studies to investigate associations between the cardiovascular system and neural activity within healthy and clinical populations, while minimizing costly equipment demands.

Materials and methods

Study design

The current investigation consists of two different data collections. The first examined the concurrent validity between the Cyton Biosensing Board with three-lead ECG (OpenBCI, Brooklyn, New York, USA) (herein referred to as cyton board) and an industry-standard ECG system (ADInstruments, Bio Amp FE231, Colorado Spring, Colorado, USA). The second quantified the between-day reliability of the cyton board. Prior to the commencement of both studies, written informed consent was collected from all participants. Ethical approval for the investigation was given by the University of Calgary Conjoint Health Research Ethics Board (REB15-1376).

Study I: concurrent validity

To assess the concurrent validity of the open source cyton board compared to an industry-standard three-lead ADI ECG system, a convenience sample of 15 participants (six females) were recruited from the university setting. The participants had a median age of 29 years (range: 21–56 years) and body mass index of 26.0 kg/m2 (range: 23.0–30.9 kg/m2). All protocols were thoroughly explained to ensure the participants were familiar with the testing procedure. As HRV is known to be influenced by exercise [2, 23], caffeine [24], food consumption [25, 26], alcohol [2]; these covariates were collected to minimize the likelihood they influence the acquired data. In addition, personal health history, including medication use, previous diagnosis of a mental or physiological illness, learning disability, neuropsychological condition, and/or a history of migraines or headaches were collected. This data was collected a priori to understand if the validity of the device was impacted by any of the aforementioned covariates.

Study II: between-day reliability

The reliability of the open source cyton board was a subsection of a larger physiological investigation examining the reliability of: HRV, EEG, and functional near-infrared spectroscopy devices. Using a convenience sample, ten participants (six females) from the University of Calgary came in for testing. The participants had a median age of 24 years (range: 20–27 years) and body mass index of 23.0 kg/m2 (range: 18.6–27.2 kg/m2). To minimize the influence of extraneous confounders, participants were asked to abstain from alcohol and smoking for twelve hours, exercise for six hours, and food consumption for two hours, prior to the study [23]. On day one, participants completed the Holmes-Rahe Life Stress Inventory [27], Perceived Stress Scale [28], and the Patient Stress Questionnaire [29] to index global stress levels of participants. On each testing day, participants filled out the Health Behavior Inventory [30] and the Daily Stress Inventory [31] to assess an individual's level of stress prior-to each testing session. As stress is known to alter HRV metrics [32], the three days with the most consistently reported data regarding the aforementioned covariates were selected to determine the between-day reliability of the cyton board across the seven days of testing. Additionally, while HRV metrics have been shown to remain stable across the day when HRV measures are calculated in an upright orthostatic position [23], any potential influence of diurnal variation in the data collections was controlled for by having each participant complete their testing at the same time as their initial assessment.

Instrumentation

For study I, heart rate data was collected from the open source cyton board plus ECG and the ADI ECG. In concordance with ECG recordings, the latter device was set-up to create three axes triangulating the heart [33]. The electrodes from both devices were placed using the standard three-lead placement: 1) inferior to the right clavicle, 2) interior to the left clavicle, and 3) superior and laterally to the right of the umbilicus [33].

The same electrode placements occurred for study II. In contrast to study I, this study only employed the cyton board. The goal of this aspect of the overall investigation was to determine the reliability of the cyton board when measuring HRV. The industry-standard ADI ECG data was sampled with commercially available software (LabChart version 8.1, ADInstruments) at a frequency of 1000Hz (PowerLab 8/30 ML880, ADInstruments). The data collected using the cyton board was sampled at 250Hz for study I and 1000Hz for study II, using an open source Graphical User Interface platform (OpenBCI, Brooklyn, New York, USA). Figure 1 displays a representative raw trace from one participant using both devices. Additionally, a step-by-step guide is provided as supplementary material to instruct individuals on how to obtain a quality ECG waveform trace that can be computed to quantify HRV parameters.

Figure 1

A representative raw trace in one subject using the industry-standard ADInstruments device (ADI) and the open source cyton board (OpenBCI) used to determine the concurrent validity of the OpenBCI cyton biosensing board.

Experimental Protocols

To determine the concurrent validity of the short-term HRV measures (study I), participants quietly stood in an upright orthostatic position for five minutes. [3]. As the second phase (study II) of the present investigation was a subsection of a larger study, participants similarly stood in an upright orthostatic position for eight minutes, in order to obtain an adequate hemodynamic response for functional near-infrared spectroscopy metrics [34]. Prior to the standing protocol, participants completed other tasks in a seated position. Therefore, a minimum washout period of one minute was given to ensure cardiovascular metrics had normalized due to the postural shift (i.e., seated to standing) [35]. Further, in conjunction with previous research, this position was chosen as it has shown to elicit greater [36, 37] or equivalent [38, 39] reproducibility compared to the supine/seated position. Moreover, quantifying HRV within the upright position was chosen to reduce the risk of parasympathetic saturation which could occur within supine or seated positions [40].

Data processing

While eight minutes of standing data were collected in study II, to be congruent with previously published short-term HRV studies, only the first five minutes of the eight minute standing protocol were analyzed [41]. All heart rate and HRV data were collected in conformance with the guidelines put forth for these measures [42]. Data were processed using commercially available software (Version 1.0, R&D Canvas, Wellington, NZ), which uses similar algorithms (i.e., Butterworth filter, infinite impulse response, Hanning filters, etc.) to other widely utilized software for HRV analysis (e.g., LabView, Kubios, etc.) [43] (https://elucimed.com/ensemble-r/extract/extract-overview/). Data were visually inspected for artifacts (i.e., ectopic beats, misaligned beat detections, etc.) and a threshold based beat correction algorithm with a low threshold was used to identify artefacts. The number of artefacts within the data was ≤0.5% across all recordings, which was comparable between devices. Artefacts were replaced using a cubic spline interpolation, based upon the average R-R interval data from each individual recording.

Statistical analysis

The independent variables in this study were the devices used to obtain heart rate data (cyton board [OpenBCI] with three lead ECG); whereas the output variables are the time- and frequency-domains metrics of HRV. As HRV measures are not normally distributed, all data were log transformed to create a normal distribution, in accordance with previous recommendations [44]. The time-domain variables included: heart rate, standard deviation between R-R intervals (SDNN), root mean square of consecutive R-R intervals (RMSSD), and percentage of successive R-R intervals differing by more than 50 ms (pNN50). The frequency-domain variables included: the relative low frequency (LF) and high frequency (HF) power, and LF/HF. Statistical analyses were conducted using RStudio (v.1.4.1060) [45].

Bland-Altman plots with 95% LOA were used to determine the agreement between devices for HRV outcome variables in both time- and frequency-domains. Additionally, simple linear regressions were run for each variable to establish the adjusted coefficient of determination (adjusted r2) between devices. To assess the agreement between the cyton board and the industry-standard three-lead ADI ECG system, ICC and their corresponding 95% confidence intervals (CI) were calculated. To ensure sufficient agreement to warrant interchangeability between the two devices, both the point-estimate ICC and the ICC 95% lower limit were calculated [46, 47]. For the point-estimate ICC a value of >0.81 was deemed excellent [46], whereas a value of >0.75 and >0.90 was required to produce good and excellent reliability for the ICC 95% lower limit, respectively [47]. Data are presented as log transformed mean ± standard deviation. Significance was set a priori at alpha = 0.05.

A one-way repeated measures analysis of variance for all outcome variables was used to determine potential differences between testing days. Tukey's honestly significant difference corrected post-hoc comparisons were performed to determine where any potential differences between days may have occurred. To measure the internal consistency, variability, and reliability of HRV measures when quantified with the cyton board between days, Cronbach's alpha (α), between-day within-subject coefficient of variation, ICC, SEM, and typical error of the measurement (TEM) values were calculated. A threshold of <70%, 70–80%, 80–90%, and >90% were utilized to classify unacceptable, acceptable, good, and excellent Cronbach's Alpha metrics, respectively [48, 49]. Moreover, consistent with previous physiological research [50, 51, 52, 53, 54, 55], coefficient of variation values were deemed acceptable/reasonable (<20%) or good (<10%) and were calculated using published reliability guidelines [56]. The ICC thresholds for the between-day reliability were consistent with those used for the concurrent validity aspect of this investigation [46, 47]. The SEM [56, 57] and TEM [46] metrics were calculated as previously outlined. Data are presented as log transformed mean ± standard deviation or 95% CI, where appropriate. Significance was set a priori at alpha = 0.05.

Results

Study I: concurrent validity

The log transformed values for all HRV values between devices are displayed in Table 1. The mean difference (range: −0.04 – 0.002) and LOA (range: −0.32 – 0.24) within the time-domain metrics between devices were minuscule (all percent differences <0.92%) (Table 1, Figure 2). Additionally, all time-domain parameters (i.e., heart rate, SDNN, RMSSD, pNN50) displayed a near-linear relationship between devices (all r2 ≥0.988) (Table 1). All time-domain ICC point-estimates (all ≥0.997) and ICC 95% lower limits (all ≥0.992) were deemed excellent (Table 1).

Table 1

Concurrent Validity of Log Transformed Heart Rate Variability Data in 15 participants assessed using Bland-Altman plot with 95% limits of agreement, adjusted coefficient of determination (adjusted r2) through simple linear regressions, and intraclass correlation coefficients (ICC).

	ADI	OpenBCI	Mean Difference (95% CI)	Limits of Agreement	Adjusted r2	ICC (95% CI)
Time-Domain
Heart Rate (bpm)	4.39 ± 0.19	4.39 ± 0.19	0.002 (−0.002, 0.006)	−0.01–0.02	0.999	0.999 (0.999, 1.000)
SDNN (ms)	3.72 ± 0.49	3.72 ± 0.50	−0.001 (−0.021, 0.020)	−0.07–0.07	0.996	0.999 (0.998, 1.000)
RMSSD (ms)	3.06 ± 0.67	3.10 ± 0.63	−0.039 (−0.118, 0.039)	−0.32–0.24	0.988	0.997 (0.992, 0.999)
pNN50 (%)	2.09 ± 0.89	2.11 ± 0.89	−0.025 (−0.082, 0.052)	−0.25–0.22	0.995	0.998 (0.995, 0.999)
Frequency-Domain
Low Frequency (n.u.)	4.10 ± 0.10	4.11 ± 0.10	−0.005 (−0.012, 0.003)	−0.03–0.02	0.992	0.994 (0.983, 0.998)
High Frequency (n.u.)	3.66 ± 0.16	3.65 ± 0.16	0.009 (−0.004, 0.021)	−0.04–0.05	0.991	0.995 (0.984, 0.998)
LF/HF Ratio (%)	0.45 ± 0.25	0.46 ± 0.25	−0.013 (−0.033, 0.066)	−0.08–0.06	0.985	0.994 (0.982, 0.998)

Values are mean ± standard deviation. Confidence Interval (CI), beats per minute (bpm), standard deviation between R-R intervals (SDNN), milliseconds (ms), root mean square of consecutive R-R intervals (RMSSD), percentage of successive R-R differing by more than 50 ms (pNN50), percent (%), normalized units (n.u.), and low frequency/high frequency (LF/HF).

Figure 2

Bland-Altman plots with 95% limits of agreement demonstrating the validity of quantifying log transformed (Ln) time-domain measures of heart rate variability using an open source cyton board compared to an industry-standard electrocardiogram (n = 15). Time-domain variables include: A) heart rate, B) standard deviation between R-R intervals (SDNN), C) root mean square of consecutive R-R intervals (RMSSD), and D) percentage of successive R-R differing by more than 50 ms (pNN50).

Likewise, there was high agreement within frequency-domain metrics between devices with a mean bias and 95% LOA ranging from −0.013 – 0.009 and −0.08 – 0.06, respectively (Table 1, Figure 3). Finally, the frequency-domain values (relative LF power, relative HF power, LF/HF) also displayed a near-linear relationship when comparing the cyton board to the “reference standard” (all r2 ≥0.985) (Table 1). Finally, excellent ICC point-estimates (all ≥0.994) and ICC 95% lower limits (all ≥0.982) were derived from the frequency-domain metrics (Table 1).

Figure 3

Bland-Altman plots with 95% limits of agreement demonstrating the validity of quantifying log transformed (Ln) frequency-domain measures of heart rate variability using an open source cyton board compared to an industry-standard electrocardiogram (n = 15). Frequency-domain variables include: A) relative low frequency (LF) power, B) relative high frequency (HF) power, and C) LF/HF ratio.

Study II: between-day reliability

No differences were noted within heart rate (F(2,9) = 0.04, p = 0.958), SDNN (F(2,9) = 0.38, p = 0.686), RMSSD (F(2,9) = 0.15, p = 0.872), and pNN50 metrics (F(2,9) = 0.11, p = 0.901) across the three testing days (Table 2, Figure 4). The reliability of heart rate (α = 0.969), SDNN (α = 0.916), RMSSD (α = 0.925), and pNN50 metrics (α = 0.965) were excellent across the three days (Table 2). The coefficient of variation values between testing days for time-domain measures were: heart rate (3.4%), SDNN (11.3%), RMSSD (14.6%), and pNN50 (23.3%) (Table 2). Both the ICC point-estimates (all ≥0.911) and ICC 95% lower limits (all ≥0.833) for all time-domain variables were deemed excellent (Table 2). Moreover, all log transformed time-domain SEM and TEM values were ≤0.07 and ≤0.67, respectively (Table 2).

Table 2

Between-Day Reliability of Log Transformed Heart Rate Variability Data in 10 participants using one-way Analysis of Variance, Cronbach's Alpha, Coefficient of Variation (CoV), Intraclass Correlation Coefficients (ICC), Standard Error of the Measurement (SEM), and Typical Error of the Measurement (TEM).

	Day 1	Day 2	Day 3	Test Statistic	Cronbach Alpha	CoV (%)	ICC (95% CI)	SEM (95% CI)	TEM
Time-Domain
Heart Rate (bpm)	4.45 ± 0.13	4.45 ± 0.13	4.46 ± 0.16	F(2,9) = 0.04, p = 0.96	0.969	3.4	0.970 (0.915, 0.992)	0.04 (0.02, 0.06)	0.18
SDNN (ms)	3.76 ± 0.16	3.84 ± 0.25	3.75 ± 0.28	F(2,9) = 0.38, p = 0.69	0.916	11.3	0.911 (0.833, 0.986)	0.07 (0.04, 0.12)	0.28
RMSSD (ms)	3.36 ± 0.26	3.33 ± 0.36	3.28 ± 0.35	F(2,9) = 0.15, p = 0.87	0.925	14.6	0.930 (0.846, 0.989)	0.07 (0.04, 0.12)	0.36
pNN50 (%)	1.78 ± 0.86	1.59 ± 1.21	1.60 ± 0.98	F(2,9) = 0.11, p = 0.90	0.965	23.3	0.967 (0.905, 0.991)	0.07 (0.03, 0.13)	0.67
Frequency-Domain
Low Frequency (n.u.)	3.96 ± 0.21	3.99 ± 0.19	4.01 ± 0.16	F(2,9) = 0.16, p = 0.86	0.981	4.4	0.979 (0.938, 0.994)	0.04 (0.02, 0.06)	0.21
High Frequency (n.u.)	3.81 ± 0.24	3.79 ± 0.22	3.77 ± 0.20	F(2,9) = 0.09, p = 0.92	0.981	4.9	0.979 (0.938, 0.994)	0.03 (0.02, 0.05)	0.21
LF/HF Ratio (%)	0.15 ± 0.44	0.20 ± 0.41	0.23 ± 0.36	F(2,9) = 0.12, p = 0.89	0.980	9.4	0.981 (0.945, 0.995)	0.03 (0.02, 0.06)	0.28

Values are mean ± standard deviation. Percent (%), beats per minute (bpm), standard deviation between R-R intervals (SDNN), milliseconds (ms), root mean square of consecutive R-R intervals (RMSSD), percentage of successive R-R differing by more than 50 ms (pNN50), normalized units (n.u.), and low frequency/high frequency (LF/HF).

Figure 4

Boxplots of log transformed (Ln) heart rate variability time-domain measures across three time points using an open source cyton board in 10 individuals. Using a one-way repeated measures analysis of variance, no differences were noted between days (all F(2,9)<0.38, all p > 0.686). The coloured dots denote the values of each subject across the three days, displaying the intra-individual variability. Time-domain variables include: A) heart rate, B) standard deviation between R-R intervals (SDNN), C) root mean square of consecutive R-R intervals (RMSSD), and D) percentage of successive R-R differing by more than 50 ms (pNN50).

Similarly, across the three testing days, no differences were noted with LF (F(2,9) = 0.16, p = 0.858), HF (F(2,9) = 0.09, p = 0.924), and HF/LF metrics (F(2,9) = 0.12, p = 0.891) (Table 2, Figure 5). Furthermore, excellent levels of reliability were observed across the three testing days for LF (α = 0.981), HF (α = 0.981), and LF/HF metrics (α = 0.980) (Table 2). The coefficient of variation values between testing days for frequency-domain measures were: LF (4.4%), HF (4.9%), and LF/HF (2.8%) (Table 2). Excellent ICC point estimates (all ≥0.979) and ICC 95% lower limit (all ≥0.938) were produced for all log transformed frequency-domain variables (Table 2). Lastly, all time-domain SEM and TEM values were ≤0.04 and ≤0.28, respectively (Table 2).

Figure 5

Boxplots of log transformed (Ln) heart rate variability frequency-domain measures across three time points using an open source cyton board in 10 individuals. No differences were noted between the three time points when using a one-way repeated measures analysis of variance (all F(2,9)<0.16, all p > 0.858). The coloured dots denote the values of each subject across the three days, displaying the intra-individual variability. Frequency-domain variables include: A) relative low frequency (LF) power, B) relative high frequency (HF) power, and C) LF/HF ratio.

Discussion

The two main findings within the present investigation were: 1) compared to an industry-standard three-lead ECG (i.e., ADInstruments), the open source cyton board plus 3-lead ECG displayed consistently high levels of validity when measuring HRV within time- and frequency-domains, and 2) the cyton board exhibited a high degree of reliability/internal consistency in the assessment of HRV between three testing days. Moreover, given the high levels of between-day reproducibility and internal consistency, this demonstrates the cyton board can be a valuable low-cost tool within future longitudinal studies to aid in extending the scope of data collections. However, an important caveat to note is that users will require at least a basic understanding of proper ECG assessment administration to set up the cyton board for this purpose in order to obtain a valid three-lead ECG waveform (Figure 1 and Supplementary Material).

Comparisons with previous literature

In recent years, large advancements have yielded new approaches to index heart rate and HRV [12, 13], as these metrics have shown to be a useful tool in the assessment of various clinical presentations [6, 7, 8, 9, 10]. For example, several investigations have examined the validity and/or reliability of various commercially available technological devices (i.e., heart rate monitors, smartphones, smartwatches, etc.) [16, 17, 18, 19, 20, 21]. A study by Nunan and colleagues [21] examined the agreement between a Polar S810 device and a 12-lead ECG, utilizing a similar validity threshold of the ICC 95% lower limit having to be ≥ 0.75. They found the mean R-R intervals, SDNN, and RMSSD were the only HRV variables that met this cut-off over three trials; whereas, the LF/HF metric was acceptable for two of the three trials [21]. Conversely, within the current investigation, the ICC 95% lower limit for all HRV metrics was ≥0.982 and ≥0.833 for the validity and reliability aims, respectively (Tables 1 and 2). Moreover, these authors also found that the Polar S810 yielded high agreement for log transformed SDNN (bias: 0.00; 95% LOA: −0.38, 0.36) and RMSSD (bias: −0.05; 95% LOA: −0.61, 0.51) metrics measured with Bland-Altman plots [21]. A second study by Porto et al. [16], similarly examined the agreement between the Polar S810 and 12-lead ECG, concluding the non-log transformed SDNN (bias: −0.2; 95% LOA: −1.5, 2.0) and RMSSD (bias: 2.3; 95% LOA: −1.0, 5.5) produced valid estimates. While the biases were similar within the present investigation to Nunan and colleagues [21], the 95% limits of agreement were much smaller for both SDNN (bias: 0.00; 95% LOA: −0.02, 0.02) and RMSSD (bias: −0.04; 95% LOA: −0.02, 0.02) measures (Figure 2). Moreover, in opposition to these studies [16, 21], the other HRV estimates in this investigation (i.e., heart rate, pNN50, relative LF, relative HF, and LF/HF) were equally valid with a near-perfect agreement (mean bias range: −0.03 – 0.01). Lastly, a study examined the between-day reliability of the Polar S810 between three time points, where the log transformed SEM metrics (SDNN, RMSSD, and LF/HF) ranged from 1.24 – 1.72 [20]. However, in this investigation the log transformed SEM were ≤0.7 and ≤0.4 for all time- and frequency-domain HRV metrics, respectively (Table 2). Therefore, when using the previously published validity/reliability studies within the literature as a comparison, the cyton board produced outstanding validity (Table 1, Figures 2 and 3) and reliability (Table 2, Figures 4 and 5) when quantifying HRV metrics.

Moreover, Nunan and colleagues [41] provided a meta-analysis of all short-term HRV normative values across 44 studies totaling 21,438 participants. In brief, the range of values for the time-domain metrics include SDNN (32–93 ms), RMSSD (19–75 ms), whereas, the frequency-domain metrics include relative LF (30–65 n.u.), relative HF (16–60 n.u.), and LF/HF (1.1–11.6 %) [41]. In comparison, all values measured using the OpenBCI cyton board fell within this range (Tables 1 and 2), which further exemplifies the utility of this tool when quantifying HRV in healthy adults. More so, a study by Dantas and colleagues [36] examined the within-day reproducibility of HRV measures using ECG in the orthostatic position, finding r-squared values of 0.75 (SDNN), 0.91 (RMSSD), 0.86 (pNN50), 0.89 (relative LF), 0.79 (relative HF), and 0.77 (LF/HF). Congruent with this previous research study, the time- (α: ≥0.916, ICC: ≥0.911, SEM: ≤0.07, and TEM: ≤0.67) and frequency-domain (α: 0.980, ICC: ≥0.979, SEM: ≤0.04, and TEM: ≤0.27) measures within study II of this investigation were highly reliable between three days within the same week (Table 2). Further, the minimal variation noted between days (Table 2) is comparable to a study by Burma et al., [23] which found the between-week coefficient of variation for all HRV metrics to be ≤ 13%. However, the one exception within the current investigation was pNN50 metrics, which displayed a larger between-day coefficient of variation (23.3%) (Table 2). This likely is attributable to the notion sympathetic activity increases when HRV is quantified within the orthostatic posture compared to seated or supine positions [35, 36]. This will elevate heart rate and lower HRV metrics, especially relevant for the pNN50 metric, as there is a reduced duration of time between subsequent heart beats. Therefore, the higher CoV is likely in part due to the fact the absolute pNN50 values were small (<5%) and thus minor differences between days substantially increased the associated relative proportion of variability noted in adjacent heart beats. This is consistent with previous research highlighting the limited utility of pNN50 values derived from a standing position [58]. Nevertheless, the HRV parameters from both devices in the present investigation are similar to standing values previously published by Porto and colleagues [16].

Conclusively, in comparison to published studies within the literature [16, 17, 18, 19, 20, 21], the results in this investigation demonstrate the cyton board displays exceptionally high validity (Table 1, Figures 2 and 3) and reliability (Table 2, Figures 4 and 5) when quantifying HRV metrics. The reason for the near-perfect validity and reliability of the cyton board likely stems from two explanations. First, in comparison to smartwatches and other heart rate monitors that obtain heart rate information from pulsatile waveforms, the cyton board is capable of detecting beat-to-beat PQRST cardiac data (Figure 1). As HRV parameters are sensitive to millisecond alterations between R-R intervals, the cyton board enabled a precise quantification from each R-spike. Conversely, compared to a cardiac waveform, there is some degree of variability with consecutive peaks derived from pulsatile waveforms, as other factors additionally modulate the pulsatile intervals (i.e., respiration, autonomic function, etc.) [59]. Additionally, there is growing evidence that pulse rate variability is not directly relatable to HRV and in fact, may be its own biomarker [59]. Therefore, compared to the past literature, the ability to detect the R-spike from the PQRST waveform explains the greater validity and reliability with the cyton board in the present investigation. Second, there are numerous extraneous covariates that are able to impact HRV recordings (i.e., exercise, caffeine, food consumption, alcohol, etc.) [2, 23, 24, 25, 26, 32]. The current investigation went beyond previously published studies to not only restrict these confounding influences but also measured each participant's daily stress level prior to each recording using validated questionnaires (i.e., Health Behavior Inventory [30] and the Daily Stress Inventory [31]). From this information, the three days with the most similar confounding influences for each individual were included in the final analysis. This ensured optimal conditions were present for classifying the reliability of the cyton board device with minimal between-day measurement confounders. For example, if stress data were not collected in study II, the reliability of the cyton board could have substantially impacted the outcome metrics, if this was compared between days of low, moderate, and high stress. The high stress day would cause a greater sympathetic response, which would artificially lower the reliability of the cyton board, even though the device was highly reliable. For study I, both ECG devices were concurrently measured and thus any slight difference in stress levels, recent food/caffeine consumption, etc. would similarly impact both recordings. However, this information was collected a priori for post-analysis processing to see if any covariates could explain any differences in validity between devices. Nonetheless, the cyton board displayed extremely high validity compared to the industry-standard ECG, demonstrating the impact of these covariates was inconsequential within study I. Therefore, the high validity and reliability in the present investigation can be attributed to both the high utility of the cyton board and the tightly controlled study design employed.

Implications for future research

The primary purpose of the cyton board is to quantify cerebral electrical activity. However, the current results demonstrate this device is able to be multimodal and thus provides the ability to simultaneously collect autonomic activity during EEG assessments [22]. This functionality will support studies aimed at increasing our understanding of the association between neuronal activity and autonomic function, which could prove useful within clinical settings to elucidate the physiological underpinnings that occur in various clinical diseases/disorders.

There is a cost-benefit as well. Currently, HRV is routinely measured using an industry-standard ADI ECG system, which generally requires single-purpose equipment pieces. It is currently estimated the cost of the standard ADI industry-standard ECG device and data acquisition board is approximately $15,500 (United States Dollars) for the equipment (ECG and data sampling device). Conversely, the OpenBCI device was purchased for approximately $600 (United States Dollars), which corresponds to roughly 4% of the cost. Furthermore, as the latter is an open source device, it enables one to access the base code and make changes as required, increasing the utility and individual functionality of the device.

Given its high degree of reliability and internal consistency, the cyton board can help within cohort groups to aid with current prognosis and recovery assessments. For example, concussion is known to disrupt HRV metrics [6]. Therefore, the current findings would be useful in this field as the dual purpose cyton board could enable researchers/clinicians a low cost means to track autonomic function with concurrent EEG assessments over time, greatly aiding in making return to play decisions and discerning when physiological recovery has occurred [60].

Limitations

The main limitation of the current investigation is attributable to the small sample size of participants used to assess the validity and reliability of the cyton board plus ECG. Nonetheless, the values obtained were comparable to normative values within previously published literature [41]. Therefore, this likely had minimal influence on either phase of the present study. It is also well known both the HF and LF bands are influenced by many other factors besides autonomic flow (i.e., respiration, cortisol, endocrine hormones, etc.) [3, 61]. Although participants in the current investigation were allowed to breathe normally, respiratory parameters (i.e., respiration rate, tidal volume, etc.) were not recorded in the current investigation. Therefore, it cannot be ruled out that slight variations to respiration across the five minutes of data collection may have had an effect on HF band measures [3]. Nonetheless, this issue likely had an inconsequential role on the results as highlighted by the high levels of accuracy (Tables 1 and 2, Figures 2 and 3) and the fact that the data are consistent with the published literature [41]. Furthermore, any changes in breathing would not affect the comparison between devices as the industry-standard ADI ECG and cyton board data collections were performed concurrently (Figure 1). Additionally, while validity and reliability of the cyton board were classified within the upright orthostatic position, these results likely would hold true within both a supine and seated position, as the cyton board is able to record a precise PQRST waveform. Finally, the sample of participants in this study primarily included young, healthy individuals. Hence, future research is warranted to examine if the validity and reliability hold true within older cohorts. However, as three-lead ECG systems have been commonly utilized within elderly and clinical populations [7, 8, 9, 10, 32, 41], there should be no issues when using this technology in other populations.

Conclusion

Readily available open source hardware provided HRV metrics consistent with an “Industry-Standard” ADI ECG. Additionally, the open source cyton board produced excellent reliability with a high degree of internal consistency at a greatly reduced cost of the commonplace ECG device (~4%). This allows the cyton board to be utilized within longitudinal studies or clinical populations (e.g., concussion, stroke, myocardial infarction, mood disorders, etc.) which could provide additional information when examining differences during baseline/pre-injury, acute phases of injury, and when determining if/when clinical recovery has occurred. Ultimately, this minimizes equipment demands and/or streaming platforms required to obtain data in a time-synced manner, at a fraction of the cost. As the intended purpose of the cyton board is to quantify cerebral electrical activity (EEG), future studies are warranted that simultaneously quantify cerebral and autonomic activity to delineate associations between the two. This may ultimately be imperative to advance the mechanistic knowledge surrounding the physiological underpinnings of various diseases/disorders. Nonetheless, an important caveat is users will require background knowledge regarding the principles and technique of ECG in order to properly and accurately instrument the cyton board for HRV analysis.

Declarations

Author contribution statement

Joel S. Burma: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Andrew P. Lapointe: Conceived and designed the experiments; Performed the experiments; Wrote the paper.

Ateyeh Soroush: Performed the experiments; Wrote the paper.

Ibukunoluwa K. Oni: Conceived and designed the experiments; Performed the experiments; Wrote the paper.

Jonathan D. Smirl: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Jeff F. Dunn: Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.