Remote heart rate monitoring - Assessment of the Facereader rPPg by Noldus.

Remote photoplethysmography (rPPG) allows contactless monitoring of human cardiac activity through a video camera. In this study, we assessed the accuracy and precision for heart rate measurements of the only consumer product available on the market, namely the FacereaderTM rPPG by Noldus, with respect to a gold standard electrocardiograph. Twenty-four healthy participants were asked to sit in front of a computer screen and alternate two periods of rest with two stress tests (i.e. Go/No-Go task), while their heart rate was simultaneously acquired for 20 minutes using the ECG criterion measure and the FacereaderTM rPPG. Results show that the FacereaderTM rPPG tends to overestimate lower heart rates and underestimate higher heart rates compared to the ECG. The Facereader™ rPPG revealed a mean bias of 9.8 bpm, the 95% limits of agreement (LoA) ranged from almost -30 up to +50 bpm. These results suggest that whilst the rPPG FacereaderTM technology has potential for contactless heart rate monitoring, its predictions are inaccurate for higher heart rates, with unacceptable precision across the entire range, rendering its estimates unreliable for monitoring individuals.

Introduction

There is a growing interest in technologies related to the recording and monitoring of personal health parameters. In the current literature there is not yet a general agreement on the definition of personal health monitoring, which includes telecare, assistive technologies, environmental intelligence and wearable health sensors [1]. A review on the subject suggests that "monitoring of personal health" refers to any electronic device or system that monitors a health-related aspect of a person's life outside a traditional clinical or hospital setting. Examples include GPS tracking devices used with patients with mental disorders, blood pressure monitors and smart clothes capable of measuring physiological parameters [2, 3, 4, 5]. Personal health recording systems are more than just static patient data containers; they combine data, knowledge, tools and software, which help both patients with identified needs and generic consumers to become active participants in their health care [6]. Health monitoring technologies are currently being developed for a multitude of customers of all ages and health conditions aiming to integrate medical care environments with health monitoring outside traditional settings [1]. A primary driving factor is the rapid ageing of the population, which is expected to heavily impact on the performance of health systems in many countries, potentially exceeding the available medical resources [7]. Patients, policymakers, providers, tax-payers, employers, and other stakeholders have increasing interest in using personal health records to reduce healthcare costs, without affecting the quality and the efficiency of the healthcare delivery [8].

The monitoring of physiological information is very important for assessing health and access to physiological data is not only necessary in clinical setting but it is becoming increasingly so also in other environments and applications related, for example, to telemedicine [9-13], personal fitness [14-17], e-commerce [18], trading [19, 20] and mental stress caused by the interaction with technology [21-26]. Accurate and precise self-monitoring devices therefore provide potential benefits both to the individual user, by providing real-time feedback on specific physiological parameters, to the health care providers and also to those involved in retail intelligence and analytics [27, 28]. For example, typical physiological and neuroscience research techniques used to study cognitive and affective processes of individuals such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), eye tracking, biometrics of heart rate, galvanic skin response, and facial expression recognition, are becoming increasingly popular consumer neuroscience methods [29]. These techniques contribute to a deeper understanding of consumers behaviours by gathering quantitative information on their physical and mental state [30]. A growing interest has also raised in the context of emotion detection and recognition, where several devices are now available (e.g. Affectiva, Emotient—An Apple Company, Eyeris, Kairos Ar. Inc., Noldus, nViso, Realeyes) [31].

The conventional and well-established methods to capture physiological information, like electrocardiogram (ECG) or photoplethysmogram (PPG), require the application of electrodes or transducers on the skin (e.g. wet adhesive Ag/AgCl electrodes) during the monitoring period. These methods, although non-invasive, are bothersome, and perhaps irritating and distracting.

Recently, there has been an increasing interest in alternative and less obtrusive methods for monitoring physiological information such as, laser doppler velocimetry for measuring red blood cell velocity, electromagnetic approaches for heart and respiration monitoring, microwave systems or ultrasonic proximity sensor for respiration detection [32, 33].

The remote-photoplethysmography (rPPG) is a low-cost, non-contact and pervasive technique for measuring heart rate (HR) and to infer other psychophysiological data including heart rate variability, respiration rate, blood pressure and oxygenation [34, 35], quality of sleep, heart rhythm disturbances [36], and also mental stress [37] and drowsiness [38]. Its ease of use, low cost and convenience make it an attractive method for biomedical and clinical research as it allows remote heart rate measurements with a simple camera or a smartphone and it can also be integrated with augmented reality platforms [39]. The information acquired through the rPPG essentially refers to the cardiovascular functioning: the periodic blood flow and therefore the variations of blood volume in tissues that follow each cardiac cycle affects the optical properties of the tissues allowing those who are using this technology to measure HR remotely. For this reason, the reflection of the light that can be observed on the regions of the facial skin. This reflection of light is influenced not only by the various phenomena of interaction between light and skin, but also by the change in the volume of blood and the movement of the wall of blood vessels [40, 41]. Based on this principle, an accurate measurement of these changes generates a plethysmographic signal. Research has shown that, given suitable illumination, ambient light can be sufficient to obtain a plethysmographic signal [42] from changes in light reflected from facial skin and thus it is possible to measure and infer on the physiological phenomena of interest. The only hardware required to perform rPPG imaging is a standard camera. Although several techniques based on the use of infra-red (IR) or near infra-red (NIR) cameras exist [43, 44], the most developed and employed algorithms use a colour model method based on red, green and blue (RGB) imaging to acquire a signal from a distance of up to several meters [45, 46, 47]. In technical terms, the main difference between these two methods lies in the fact that both IR and NIR cameras allow a more accurate estimate of HR parameters and exploit the information provided by blood volume variation of vessels. In turn, RGB camera-based method (green light channel), does not provide such a profound and focused estimate of HR, and consider a wider and less focused range of processes which influence the optical properties of the tissues [48]. According to Wang and colleagues [49], the RGB camera-based method presents two main limitations: it is difficult to accurately estimate HR under low-light conditions and under significant ambient light fluctuations; these last two factors together with the head and body movements can drastically affect rPPG signal detection by generating strong artefacts. The general recommendation for proper measurement of HR is to keep the illumination constant and restrict individual movements.

The described methods capture the subject’s face on a video from which the plethysmographic signal is recovered using several image processing techniques and transformations. The rPPG technology, from its first presentation in 2007 [50], has been studied and developed to demonstrate its feasibility first in controlled environments and conditions, and then in increasingly realistic conditions and scenarios. Research has shown that reliable HR measurement can be achieved using low-cost, consumer-grade digital cameras and ambient light sources. Current literature on rPPG focuses on improvement over existing methodology by considering those imaging acquisition factors (environmental lighting, subject movement, and image sensor spectrum sensitivity) that, at this time, represent the main limitation to an optimal rPPG measurement and therefore to collect accurate physiological data (e.g. [51, 52]). All these methods are of interest for the easy, convenient and large-scale deployment of the non-contact HR monitoring technologies.

In this respect the goal of this study was to critically assess the accuracy of a consumer rPPG system by Noldus with respect to HR monitoring and compared its performance to a gold standard electrocardiograph. The FacereaderTM rPPG system by Noldus monitors HR activity through a patented rPPG technology [53]. Although on the market there are probably alternative and more advanced technologies such as the Vital Signs Camera by Philips, which is available for licensing to third part manufacturers, to the authors’ knowledge, the rPPG by Noldus is the only consumer product available on the market and up until today just one study involving this specific tool has been carried out [54]. Although recent evidence suggests that reliable HR measurement can be achieved using different rPPG algorithms [25, 49, 51, 55, 56], the need of further validations and cross comparisons is crucial. The objective of this study is to contribute to the improvement of this kind of technology, which has the potential to assess and monitoring the personal psychophysical status in a simple, convenient and non-invasive way with important applications (e.g. consumer analysis, e-commerce, personal fitness, driving conditions, telemedicine, customer neuroscience) and therefore to its diffusion. Our validation, in case of significant results, could assume a strong relevance regarding the potential but realistic application of remote heart rate monitoring in workplace environments.

Materials and methods

The accuracy and precision of the Facereader rPPG by Noldus (Noldus Information Technology bv, Wageningen—The Netherlands) for measuring HR was assessed with respect to an ECG criterion measure. The ProComp Infiniti T7500M (Thought Technology LTD, Toronto, Canada), is a professional 8 channel multi-modality encoder for real-time, computerized biofeedback and data acquisition used in the clinical and experimental field and constitutes a gold-standard for the measurement of physiological signals. For ECG recorded, the electrode placement sites were prepared by standardized procedures of cleaning, shaving, and abrading the skin to improve signal acquisition and to minimize noise artefacts. Three silver/silver-chloride self-adhesive electrodes were placed in proximal position on the upper torso following the second standard deviation according to the Einthoven triangle [57]. On Einthoven's triangle, the theory of unipolar electrocardiographic leads, and the interpretation of the precordial electrocardiogram). ECG data were recorder and processed using BioGraph Infinity (Thought Technology LTD, Toronto, Canada). HR data was converted to beats per minute (bpm) automatically by the data acquisition software program prior to analysis. For rPPG, the facial landmark estimation is achieved using the Active Appearance Modelling (AAM) technique [54, 58] that has been improved and integrated in the Facereader framework [59]. The AAM is a method of matching statistical models of appearance to images that consists in an efficient iterative matching algorithm by learning the relationship between perturbations in the model parameters and the induced image errors [54, 58]. Using the selected facial regions, skin colour changes are tracked for observing the periodic components caused by the blood volume changes at each heartbeat [60].

The rPPG system requires the use of a video acquisition source for recording the face of the participants. To this end a Logitech HD Pro Webcam C920–1080 HD was employed. The resolution of the videos was 1280 x 720 pixel, and the frame-rate acquisition was 30 fps for a duration of 20 minutes. In order to ensure a high quality of rPPG signal acquired from the skin surface, the whole experiment was carried out under constant lighting conditions. Both the ambient (illumination) and the screen (luminance) were controlled during the entire experiment. These parameters were assessed by an Extech 403125 digital light meter (Extech Instruments, Nashua, NH) pointed towards the screen and placed 5cm above participants’ head and laterally centred with respect to their head. Overall, the total amount of light impacting on participants’ face coming from both the ambient lighting and the screen was kept constant during the entire experiment and did not vary because of the stimuli presentation triggered by the Go/No-Go task. The distance between participants and the 24” LCD stimulus screen (Dell P2414H; www.dell.com) was approximately 60 cm (see Fig 1).

Fig 1

Experimental setup and software.

On the left, a schematic representation of the experimental setup and the devices: ProComp Infiniti T7500M (ECG recording) and Logitech HD Pro Webcam (video acquisition). On the right, a screenshot of the Facereader rPPG software by Noldus.

Twenty-Four healthy participants (11 females) of Western European descent took part in the experiment. The selection of such a specific population was mainly due to the fact that we wanted to reduce the possible effects of skin tone, which constitutes one of the main detection issues for rPPG. All participants gave written informed consent before participation. We excluded participants with cardiovascular diseases (CVD) and with neurological or cognitive disorders. The study was performed in a controlled experiment room at TSW XP Lab, Treviso—Italy (www.tsw.it) complying with the Declaration of Helsinki. The TSW XP Lab Ethics Committee approved the study. Participants were asked to sit in front of the computer screen and alternate two periods of rest with two stress tests (Go/No-Go task), as follows:

Rest [5 min]

Stress test (Go/No-Go task) [5 min]

Rest [5 min]

Stress test (Go/No-Go task) [5 min]

The Go/No-Go task required participants to press the spacebar when they saw a green rectangle appeared (Go) but refrain from pressing the spacebar when they saw a blue rectangle (No-Go). The blue and green rectangles could be either vertically or horizontally aligned. The vertical rectangle had a higher probability of being green (Go trial) and the horizontal rectangle had a higher probability of being blue (No-Go trial). Participants got information about the orientation of the rectangle (cue) shortly before the colour of the rectangle was revealed [61].

The goal of the experiment was to collect enough HR data spanning a range of BPMs as wide as possible. HR was simultaneously acquired for 20 minutes using the ECG criterion measure and the Facereader rPPG by Noldus. With the aim of improving the overall data quality, the rPPG analysis was carried out offline. Given that Facereader allows collecting data at 8 Hz and that BioGraph Infinity allows pre-processing at 8 Hz, we decided to use this frequency for processing cardiac data. Moreover, since movements from the participant may cause artefacts in the HR monitoring, we have manually removed any motion-induced artefacts. Agreement between the Facereader rPPG and the ECG gold standard was estimated using the Bland-Altman method, adapted to consider repeated measures from the same person when the true value varies over time [62, 63]. Bland-Altman plots are widely used to evaluate the agreement among two different instruments or two measurements techniques. This provided an estimate of agreement between the rPPG and ECG in the instantaneous value of the changing heart rate. We also estimated the intraclass correlation coefficient (ICC), which indicates how strongly units in the same group resemble each other, as a complementary measure of agreement. All statistical analyses were performed using StataCorp Stata 15.1.

Results

Table 1 shows means, standard deviations (SD) and ranges for age, weight, height, and Body Mass Index (BMI) of the participants.

Table 1

Means, standard deviations and ranges for Age, Weight, Height, and BMI.

	Male		Female
Variables	Mean ± SD	Range	Mean ± SD	Range
Age (Years)	31 ± 5	23–38	27 ±3	23–31
Weight (kg)	76 ± 10	62–92	59 ± 8	45–71
Height (cm)	178 ± 5	168–189	166 ± 3	160–174
BMI (kg/m2)	24 ± 2	20–27	21 ± 3	16–25

The dataset consisted of 230400 samples of data (1200 seconds x 24 participants x 8 Hz). However, since both the ECG and the Facereader rPPG produced several disruptions to continuous HR detection, the dataset was reduced by around 23%. The final dataset was then made of 177629 samples. Fig 2 shows all time-synced ECG and Facereader rPPG ordered by ECG data in aggregate, with the rPPG estimate demonstrating wide variability and lack of responsiveness to changing heart rate recorded by ECG.

Fig 2

Ordered HR data (Facereader rPPG vs. ECG).

Data have been ordered according to the frequencies collected by the criterion measure (ECG). (n = 230400).

The Facereader rPPG revealed a mean bias of 9.8 bpm (95% CI—Confidence Interval: 9.7 to 9.9 bpm). As to the limits of agreement (LoA) between the Facereader rPPG and criterion measure the upper LoA was 46 bpm, whereas the lower LoA was -26 bpm (Fig 3). The ICC between Facereader rPPG and gold standard ECG was 0.75 (95% CI: 0.64 to 0.86).

Fig 3

HR data (Facereader vs. ECG).

Bland-Altman Plot indicating mean difference in HR detection between the Facereader rPPG and ECG criterion measure.

Furthermore, the extent of agreement varied substantially across the range of heart rates (see Fig 4). The Facereader rPPG tends to overestimate lower heart rates (< 80 bpm) compared to the ECG and underestimates higher heart rates (> 80 bpm) compared to the ECG. Since the previous Bland-Altman Plot (see Fig 3) and relative statistics related to the mean bias ignore the general trend, we also provide the Bland-Altman plot with the trend incorporated (Fig 4). At 70 bpm the rPPG under-estimated by just 5 bpm compared to the mean, but with very wide limits of agreement from -18 to 28 bpm. At 80 bpm the rPPG under-estimated by 17 bpm, again with wide limits of agreement from -9 to 43 bpm.

Fig 4

HR data with trend (Facereader vs. ECG).

Bland-Altman Plot modelling a trend over continuous heart rate indicating mean difference in HR detection between the Facereader rPPG and ECG criterion measure.

Discussion

The aim of the present study was to assess in a controlled experimental setting the accuracy of the Facereader rPPG for remote HR monitoring with respect to a gold standard electrocardiograph. Although, recent evidence suggests that reliable HR measurement can be achieved using different rPPG algorithms [25, 49, 51, 55, 56], the need of further validations and cross comparisons is crucial.

To the authors’ knowledge the rPPG by Noldus is the only consumer product available on the market and up until today just one study involving this specific tool has been carried out [54]. Our results show that the agreement between the Facereader rPPG and the ECG is poor. The Facereader tends to over-estimate lower heart rates, and under-estimates higher heart rates compared to the ECG and the error ranges from almost -30 up to +50 bpm (see Fig 2). A first and single validation of the Facereader rPPG has been carried out by the inventors of this patented technology [53, 60]. In their study, the estimated rPPG signal was compared to a ground truth contact PPG sensor and the results of the objective performance tests show strong correlation between the estimated remote and the reference HR [60]. However, the validation was carried out using an unspecified PPG sensor, which cannot be considered a gold standard, no information was provided on the exact value of the correlation and no ICC was calculated. These issues limit our ability to compare our results with theirs. When comparing our results with those of two other related studies, it is possible to observe that in Gonzalez Viejo et al. [54] no correlation was found between the HR results of the oscillometric monitor and those obtained by analysing the video with the Facereader; and that Tasli HE et al. [60], which proposed a novel signal processing approach to extract the periodic component of the raw colour signal for the heart rate and variation estimation, found that the higher HR values (> 100 bpm) are underestimated by Facereader (Fig 3, points outside the confidence limit).

The lack of accurate and precise estimates from the rPPG monitor are in line with our expectations; the technology employed by the Facereader is still in its infancy and other improvements are needed to increase the precision and accuracy of this tool. The Facereader technology and the underlying technique do not overcome the limits of PPG methods in terms of accuracy of HR measurements [17].

With respect to validation studies involving other rPPG methods and ECG criterion measures, the literature is very dense, and the general idea is that rPPG seems to perform quite well. Our results are in contrast with these recent studies in which HR was reliably estimated in various scenarios through rPPG when compared to an ECG reference signal. For example, van Gastel et al. [64] found that, when rPPG was employed for estimate the cardiac activity of infants, the pulse rate can be detected with an average error which ranges from 1.5 to 2.1 bpm and overall the correct HR is detected for 87% of the time; also Fukunishi et al. [65] performed an experiment to measure participants at rest and under cognitive stress in which the remote measurement of HR (rPPG) showed an high correlation with the ECG (around 99% accuracy). In contrast with previous investigations [50, 63, 64], our study has a large sample size (24 participants) and dataset (230400 samples): these elements together are important factors of study validity.

In general, technical features of the camera, body and head movements and ambient lighting are the main causes for the inaccuracy of any rPPG acquisition. Compatibly with the guidelines suggested by the FaceReader Reference Manual about lighting (light diffuse, no strong shadows on the face, preferably from a frontal direction), the camera and its setting (which should be able to capture a frontal view of the subject's face throughout the session with a recommended video resolution: 1280 x 720; frame rate: at least 15 fps, preferred 30 fps; distance between camera and subject: 0.5–1 m) and the skin tones of the participants, our experimental design was specifically built to control all these factors which if not respected would constitute limits to the accuracy of an optimal rPPG measurement. The fact the ICC, which indicates how strongly units in the same group resemble each other, is good (0.75), depends on the way the experimental procedure was conceived: all the participants performed the same task in the same order, under the same experimental conditions, which included stable and constant environmental lighting, temperature and minimization of head and body movements.

In case other commercial rPPG will be developed, future work will be devoted to a more naturalistic assessment and validation of these devices. Our suggestion for future research is to deepen the study of consumer products that can be used without the need for special expertise or sophisticated software and hardware. In fact, our study did not aim to assess the effectiveness and the accuracy of under development rPPG algorithms and methodologies. We aimed to evaluate the precision and accuracy of a consumer product, easily accessible by final users and industrial partners interested in monitoring HR. In this respect, our further studies will explore remote psychophysiological monitoring technologies and, considering its main imaging acquisition issues will include assessment of environmental lighting conditions, correlation with head and body movements, as well as widen the sample and testing conditions. This will allow to evaluate potential variability of the instrument performances with respect to different users’ characteristics (e.g. skin type, ages) and different application scenarios (e.g. clinical settings, fitness environments, driving conditions, working environments).

Conclusion

The FacereaderTM rPPG allows for remote HR measurement through a video camera. Although the Facereader™ rPPG’s algorithm does not represent the state-of-the-art, our assessment revealed that the agreement between the Facereader rPPG and the ECG is poor, with a mean bias of 9.8 bpm compared to the ECG gold standard. The mean bias is highly influenced by the fact that the Facereader tends to over-estimate lower heart rates, and under-estimates higher heart rates compared to the ECG. The error ranges from almost -30 up to +50 bpm. The infancy of this peculiar technology may potentially explain these results. Future investigations will further allow improvement and diffusion of this kind of technology, which has the potential to assess and monitor the personal psychophysical status in a simple, convenient and non-invasive way with important applications beyond clinical patients’ monitoring [66], such as consumer analysis, e-commerce, personal fitness, driving conditions, telemedicine, and customer neuroscience.

Study data.

(XLSX)

Click here for additional data file.

11 Jun 2019

Submitted filename: Response to Reviewer_16489.docx

Click here for additional data file.

12 Aug 2019

PONE-D-19-16489

Remote heart rate monitoring - Assessment of the FacereaderTM rPPg by Noldus

PLOS ONE

Dear Simone Benedetto,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Sep 08 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Wajid Mumtaz

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

1. In your manuscript, "Caucasian" should be changed to “white” or “of [Western] European descent” (as appropriate).

2. In the manuscript and in the online submission form, please clarify whether the affiliation with TSW-XP LAB constitutes a conflict of interest.

3. Thank you for submitting the above manuscript to PLOS ONE. During our internal evaluation of the manuscript, we found significant text overlap between your submission and the following previously published works:

https://doi.org/10.1109/EMBC.2012.6346371

https://doi.org/10.1007/s11704-016-6243-6

https://doi.org/10.1109/EMBC.2015.7319857

We would like to make you aware that copying extracts from previous publications, especially outside the methods section, word-for-word is not acceptable. In addition, the reproduction of text from published reports has implications for the copyright that may apply to the publications.

Please revise the manuscript to rephrase the duplicated text, cite your sources, and provide details as to how the current manuscript advances on previous work. Please note that further consideration is dependent on the submission of a manuscript that addresses these concerns about the overlap in text with published work.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This article on evaluating a commercial rPPG solution is well-written and researched in its introduction of rPPG, and the method and presentation of the study look good to me.

The authors state that their goal is only to review end-user rPPG products, of whom there are not many since rPPG is still in its early development. This somewhat lowers the contribution the article could make to the rPPG research community.

Major comments

The main issue I see with this article is that most state-of-the-art algorithms, for which such an independent evaluation would be of the major interest, are not in use commercially.

Although the authors clearly state that they want to review only commercially available algorithms, there should be greater emphasis (i.e., abstract/conclusion) that the reviewed algorithm does NOT represent the state-of-the-art.

On this note, a commercial product that is probably more advanced (and not mentioned here) is the VitalSigns Camera by Philips (http://www.ip.philips.com/licensing/program/115), but I am not sure how easy it is to get access. This needs to be reflected in the article.

Another question that I had to ask myself is how this paper could help rPPG development going forward. In my opinion, open-sourcing the dataset (of whom there are not many) for evaluation of any rPPG algorithm would be a bigger contribution than this "one off" evaluation of an outdated algorithm.

Minor comments

l. 31 This statement is only acceptable of the VitalSigns Camera by Philips is not classified as a consumer product. This should be clarified (outside the abstract).

l. 39 After a quick look at the information available, it does not seem that the Noldus FaceReader uses "recently developed" rPPG technology. The papers cited by Noldus are as old as 2014. I would drop the words "recently developed".

l. 87 There is indeed a growing interest in affect detection, but a citation is missing. The recent review "Deep Learning for Human Affect Recognition: Insights and New Developments" to be published in IEEE Transactions on Affective Computing could be suitable: https://ieeexplore.ieee.org/abstract/document/8598999

l. 139 When talking about the state-of-the-art, the paper should mention "DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks" by Weixuan Chen and Daniel McDuff (published at ECCV 2018), which is the most advanced approach in rPPG that I am aware of.

l. 150 The last two sentences of this paragraph lead me to expect a list of areas or methods? This could be rephrased.

l. 156 This could be a point to mention the product by Philips.

l. 228 Typo: "may causes artefacts" => "may cause artefacts"

l. 293 Again, this phrasing may have to be changed since there is the product by Philips.

l. 303 That is exactly the problem in rPPG - most studies are evaluating on their own private databases which are not comparable. Researchers should benchmark on publicly available datasets (e.g., MAHNOB-HCI) or publish their own. Why not publish this dataset?

l. 320 "larger"

l. 320 Again same point from l. 303.

l. 322 This statement of which cases are the "most plausible" should rephrased or backed up somehow.

l. 329 This is confusing: How can other commercial rPPG be evaluated if they don't exist?

Reviewer #2: The paper presents an assessment of FaceReader by Noldus, a product developed to measure remote photoplethysmographic signals and estimate pulse rate.

To be more useful to the authors, my background is as a researcher in biomedical signal and image processing. One aspect of my research consists in sensing and estimating physiological parameters from video recordings by the analysis of remote photoplethysmographic signals (which is well-correlated with the topic of the paper they submit).

I hope that these comments will help the authors to improve their paper.

1. General comments:

Introduction is in my opinion too long and can be shortened. E.g. from L. 139 to 152: the authors have specifically chosen to present two techniques (L. 139 to 152): it is not exhaustive and not particularly correlated with the main goal of this study (assessing the potential and limits of FaceReader and not presenting a signal, image or deep learning method). These parts can therefore be removed.

A section or a dedicated part in the discussion that presents comparison of the results with those from related studies (ref. [67] and [69], the latter being the original contribution) must be added.

- From [67], page 15: “When comparing the HR results from the oscillometric monitor with those obtained by analyzing the videos through FaceReader™ there was no correlation between the two methods”.

- From [69], figure 3: high HR values (>100 bpm) are underestimated (points outside the confidence limits).

Performances of FaceReader: what is the impact of subject-experiment-hardware specifications like image resolution, sensor quality (quantum efficiency), distance camera-subject, skin tone? A limitation section or a complete paragraph must be added in the discussion. A Bland-Altman for each skin tone would have been of great interest because the rPPG signal to noise ratio tends to decrease with darker skin colors. The same remark goes for motion (we could expect a worse agreement).

L. 350 to 352: these conclusions cannot be supported by the experiments (illumination is constant and artifacts due to motion have been removed).

Data: could the authors provide the data used to compute the results and, at least, some excerpts (e.g. frames from some participants)? It seems that no link(s) or archive(s) were provided.

2. Specific comments:

References: the paper is not a survey and, in my opinion, employs too many references. In addition, some references are sometimes not well chosen or are presented in too large groups (5 to 10). I would then suggest reducing this number by dropping some unrelated references. See for example L. 64, 77, 78, 87, 105 (ref. 45 to 47 are no related to augmented reality, the authors should instead cite 64), 118, 159.

On the other side, some parts need additional references:

- L. 85: “consumer neuroscience methods”

- From L. 90 to 98

- L. 128 (“Nevertheless, this method provides highly usable and accessible daily health monitoring and it is recognized to be more robust to motion artifacts if compared to infrared rPPG”)

- L. 147, ref. [64]: a better reference can be selected

- L. 177: “The ProComp Infiniti […] constitutes a gold standard for the measurement of physiological signals”

- L. 181: Einthoven triangle

L. 118 to 123: “NIR cameras allow a deeper estimate of HR” what “deeper” means in this context? I found this paragraph unclear.

L. 124: main limitations: I would recommend adding, in complement to illumination considerations, that motion can drastically affect PPG signals by engendering strong artifacts.

L. 130: in reference [61], the authors studied contact PPG signals. I am not sure that their conclusions can be directly transposed to remote PPG.

L. 133: Takano et al. [87] proposed a method to detect rPPG and estimate pulse rate in 2007, before [50].

L. 140 to 142 are misleading: the CNN is a deep learning model that detect skin pixels in a frame. rPPG is subsequently computed on these pixels of interest.

L. 194 to 196: the experiments are in fact very controlled (lab conditions, no motion), which contrast with some statements from the introduction, in particular in L. 138 or from L. 160 to 167.

Go/NoGo: I would recommend adding some details about the task.

Artefact removal (L. 229): how? Manually?

I believe that other commercial solutions like FaceReader are available, in particular on mobile devices (Android, iOS). After a quick search on Google, I also found i-virtual (http://www.i-virtual.fr/cardiasens.html) which proposes “Cardiasens”, a product apparently similar to FaceReader (the site is in French).

3. Minor corrections

- Introduction: I would recommend removing the quote marks.

- Suggestion for L. 63: mental health patients -> patients with mental disorders (it is a suggestion)

- Suggestion: imaging PPG (iPPG) instead of rPPG (remote can also correspond to the measurement of PPG signals at a distance using LEDs, photodetectors and optical components).

- Suggestion: I recommend the use of pulse rate instead of heart rate throughout the entire article (heart rate being more employed for the ECG). PPG -> pulse rate, ECG -> heart rate.

- Abbreviations that could be removed: VCG, EM, CNN.

- Abbreviations that must be defined: CI (confidence interval).

- L. 123: Kado and colleagues -> Wang and colleagues

- Suggestion: the organization of the paper can be presented at the end of the introduction.

- L. 181: recorder -> recorded

- Format of references L. 301 and 308 (Tasli et al., Benedetto et al.)

Reviewer #3: The authors compared the digital camera-based Facereader software’s ability to measure heart rate to heart rate as measured by ECG signal and found that Facereader performs poorly at (relative, but completely physiological) extremes of heart rate with error ranging from -30 to +50 bpm (mean error 9.8 bpm).

1. The authors have carefully designed their study in a way to be easily reproducible, with clear diagrams as to its setup.

2. The background provided is very detailed, with a clear overview of the underlying technologies.

3. This work aligns well with the current efforts on measuring biomarkers using the new technology. The interest in this area is growing at a fast pace and there is a need for measuring such markers as accurately as possible with as less distraction to the user as possible.

However, a few points remain that are a bit unclear in analysis:

1. It seems potentially unfair to compare Facereader only to ECG signal. Why not also compare with on-finger PPG, as in the original Facereader validation study? PPG is more commonly used for heart rate measurement than ECG is and is generally accepted as being valid. So why did authors choose not to do this?

2. They mention that their test participants were exclusively Caucasian and that this was a study limitation, but don’t touch more on why their study was designed in this manner. This point appears to have been raised by previous Reviewer #1, but the authors did not address this in detail beyond citing it as a study limitation.

3. Study participants were limited to those without neurological or cognitive disorders. What about cardiac disorders? For instance, if any participant had some type of arrhythmia, such as AFib or PVCs, those could reasonably throw off any heart rate calibration.

4. More background about the method and the basic statistics collected could have been useful. In particular, it would be useful to include more detail about the Go/No-Go task, and to include metrics such as mean/max/min for each of rest period 1, stress test 1, rest period 2, and stress test 2. It would also be useful to see the mean difference between each transition. This data was requested by Reviewer #1, but the authors did not agree with this request.

5. On a more minor note - there were a number of grammatical errors throughout and it would be good to have an external reviewer edit the paper for quality of presentation.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

30 Sep 2019

Response to reviewers

We would like to thank the reviewers for the time they spent in critically reading our manuscript. We deeply appreciate their feedback and we have addressed their comments/remarks below:

Journal Requirements:

1. In your manuscript, "Caucasian" should be changed to “white” or “of [Western] European descent” (as appropriate).

Authors: We changed “Caucasian” to Western European descendent.

2. In the manuscript and in the online submission form, please clarify whether the affiliation with TSW-XP LAB constitutes a conflict of interest.

Authors: We added the required information. Here below the details.

Funding: This work was supported by TSW XP Lab, which only provided financial support in the form of authors’ salaries [SB, CC, NB] and/or research materials. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the “Author Contributions” section.

Competing interests: The authors declare that the funding organization (TSW XP Lab) only provided financial support in the form of authors’ salaries [SB, CC, NB] and/or research materials, and did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the “Author Contribution” section. The authors also confirm that this commercial affiliation does not alter their adherence to all PLOS ONE policies on sharing data and materials.

3. Thank you for submitting the above manuscript to PLOS ONE. During our internal evaluation of the manuscript, we found significant text overlap between your submission and the following previously published works:

https://doi.org/10.1109/EMBC.2012.6346371

https://doi.org/10.1007/s11704-016-6243-6

https://doi.org/10.1109/EMBC.2015.7319857

We would like to make you aware that copying extracts from previous publications, especially outside the methods section, word-for-word is not acceptable. In addition, the reproduction of text from published reports has implications for the copyright that may apply to the publications.

Please revise the manuscript to rephrase the duplicated text, cite your sources, and provide details as to how the current manuscript advances on previous work. Please note that further consideration is dependent on the submission of a manuscript that addresses these concerns about the overlap in text with published work.

Authors: We apologize for any overlap in text. We have not copied from these three reviews, but instead may have referred to the same sources at they have. In each case, we have carefully cited in our manuscript those original older sources we have in common with the reviews you list. However, to avoid re-using the same wording as the original publications, we rephrased the relevant text in our manuscript.

Reviewers' comments:

Reviewer #1: The main issue I see with this article is that most state-of-the-art algorithms, for which such an independent evaluation would be of the major interest, are not in use commercially. Although the authors clearly state that they want to review only commercially available algorithms, there should be greater emphasis (i.e., abstract/conclusion) that the reviewed algorithm does NOT represent the state-of-the-art.

Authors: We have given more emphasis to the fact that the validated algorithm does not represent the state of the art by further discussing this point both in the conclusions.

Reviewer #1: On this note, a commercial product that is probably more advanced (and not mentioned here) is the VitalSigns Camera by Philips (http://www.ip.philips.com/licensing/program/115), but I am not sure how easy it is to get access. This needs to be reflected in the article

Authors: Philips VitalSigns Camera is not a commercial product, but rather a technology available for licensing to 3rd party manufacturers. We have nevertheless introduced a reflection on Philips' product in the introduction of the paper.

Reviewer #1: Another question that I had to ask myself is how this paper could help rPPG development going forward. In my opinion, open-sourcing the dataset (of whom there are not many) for evaluation of any rPPG algorithm would be a bigger contribution than this "one off" evaluation of an outdated algorithm.

Authors: We agree with the reviewer: providing an open-source dataset would be useful, though we would wish to avoid over-fitting of algorithms to any single dataset. However, in our specific case, none of the participants gave their written consent to publish a video that depicts their face. We will consider seeking consent to open-source the datasets in our future investigations.

Reviewer #1: l. 31 This statement is only acceptable of the VitalSigns Camera by Philips is not classified as a consumer product. This should be clarified (outside the abstract).

Authors: As stated before, the product by Philips is not a consumer product.

Reviewer #1: l. 39 After a quick look at the information available, it does not seem that the Noldus FaceReader uses "recently developed" rPPG technology. The papers cited by Noldus are as old as 2014. I would drop the words "recently developed".

Authors: We agree with the reviewer. We removed the words “recently developed”.

Reviewer #1: l. 87 There is indeed a growing interest in affect detection, but a citation is missing. The recent review "Deep Learning for Human Affect Recognition: Insights and New Developments" to be published in IEEE Transactions on Affective Computing could be suitable: https://ieeexplore.ieee.org/abstract/document/8598999

Authors: This is definitely a very nice piece of work. We added this citation.

Reviewer #1: l. 139 When talking about the state-of-the-art, the paper should mention "DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks" by Weixuan Chen and Daniel McDuff (published at ECCV 2018), which is the most advanced approach in rPPG that I am aware of.

Authors: Thanks for the suggestion. We have added this citation.

Reviewer #1: l. 150 The last two sentences of this paragraph lead me to expect a list of areas or methods? This could be rephrased.

Authors: We agreed with Reviewer #2 “introduction is in my opinion too long and can be shortened from L. 139 to 152”. We have therefore deleted this part.

Reviewer #1: l. 156 This could be a point to mention the product by Philips.

Authors: We have introduced here a reflection on Philips' product.

Reviewer #1: l. 293 Again, this phrasing may have to be changed since there is the product by Philips.

Authors: As state above, although Philips VitalSigns Camera is not a commercial product, but rather a technology available for licensing to 3rd party manufacturers, we have introduced a reflection on Philips' product in the introduction of the paper, but not in the conclusion.

Reviewer #1: l. 303 That is exactly the problem in rPPG - most studies are evaluating on their own private databases which are not comparable. Researchers should benchmark on publicly available datasets (e.g., MAHNOB-HCI) or publish their own. Why not publish this dataset?

Authors: As stated before, we agree with the reviewer that the need of open-source dataset would be useful. However, in this specific case, none of the participants gave their written consent to publish a video that depicts their face.

Reviewer #1: l. 322 This statement of which cases are the "most plausible" should rephrased or backed up somehow.

Authors: The sentence has been rephrased.

Reviewer #1: l. 329 This is confusing: How can other commercial rPPG be evaluated if they don't exist?

Authors: We apologize for the confusing statement and we have edited the paragraph for clarity.

______________________________________________________________________________________

Reviewer #2: Introduction is in my opinion too long and can be shortened. E.g. from L. 139 to 152: the authors have specifically chosen to present two techniques (L. 139 to 152): it is not exhaustive and not particularly correlated with the main goal of this study (assessing the potential and limits of FaceReader and not presenting a signal, image or deep learning method). These parts can therefore be removed.

Authors: We agreed with Reviewer #2. This part (i.e. L.139 to 152) has been removed.

Reviewer #2: A section or a dedicated part in the discussion that presents comparison of the results with those from related studies (ref. [67] and [69], the latter being the original contribution) must be added.

- From [67], page 15: “When comparing the HR results from the oscillometric monitor with those obtained by analyzing the videos through FaceReader™ there was no correlation between the two methods”.

- From [69], figure 3: high HR values (>100 bpm) are underestimated (points outside the confidence limits).

Authors: We agree with the reviewer and we have carefully added a part in the discussion section where the results of the two related studies are presented and compared with ours.

Reviewer #2: Performances of FaceReader: what is the impact of subject-experiment-hardware specifications like image resolution, sensor quality (quantum efficiency), distance camera-subject, skin tone? A limitation section or a complete paragraph must be added in the discussion.

Authors: We agree with reviewer and we included a section on the Facereader rPPG performances issues in the discussion.

Reviewer #2: A Bland-Altman for each skin tone would have been of great interest because the rPPG signal to noise ratio tends to decrease with darker skin colors. The same remark goes for motion (we could expect a worse agreement).

Data: could the authors provide the data used to compute the results and, at least, some excerpts (e.g. frames from some participants)? It seems that no link(s) or archive(s) were provided.

Authors: Our sample was constituted by Western European descents (white skin). Therefore, we did not measure skin colour differences between our participants. As to the data, all data are already fully available without restriction in the original version of the manuscript.

Reviewer #2: L. 350 to 352: these conclusions cannot be supported by the experiments (illumination is constant and artifacts due to motion have been removed).

Authors: We apologize for the confusing statement and we have edited the paragraph for clarity.

Reviewer #2: References: the paper is not a survey and, in my opinion, employs too many references. In addition, some references are sometimes not well chosen or are presented in too large groups (5 to 10). I would then suggest reducing this number by dropping some unrelated references. See for example L. 64, 77, 78, 87, 105 (ref. 45 to 47 are no related to augmented reality, the authors should instead cite 64), 118, 159.

Authors: We agree with the reviewer. We reduced the number of references by dropping the unrelated ones.

Reviewer #2: On the other side, some parts need additional references:

- L. 85: “consumer neuroscience methods”

- From L. 90 to 98

- L. 177: “The ProComp Infiniti […] constitutes a gold standard for the measurement of physiological signals”

- L. 181: Einthoven triangle

Authors: We added citations for all of them. As to the Procomp Infiniti, the device is a professional tool used in the clinical and experimental field (for example in biofeedback training and therapy) and constitutes a gold-standard for the measurement of ECG signal. It has been employed in more than 300 (published) experimental studies, including our previous study on the assessment of the Fitbit Charge 2 for monitoring heart rate, published in PlosOne one year ago. (Benedetto, S., Caldato, C., Bazzan, E., Greenwood, D. C., Pensabene, V., & Actis, P. (2018). Assessment of the Fitbit Charge 2 for monitoring heart rate. PloS one, 13(2), e0192691).

Reviewer #2: L. 128 (“Nevertheless, this method provides highly usable and accessible daily health monitoring and it is recognized to be more robust to motion artifacts if compared to infrared rPPG”)

Authors: The sentence has been removed.

Reviewer #2: L. 147, ref. [64]: a better reference can be selected

Authors: This part (i.e. L.139 to 152) has been removed.

Reviewer #2: L. 118 to 123: “NIR cameras allow a deeper estimate of HR” what “deeper” means in this context? I found this paragraph unclear.

Authors: We apologize for this. The sentence has been rephrased.

Reviewer #2: L. 124: main limitations: I would recommend adding, in complement to illumination considerations, that motion can drastically affect PPG signals by engendering strong artifacts.

Authors: We agree with the reviewer. We added the suggested sentence.

Reviewer #2: L. 130: in reference [61], the authors studied contact PPG signals. I am not sure that their conclusions can be directly transposed to remote PPG.

Authors: The sentence has been removed.

Reviewer #2: L. 133: Takano et al. [87] proposed a method to detect rPPG and estimate pulse rate in 2007, before [50].

Authors: We apologize for this. The sentence has been rephrased.

Reviewer #2: L. 140 to 142 are misleading: the CNN is a deep learning model that detect skin pixels in a frame. rPPG is subsequently computed on these pixels of interest.

Authors: This part (i.e. L.139 to 152) has been removed.

Reviewer #2: L. 194 to 196: the experiments are in fact very controlled (lab conditions, no motion), which contrast with some statements from the introduction, in particular in L. 138 or from L. 160 to 167.

Authors: We apologize for this. The sentence has been rephrased.

Reviewer #2: Go/NoGo: I would recommend adding some details about the task.

Authors: We added more details regarding the task.

Reviewer #2: Artefact removal (L. 229): how? Manually?

Authors: Yes, manually. We added this detail in the revised manuscript.

Reviewer #2: I believe that other commercial solutions like FaceReader are available, in particular on mobile devices (Android, iOS). After a quick search on Google, I also found i-virtual (http://www.i-virtual.fr/cardiasens.html) which proposes “Cardiasens”, a product apparently similar to FaceReader (the site is in French).

Authors: The solutions available on mobile devices were excluded because they are closer to game/fun applications rather than real commercial ones. For this reason, we did not consider them as reliable benchmark. As to Cardiasens, we contacted several times the company but did not receive any reply. We also verified if any patent or publication was referred to Cardiasens but nothing was found.

Reviewer #2:

- Introduction: I would recommend removing the quote marks.

- Suggestion for L. 63: mental health patients -> patients with mental disorders (it is a suggestion)

- Abbreviations that could be removed: VCG, EM, CNN.

- Abbreviations that must be defined: CI (confidence interval).

- L. 123: Kado and colleagues -> Wang and colleagues

- L. 181: recorder -> recorded

- Format of references L. 301 and 308 (Tasli et al., Benedetto et al.)

Authors: These issues were addressed in the revised manuscript.

Reviewer #2:

- Suggestion: imaging PPG (iPPG) instead of rPPG (remote can also correspond to the measurement of PPG signals at a distance using LEDs, photodetectors and optical components).

- Suggestion: I recommend the use of pulse rate instead of heart rate throughout the entire article (heart rate being more employed for the ECG). PPG -> pulse rate, ECG -> heart rate.

- Suggestion: the organization of the paper can be presented at the end of the introduction.

Authors: We thank the reviewer for these suggestions, but we would prefer to keep the text as it is. As to the employment of iPPG in lieu of rPPG, we do not think it is correct. Noldus infact refers to rPPG and to heart rate for its patented technology (see https://www.noldus.com/facereader/remote-photoplethysmography-facereader). Furthermore, the large majority of the literature refers to rPPG. As to the suggestion regarding the organization of the paper at the end of the introduction, we do not think it is necessary.

______________________________________________________________________________________

Reviewer #3: 1. It seems potentially unfair to compare Facereader only to ECG signal. Why not also compare with on-finger PPG, as in the original Facereader validation study? PPG is more commonly used for heart rate measurement than ECG is and is generally accepted as being valid. So why did authors choose not to do this?

Authors: We decided to employ an ECG signal, just because the ECG is the reference method for this kind of assessment. Unfortunately, the on-finger PPG cannot be considered a gold standard, and therefore cannot be employed in any (official) validation study. If we used anything other than the ECG, we would not know if disagreements were because of the Facereader or the reference measure.

Reviewer #3: 2. They mention that their test participants were exclusively Caucasian and that this was a study limitation, but don’t touch more on why their study was designed in this manner. This point appears to have been raised by previous Reviewer #1, but the authors did not address this in detail beyond citing it as a study limitation.

Authors: We addressed this issue by reinforcing the motivations in the Materials and Methods’ section.

Reviewer #3: 3. Study participants were limited to those without neurological or cognitive disorders. What about cardiac disorders? For instance, if any participant had some type of arrhythmia, such as AFib or PVCs, those could reasonably throw off any heart rate calibration.

Authors: None of the participants suffered from cardiac disorders. We now include this information.

Reviewer #3: 4. More background about the method and the basic statistics collected could have been useful. In particular, it would be useful to include more detail about the Go/No-Go task, and to include metrics such as mean/max/min for each of rest period 1, stress test 1, rest period 2, and stress test 2. It would also be useful to see the mean difference between each transition. This data was requested by Reviewer #1, but the authors did not agree with this request.

Authors: We thank the reviewer for this comment. We have included more details about the Go/No-Go task in the Materials and methods section. Regarding the additional metrics required, we think that these kinds of parameters were not part of the research objectives. The objective of the study was to assess the accuracy of a consumer rPPG system with respect to HR monitoring and compare its performance to the gold standard ECG collecting a large amount of data, spanning the widest possible range of HR frequencies and we have not catalogued HR data across the stress/rest phases.

Reviewer #3: 5. On a more minor note - there were a number of grammatical errors throughout and it would be good to have an external reviewer edit the paper for quality of presentation.

Authors: We would like to thank the reviewer for pointing out this issue. We have carefully revised the manuscript.

Submitted filename: Response to Reviewers_Assessment of the Facereader rPPG.docx

Click here for additional data file.

8 Nov 2019

Remote heart rate monitoring - Assessment of the FacereaderTM rPPg by Noldus

PONE-D-19-16489R1

Dear Dr. Simone Benedetto,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Wajid Mumtaz

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I am happy with the changes made regarding my previous comments. As a result I consider the paper ready for publication, given the following minor typos are addressed:

- l. 188-189: There is an unmatched parenthesis and the sentence seems incomplete

- l. 308: There should be no comma after "Although"

Reviewer #3: I'm still not convinced about the lack of finger PPG - if you want to use ECG as a reference standard fine, but it would be helpful to also have finger PPG and be able to discuss commonly used heart rate measurements.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: No

14 Nov 2019

PONE-D-19-16489R1

Remote heart rate monitoring - Assessment of the FacereaderTM rPPg by Noldus

Dear Dr. Benedetto:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Wajid Mumtaz

Academic Editor

PLOS ONE