Working accuracy of pulse oximetry in COVID-19 patients stepping down from intensive care: a clinical evaluation.

INTRODUCTION: UK guidelines suggest that pulse oximetry, rather than blood gas sampling, is adequate for monitoring of patients with COVID-19 if CO2 retention is not suspected. However, pulse oximetry has impaired accuracy in certain patient groups, and data are lacking on its accuracy in patients with COVID-19 stepping down from intensive care unit (ICU) to non-ICU settings or being transferred to another ICU.
METHODS: We assessed the bias, precision and limits of agreement using 90 paired SpO2 and SaO2 from 30 patients (3 paired samples per patient). To assess the agreement between pulse oximetry (SpO2) and arterial blood gas analysis (SaO2) in patients with COVID-19, deemed clinically stable to step down from an ICU to a non-ICU ward, or be transferred to another ICU. This was done to evaluate whether the guidelines were appropriate for our setting.
RESULTS: Mean difference between SaO2 and SpO2 (bias) was 0.4%, with an SD of 2.4 (precision). The limits of agreement between SpO2 and SaO2 were as follows: upper limit of 5.2% (95% CI 6.5% to 4.2%) and lower limit of -4.3% (95% CI -3.4% to -5.7%).
CONCLUSIONS: In our setting, pulse oximetry showed a level of agreement with SaO2 measurement that was slightly suboptimal, although within acceptable levels for Food and Drug Authority approval, in people with COVID-19 judged clinically ready to step down from ICU to a non-ICU ward, or who were being transferred to another hospital's ICU. In such patients, SpO2 should be interpreted with caution. Arterial blood gas assessment of SaO2 may still be clinically indicated. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

What is the working (real world) accuracy of pulse oximetry in patients with COVID-19 stepping down from intensive care unit?

In our setting, pulse oximetry shows levels of agreement with arterial blood gas assessment of haemoglobin oxygen saturation, which are slightly suboptimal, although within acceptable levels for Food and Drug Authority approval.

To our knowledge, this is the largest study to compare pulse oximetry with arterial blood gas assessment of oxygenation in people with COVID-19 in a ‘real world’ setting. Given the central role of pulse oximetry in the management of COVID-19, these are important findings.

Introduction

The COVID-19 pandemic has presented multiple challenges regarding clinical management. Accurate clinical monitoring is fundamental to inform both patient safety and management decisions. Of particular importance is the monitoring of blood oxygen saturation due to both the direct impact of the disease on the respiratory system and the complications such as thromboembolic disease.

In clinical practice, arterial blood sampling is the most accurate commonly used method to assess oxygenation. Analysers report arterial haemoglobin oxygen saturation (SaO2) using a variety of methods, including measuring the relative proportions of haemoglobin species present in the sample using spectroscopic analysis.1 This method is accurate but invasive, requiring either puncture of an artery for the specific sample, or for blood to be drawn from an arterial line. As such, procedure-related complications exist, including pain, bleeding and damage to the blood vessel. Furthermore, such methods require specialist equipment and staff, which represents an additional strain on healthcare providers.

An alternative approach used in clinical practice is the assessment of peripheral oxygen saturation (SpO2) using pulse oximetry. This serves as a rapid, non-invasive method of estimating oxygenation and has other benefits such as being continuous, so is able to highlight sudden changes in a patient’s clinical status. The National Health Service (NHS) guidance suggests that, in general, pulse oximetry rather than invasive arterial blood gas sampling should be used in people with COVID-19, stating:

Unless there are reasons to suspect CO (SpO2) with an appropriate level of nursing support.2

However, the accuracy of pulse oximetry can be influenced by multiple factors including motion, perfusion and skin pigmentation.3 It is well known that the two-wavelength spectroscopy technique employed in the pulse oximeter is inaccurate in the presence of certain haemoglobin species, such as methaemoglobin and carboxyhaemoglobin. Although pulse oximeters are tested extensively in healthy volunteers under controlled settings, the ‘working accuracy’, that is, the real-world accuracy in patients in clinical settings can, at times, be suboptimal. Previous studies have suggested suboptimal accuracy of pulse oximetry in critically unwell intensive care unit patients,4 and people with conditions such as sickle cell disease during vaso-occlusive crises,5 which may be relevant given the high incidence of thrombotic disease in people with COVID-19.6 The accuracy of pulse oximetry in these patients has also been identified as a potential contributing factor to apparent ‘silent hypoxia’ seen in COVID-19.7 Cautions and potential limitations of pulse oximetry in COVID-19 have been highlighted,3 8 and one study of 17 patients with COVID-19 on intensive care unit (ICU) suggested that SpO2 does not reliably predict SaO2.9 However, specific data on people with COVID-19 being stepped down to a non-ICU setting, or being transferred to another ICU, are lacking. Furthermore, anecdotal experience from our ICU, and others, suggests that pulse oximetry measurements (SpO2) may not accurately reflect SaO2 in patients with severe COVID-19, potentially bringing into question the appropriateness of the NHS guidance referenced above in our setting. This is particularly relevant for our patients who are being stepped down from our ICU onto non-ICU level wards, or during transfers of our patients to other ICUs, because in both of these situations, pulse oximetry-derived SpO2 would be used to monitor oxygen saturation and guide management.

Therefore, to assess if the NHS England2 guideline was appropriate for our setting, we assessed the agreement between pulse oximetry (SpO2) and arterial blood gas analysis (SaO2) in our patients with COVID-19, who were being stepped down from an ICU to a non-ICU ward or being transferred to another ICU.

Methods

We retrospectively analysed routinely collected clinical data from patients with COVID-19, admitted to one of our hospitals’ ICUs during March and April 2020. We included non-hypoxic adults with COVID-19 deemed clinically suitable to be managed in a non-ICU setting, or for ICU to ICU transfer. None of the patients stepping down to a non-ICU setting required ongoing cardiovascular support, or had a clinical indication to still have an arterial line, such as for cardiovascular or gas exchange monitoring purposes. However, in patients being transferred to another ICU, clinical indications for arterial lines remained, as such, a sensitivity analysis was conducted excluding this group. Four patients were excluded as they died on the ward, all other patients on the ward during this time were included. We extracted the final three paired SpO2 and arterial blood gas SaO2 measurements from the electronic patient record (Intellispace Critical Care & Anaesthesia, Phillips Healthcare) prior to stepping down to a less intensive ward, or before being transferred to another ICU. The sample size resulted from the time frame on which we were the clinical team on the ward. The use of three paired samples per patient aimed to increase the number of samples included, given that the total number of patients was relatively small, and to give us some indication of the variation in measures in individual patients. An adjustment was made in the analysis regarding having multiple samples from each patient.

To ensure that comparisons could be made between the two methods of assessment, the measurements (pulse oximetry and blood gas analysis) must have been taken within 15 min of each other, with no changes to the patient’s ventilation parameters (if ongoing ventilatory support), inspired oxygen concentration or positioning (eg, proning), either between assessments or in the preceding hour.

All arterial blood samples were analysed using a Radiometer ABL90 FLEX blood gas analyser. The specific method used for deriving SaO2 in this device is available in the manufacturer data sheet,10 but in brief, uses an ultrasonic haemolyser and a 256-wavelength spectrophotometer in order to measure the proportions of haemoglobin species present in the sample, namely, oxyhaemoglobin (FO2Hb), deoxygenated reduced haemoglobin (FHHb), carboxyhaemoglobin and methaemoglobin. This function is separate from the potentiometric and optical modules used to measure ion concentrations and partial pressures of O2/CO2. The analyser then calculates SaO2 according to the formula, SaO2 = (FO2Hb/(FO2Hb+FHHb)).

Pulse oximeter data were continuously collected as part of routine clinical care using Masimo LNCS DCI digital probes (placed on the patient’s finger) with signal extraction technology, displayed on a Phillips IntelliVue MP70 monitor. The use of the former technology is notable, as the manufacturers state it outperforms conventional red/infrared oximetry through the use of a multi-algorithmic approach, seeking to improve accuracy in poorly perfused or moving patients.11 This device has been validated already in the adult critical care population, demonstrating improved performance in comparison to conventional pulse oximetry in patients.12 The requirements for maintenance of these devices was discussed with the clinical engineering department of the hospital, who reported that Phillips recommend a functional check of SpO2 performance by testing the probe on the finger of the technician on a biannual basis. The manufacturer recommendations from Masimo indicate that under normal operation, no internal adjustment or recalibration of the pulse oximeter is required for this model. All devices were subject to generic safety testing and labelled with an in-date ‘licence plate’ sticker according to industry standards. Finally, in the event that a device is broken, the engineering department use test devices to ensure that it is functioning within the accepted range of accuracy following repair.

For all samples taken in our population, no additional testing or measurements took place outside those performed as part of routine clinical care, all blood samples were taken from arterial lines that had been inserted due to the severity of the patient’s condition and this report represents a retrospective evaluation which was conceived of after all the data included had been collected. Of note, the wards were fully staffed throughout this period, with a nurse to patient ratio of 1:1, with appropriately trained staff. Hence, sampling and results are likely representative of optimal clinical care in this setting, and substantial delay between sampling and testing of blood gas samples is unlikely. Additionally, blood gas machines were appropriately maintained and calibrated.

Statistical Analysis

Accuracy was assessed by establishing the level of agreement between paired SpO2 and SaO2 measurements. This was achieved using the statistical methods described by Bland and Altman,13 in which we calculated the following metrics:

Bias (mean difference, SpO2−SaO2).

Precision (SD of the differences).

Limits of agreement (LOA) (bias±1.96×SD).

Analysis was done using MedCalc V.19 (https://www.medcalc.org/). We used the Bland-Altman plot with multiple measurements per subject, in which the true value was not assumed constant in each subject. We also report the accuracy value, which is used in Food and Drug Administration (FDA) approvals of pulse oximeters, where . This is also referred to as ARMS (root mean square error). The FDA currently requires a value of ≤3.0 for finger oximeters,3 such as that used in our setting.

Authorisation

This evaluation was authorised by the Royal Brompton and Harefield NHS Foundation Trust Quality and Safety team. No external ethical approvals were required.

Patient and public involvement

Patients and the public were not involved in this study as it was not deemed to be appropriate by the authors.

Results

Ninety paired observations were analysed (three paired measurements from 30 different patients).

Table 1 shows summary characteristics of patients (n=30). Variables, such as admission D-dimer, confirmed Deep Vein Thrombosis/ Pulmonary Embolism (DVT/PE) and Extracorporeal membrane oxygenation (ECMO) received, have been included as indicators of the severity of disease in this cohort. Ethnicity and preadmission cardiovascular disease and type 2 diabetes mellitus have been included as relevant risk factors for severe COVID-19, and factors that may be of relevance regarding the accuracy of oximetry measurement. However, they have not been used for any form of subgroup analyses, as this was not the purpose of the evaluation.

Table 1

Characteristics of patients included

Sex (n)	24 males (80%)
Age (mean years)	52.6 (SD 9.96, range 29 to 71)
BMI (mean kg/m2)	28.8 (SD 5.5, range 22.1 to 41.6)
Premorbid American Society of Anesthesiologists (ASA) grade (mean)	1.5 (SD 0.5, range 1 to 2)
Preadmission cardiovascular disease	16 (53%)
Preadmission type 2 diabetes mellitus	5 (17%)
Black, Asian and Minority Ethnic (BAME)	20 (67%)
Admission D-dimer (mean)	5276 (SD 9263, range 362 to 39 840)
Confirmed Deep Vein Thrombosis (DVT)or Pulmonary Embolism (PE) (n)	15 (50%)
Treatment dose anticoagulation received (n)	29 (97%)
Tracheostomy (n)	20 (67%)
Extracorporeal membrane oxygenation (ECMO) received (n)	8 (27%)
Discharge destination (from ICU)
Same hospital, non-ICU ward (n)	23 (77%)
Other hospital ICU (n)	7 (23%)
Length of stay prior to samples being taken (mean)	19.5 days (SD 12.6)
SpO2 (mean)	96.44% (SD 2.20, range 92 to 100)
SaO2 (mean)	96.02% (SD 2.11, range 91 to 100)

BMI, body mass index; ICU, intensive care unit.

The mean difference between SpO2 and SaO2 (bias) was 0.4%, with an SD of 2.4 (precision). The limits of agreement between SpO2 and SaO2 were as follows: upper limit of 5.2% (95% CI 6.5% to 4.2%) and lower limit of −4.3% (CI −3.4% to −5.7%), which means that from our data we would expect 95% of measurements to lie within these values (see figure 1). Of note, mean pH of blood gas samples was 7.46 (SD 0.05); mean temperature at time of sample 36.88°C (SD 0.63); mean methaemoglobin 1.00% (SD 0.50); and carboxyhaemoglobin 1.50% (SD, 0.42). Of note, the accuracy value, where for our measurements is 2.44%, which lies outside of the manufacturers’ reported laboratory testing accuracy value of 1.79% for SpO2 between 70% and 100%.14 However, some difference between our study and the laboratory testing values is to be expected, given the higher levels of control possible in a laboratory setting, differences in study population characteristics, and for their assessment, paired measurements are likely to have been completely simultaneous. Importantly, this value lies within the acceptable range for FDA approval of ≤3.0%.3

Figure 1

The difference between SpO2 and SaO2 against mean of SaO2. The line with the error bars represents the 95% CI of the limit of agreement.

Samples from the same patient have the same marker.

Additionally, a sensitivity analysis was conducted, considering only people stepped down to a non-ICU ward (excluding those transferred to another ICU), for which the results were materially unchanged, with a bias of 0.8%, limits of agreement of 5.3% (upper limit) to −3.7% (lower limit).

Discussion

These findings from 90 paired samples, taken from 30 people recovering from severe COVID-19, suggest that pulse oximetry shows slightly suboptimal levels of agreement in this patient group, although remains a valuable clinical assessment tool, with limits of agreement of 5.2% to −4.3. Whether this is considered adequate will depend on the specific clinical situation in which it is being interpreted.

Importantly, the LOA seen are only slightly larger than those seen in the Masimo white paper,14 and studies of people in ICU. As such, our findings suggest that pulse oximetry is still a valuable component of assessments; however, a degree of caution is required.

As highlighted by Wilson-Baig et al,9 the cause of these suboptimal LOA remains unclear, but suggested hypotheses include altered spectral properties of high ferritin, D-dimers or other proteins raised in COVID-19 impacting oximetry precision, to COVID-19-related microvascular complications and tissue hypoxia.9 In our sample, it is likely that established factors, such as skin colour,15 may also have contributed, with 67% being of Black, Asian and Minority Ethnic (BAME) groups. Additionally, having allowed up to 15 min between SpO2 and SaO2 is likely to have contributed, as even in the absence of changes to other parameters (probe position, FiO2, ventilation parameters), fluctuations to oxygen saturation occur.3

However, it is probable that in certain situations, pulse oximetry-derived SpO2 will not be deemed sufficient to guide clinical management and that SaO2 assessment may be required.

Certain limitations and considerations are important to mention. First, although we applied stringent criteria to the selection of paired SpO2 and SaO2 recordings, we allowed up to 15 min between measurements, as such, paired measurements were not all taken precisely the same time. Oxygen levels can fluctuate over time,3 hence these natural fluctuations are likely to be one of the factors contributing to differences seen between measurements. However, as the purpose of this study is to identify the levels of agreement between SpO2 and SaO2 as assessed in clinical practice, accepting up to 15 min between the two assessments is representative of ‘real world’ clinical practice, in which these measures are unlikely to be taken at precisely the same time. Even so, further research comparing simultaneous SaO2 and SpO2 values is required before firm conclusions regarding accuracy of oximetry can be made. Second, the SpO2 and SaO2 measurements evaluated here were all ≥91%. Commercially available pulse oximeters are calibrated using healthy volunteers across a range of arterial saturations, some as low as 60%, but it is recognised that accuracy reduces at lower saturations. In these patients, it is likely that blood gas sampling would be required anyway. Peripheral vasoconstriction, due to hypothermia or vasopressor use, may also impair the pulse oximeter, but our sensitivity analysis in patients who were deemed well enough to leave ICU (and so were unlikely to be suffering from this issue) reveals the same result. Central temperature recording from patients at the time of sampling were not excessively abnormal. Additionally, given that Masimo have reported comparable accuracy data even in poorly perfused and moving patients at SpO2 values far lower than those demonstrated on our own ward,14 this likely mitigates many of these concerns.

Third, this study does not investigate the causes of differences observed, so it is not possible to establish to what extent they resulted from a variety of potentially important factors, including recent critical illness, ethnicity, pre-existing comorbidities, thrombotic disease or COVID-19 specifically. However, the aim of the study was to provide clinicians with information about the LOA of these measures in this context. Hence, identifying the causes of any discrepancies observed was beyond the scope of the evaluation. However, such questions would be of interest for future research, including studies comparing patients with other (non-COVID-19) diagnoses stepping down from ICU. Fourth, given our study population, extrapolation of these findings to patients with COVID-19 in other settings requires extreme caution. The data included in this evaluation came from patients who had been extremely unwell, many requiring ECMO, with a high burden of thromboembolic complications. These patients are not likely to be representative of all people with COVID-19, or even all patients requiring hospital admission with COVID-19. Similarly, this evaluation has used data from a single ICU, adding to the requirement for caution.

Nevertheless, even when taking these considerations into account, given previous studies showing suboptimal accuracy of pulse oximetry in certain patient groups,4 5 15 we caution that there may be clinically significant inaccuracies in SpO2 measurement in people recovering from severe COVID-19, and suggest a low threshold for SaO2 measurement if clinically justified. Of note, these concerns have arisen with the use of a market-leading pulse oximeter, further drawing into question the accuracy of older or lower quality devices. Consequently, these findings should prompt research into the accuracy of pulse oximetry in different subgroups of patients with COVID-19 and using different models of pulse oximeter, particularly given the current NHS England advice2 and the widespread use of pulse oximetry for the purposes of monitoring patients in the community, assessment of disease severity and requirement for hospital admission.

Conclusion

In our particular setting, regarding people with COVID-19 judged clinically ready to step down from ICU to a non-ICU ward, or who were being transferred to another hospital’s ICU, pulse oximetry showed a slightly suboptimal level of agreement with SaO2 measurement, however, within acceptable levels for FDA approval. In such patients, SpO2 remains a valuable tool for clinical assessment but should be interpreted with caution. Arterial blood gas assessment of SaO2 may still be required depending on the clinical context. Future research studies should assess the accuracy of pulse oximetry in other groups of patients with COVID-19, and patients stepping down from ICU with other (non-COVID-19) conditions to assess what factors are responsible of any differences found between methods of assessment.