Accuracy and usability of AcuPebble SA100 for automated diagnosis of obstructive sleep apnoea in the home environment setting: an evaluation study.

OBJECTIVES: Obstructive sleep apnoea (OSA) is a heavily underdiagnosed condition, which can lead to significant multimorbidity. Underdiagnosis is often secondary to limitations in existing diagnostic methods. We conducted a diagnostic accuracy and usability study, to evaluate the efficacy of a novel, low-cost, small, wearable medical device, AcuPebble_SA100, for automated diagnosis of OSA in the home environment. SETTINGS: Patients were recruited to a standard OSA diagnostic pathway in an UK hospital. They were trained on the use of type-III-cardiorespiratory polygraphy, which they took to use at home. They were also given AcuPebble_SA100; but they were not trained on how to use it. PARTICIPANTS: 182 consecutive patients had been referred for OSA diagnosis in which 150 successfully completed the study. PRIMARY OUTCOME MEASURES: Efficacy of AcuPebble_SA100 for automated diagnosis of moderate-severe-OSA against cardiorespiratory polygraphy (sensitivity/specificity/likelihood ratios/predictive values) and validation of usability by patients themselves in their home environment.
RESULTS: After returning the systems, two expert clinicians, blinded to AcuPebble_SA100's output, manually scored the cardiorespiratory polygraphy signals to reach a diagnosis. AcuPebble_SA100 generated automated diagnosis corresponding to four, typically followed, diagnostic criteria: Apnoea Hypopnoea Index (AHI) using 3% as criteria for oxygen desaturation; Oxygen Desaturation Index (ODI) for 3% and 4% desaturation criteria and AHI using 4% as desaturation criteria. In all cases, AcuPebble_SA100 matched the experts' diagnosis with positive and negative likelihood ratios over 10 and below 0.1, respectively. Comparing against the current American Academy of Sleep Medicine's AHI-based criteria demonstrated 95.33% accuracy (95% CI (90·62% to 98·10%)), 96.84% specificity (95% CI (91·05% to 99·34%)), 92.73% sensitivity (95% CI (82·41% to 97·98%)), 94.4% positive-predictive value (95% CI (84·78% to 98·11%)) and 95.83% negative-predictive value (95% CI (89·94% to 98·34%)). All patients used AcuPebble_SA100 correctly. Over 97% reported a strong preference for AcuPebble_SA100 over cardiorespiratory polygraphy.
CONCLUSIONS: These results validate the efficacy of AcuPebble_SA100 as an automated diagnosis alternative to cardiorespiratory polygraphy; also demonstrating that AcuPebble_SA100 can be used by patients without requiring human training/assistance. This opens the doors for more efficient patient pathways for OSA diagnosis. TRIAL REGISTRATION NUMBER: NCT03544086; ClinicalTrials.gov. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study demonstrates the diagnostic accuracy and unattended/untrained usability in real-world conditions of a new commercial wearable medical device for fully automated diagnosis of obstructive sleep apnoea.

In addition to the diagnosis, it assesses the accuracy in classification of apnoea events and their type.

Four different diagnostic indices were used to test the accuracy of the medical device.

The study used multichannel cardiorespiratory polygraphy as gold-standard reference, with no neurological channels.

The study is single centre.

Introduction

Obstructive sleep apnoea (OSA) is a common condition with prevalence exceeding 50% in some countries.1 2 Recent data suggest that up to 1 billion adults globally aged 30–60 have OSA1; and if left untreated, OSA significantly increases the risk of cardiovascular and cerebrovascular complications such as myocardial ischaemia, stroke and arrhythmias.3 4 The repetitive awakenings from sleep that characterise OSA and the resultant sleep fragmentation and daytime somnolence can adversely impact on quality of life, cognitive function and mental health.5 6 OSA also has a significant global economic impact. In the USA, approximately 80% of patients with moderate or severe OSA are not diagnosed, costing the economy between US$60bn and US$160bn a year.7 A Health Economics Report commissioned by the UK British Lung Foundation (BLF) estimated that 85% of patients with OSA are undiagnosed and that diagnosing and treating these patients could generate cost savings of £55 million for the National Health Service (NHS) and increase survival rates for those patients by 25%.8 Furthermore, around 25% of car accidents in Europe are attributed to OSA,9 and in the UK, up to 40 000 road accidents could be avoided per year if all patients were diagnosed and treated.8 Additionally, recent data suggest that patients with mild OSA may benefit from treatment; thus, it is likely that an increasing number of individuals will be referred for testing.10

Diagnosis of OSA typically occurs in secondary care sleep clinics where arrangements for patients to be monitored either with full polysomnography or limited cardiorespiratory polygraphy (CR-PG) are made. CR-PG involves wearing several bands, sensors and probes overnight, ideally attended as an inpatient or, due to services being overwhelmed by demand and costs, unattended at the patient’s own home. The domiciliary approach is more commonly undertaken in UK sleep clinics and involves the patients attending the hospital to be trained on how to attach and use a CR-PG system, wearing the device overnight at home and then returning it the following day.11

This method of diagnosis has significant limitations since the process of training patients can be time-consuming and some patients struggle to set up the device properly, thereby generating invalid signals and tests. Additionally, the devices can be cumbersome and may move or deattach during the night or be a source of discomfort for patients, affecting their sleep quality and in turn the validity of the result. Failure rates from 5% to 39% have been reported.12 The high cost of these devices often means that they can only be offered to a relatively small number of patients at any one time.13 Furthermore, the complexity of interpretation often leads to diagnostic delays as the signals require manual ratification and analyses by specialists, since automated diagnostic software is not reliable. This process can take up to 1–2 hours per patient.14 Thus, despite the convenience of an ambulatory approach, services remain overwhelmed by demand, cost and clinician time constraints. Finally, more recently, due to the COVID-19 pandemic, these traditional methods can no longer easily be used due to infection prevention and control, and novel approaches for ‘virtual’ diagnostics are desperately required to overcome these issues.

This paper presents the outcomes of a diagnostic accuracy and usability study carried out to evaluate the efficacy of a novel, small, wearable medical device, Acupebble and accompanying software for the automated diagnosis of OSA (AcuPebble SA100). AcuPebble SA100 was conceived to be used by patients without the need for intensive face-to-face training by a healthcare professional (HCP), aiming to overcome some of the limitations of current systems, consequently facilitating the diagnosis of OSA and helping to unblock some of the existing bottlenecks in current diagnostic pathways.

Methods

A prospective study of diagnostic accuracy was undertaken. The study was registered on ClinicalTrials.gov.

Study design

The study was designed following the recommendations given by the British Medical Journal on assessments and critical appraisals of diagnostic tests, the USA Agency for Healthcare Research and Quality for studies specific to diagnosis and treatment of sleep apnoea in adults15 and the ISO Standard on Clinical Investigation of Medical Devices for Human Subjects EN ISO 14155:2011.

Consecutive patients aged between 18 and 70 who were referred for evaluation of possible OSA to the Sleep and Ventilation clinic at the Royal Free London Hospital NHS Foundation Trust were recruited for the study.

Patient and public involvement

Members of the public were involved in prior formative usability evaluations (IEC 62366) that informed the design of the study.

Eligibility criteria

All adult patients were included except for those above the age of 70; subjects who were not fluent in English or had special communication needs; those with a known allergy to adhesive dressings; subjects with physical or mental impairments, which would make them unable to use the new technology on their own; subjects with electronic body implants and subjects with extremely lose skin in the neck area, which would make the device swing if the neck moved.

Test methods

Patients who consented to participate in the study were issued with both a CR-PG device as per usual clinical care and an AcuPebble SA100 device with a smart phone. Both the CR-PG and AcuPebble SA100 were to be used simultaneously. The patient was first trained on the use of the CR-PG device, including a demonstration of how to wear it and basic troubleshooting. This training is part of the typical pathway and takes approximately 30 min per patient. No specific face-to-face training was offered to the patient on the use of the AcuPebble SA100, since an objective of the study was to demonstrate usability in the absence of prior training. Once at home, the smart phone app guided the patient through the process of setting up the sensor, attaching it, starting and finishing the study. On completion of the study, the app would offer the patient the possibility of filling a simple voluntary questionnaire designed to evaluate the user experience. Patients were asked to return the devices the following day. This would have been required regardless of the study, since it forms part of the conventional diagnostic pathway. If a patient did not undertake the study once they were at home, the patient was not rerecruited.

Reference standard: CR-PG

The domiciliary CR-PG system used in this study was the type III monitor; Embletta MPR Sleep System (Natus Medical, California) and accompanying Embla Remlogic software (Natus Medical, California). The following signal channels from the CR-PG device were used for analysis: thoracic and abdominal piezoelectric respiratory movement sensors peripheral pulse oximetry, nasal thermistor air flow sensor, snore and body position. This device was chosen for two reasons: the system is routinely used in the Sleep and Ventilation clinic at Royal Free London NHS Foundation Trust for diagnosis of sleep disordered breathing and meets the American Academy of Sleep Medicine’s (AASM) technical adequacy requirements to be considered a gold standard for ambulatory diagnosis of the disease.16 17 Given that the intended use of AcuPebble SA100 is for ambulatory home testing, and one objective of this study was to test usability in real conditions, the current gold-standard type III home CR-PG monitors were felt to be the appropriate comparator. In clinic/hospital, Polysomnography (PSG) would not have been representative of the real use scenario, and neither common clinical practice. Furthermore, the principle of operation against in clinic-PSG had already been demonstrated in a prior pilot evaluation, where a less advanced, non-commercial, research variant of the sensor and algorithms demonstrated an 88.6% sensitivity and 99.6% specificity for apnoea detection.14

Index test: AcuPebble SA100

The device under evaluation, AcuPebble SA100, is a wearable electronic technology, measuring 2.9 cm diameter, 1.4 cm height and weighing 7 g, figure 1. The device attaches to the neck, anywhere between the laryngeal prominence of the thyroid cartilage and the supra-sternal notch, with a disposable medical grade adhesive. The exact location is not important. The device functionality is based on the principle of acoustic sensing of a number of physiological sounds, including those generated by the respiratory and cardiovascular systems.

Figure 1

From left to right: AcuPebble SA100 sensor; AcuPebble SA100 sensor with accompanying app; Model (not patient) wearing the sensor (photo obtained from https://acurable.com/products/acupebble-SA100/patients).

AcuPebble SA100 works with an accompanying self-explanatory application (App) running on a smartphone/tablet. The App has two distinctive sections: section 1 provides access to a HCP portal and section 2 is for patients. The HCP portal allows the clinician to setup the sleep study for individual patients and also visualises/reviews its outcomes. Once the study is setup, the patient portal of the App for that particular study becomes enabled allowing the patients to complete the overnight study, during which acoustic signals are collected from the wearable device. At the end of the study, the AcuPebble SA100 software algorithms derive the breathing segments, cardiac information and their time–frequency characteristics from the sensed acoustic signal. These are then used to automatically detect disordered breathing events and generate the diagnostic output based on the AASM recommendations.

Sample calculation

The sample size for the study was calculated to be 150 valid patients. This sample was based on a conservative estimation, to accommodate prevalence values for moderate/severe sleep apnoea from 30% to 50%, and assuming the sensitivity and specificity of the system would be 91%–99%. With these ranges, the positive likelihood ratio (LR+) would be over 10 and the negative likelihood ratio (LR−) would be under 0.1. If the system achieved these numbers, its efficacy as a method of diagnosis for OSA would be proven.15

Data analysis

The primary objective of this study was to assess the efficacy of AcuPebble SA100 for automated diagnosis of moderate-severe OSA against CR-PG as well as validate the usability of the system by patients themselves in their home environment. Secondary objectives included an assessment of AcuPebble SA100’s performance in differentiating between apnoeic and hypopnoea obstructive events and for differentiating central and OSA.

AcuPebble SA100 can automatically generate four different diagnostic outputs. The rational for having four alternative diagnostic outputs was to afford clinicians the choice of the appropriate diagnostic output for their service, since globally, differing sleep clinics use different diagnostic criteria and may use different versions of the AASM diagnostic recommendations.17 The four outputs are:

Diagnosis based on AHI defined by the current recommended AASM criteria17 (ie, with ≥3% as the threshold for oxygen desaturation).

Diagnosis based on AHI defined by the AASM criteria,17 but with the exception of having ≥4% as the threshold for desaturation.

Diagnosis based on ODI considering ≥3% desaturation as the threshold for events.

Diagnosis based on ODI considering ≥4% desaturation as the threshold for events.

In order to test the efficacy of AcuPebble SA100 for the four different diagnostic outputs, the CR-PG signals were manually and independently scored by two experienced clinician scorers, resulting in these four diagnostic indices and corresponding diagnosis. The clinicians were blinded to the outputs from the AcuPebble SA100. At the end of the study, the diagnostic results derived from the experts manual scoring were compared with AcuPebble SA100 automated diagnosis.

The following criteria were used to evaluate the diagnostic performance:

A diagnostic output was considered a true positive if both AcuPebble SA100 and the Gold Standard agreed on the diagnosis of a patient as having either moderate (ie, 15–30 AHI or ODI) or severe (ie, >30 AHI or ODI) sleep apnoea. Moderate and severe OSAs were considered to be true positives since these are the diagnostic categories for which gold-standard treatment (Continuous Positive Airway Pressure (CPAP)) is recommended.7

A diagnostic output was considered a true negative if both AcuPebble SA100 and the gold standard agreed on the patient not having moderate or severe OSA.

A diagnostic output of AcuPebble SA100 was considered a false negative if the gold standard diagnosed the patient with moderate or severe OSA, and AcuPebble diagnosis was mild or normal.

A diagnostic output of AcuPebble SA100 was considered a false positive if the gold standard diagnosed the patient with normal or mild sleep apnoea, and AcuPebble SA100 diagnosis was moderate or severe.

The performance metrics used for evaluation followed the recommendations for validation of efficacy in diagnostic methods: sensitivity, specificity, positive predictive value, negative predictive value, LR+, LR−, accuracy and Cohen’s Kappa (K).15 18

Usability analysis

Patient feedback regarding the usability, comfort and confidence of using the AcuPebble SA100 device was collected using a voluntary questionnaire included within the App.

Usability was quantified in terms of:

Percentage of patients who used the system correctly.

Percentage of patients who did the test and returned diagnostically valid AcuPebble signals.

Percentage of patients that placed the sensor correctly.

Usability answers in the App questionnaire.

Results

Participants

One hundred and eighty-two consecutive patients were recruited for evaluation over an 8-month period between November 2018 and July 2019. One hundred and twenty-nine (71%) were men. Of these studies, 150 could be used for evaluation purposes. A subject participation and data sufficiency diagram are shown in figure 2. It is noteworthy that all of subjects were able to use the AcuPebble SA100 and equally all the signals from AcuPebble SA100 were valid and analysable. Demographic data and comorbidities for the participants are presented in table 1.

Figure 2

Flow of participants and data sufficiency.

Table 1

Characteristics of the 150 patients used for evaluation

Age (years)	Median	45
	Mean	44
	SD	11
	Range	(21,65)
BMI self-reported. Data available from 128 patients (84.2%)	Median	29.9
	Mean	31.2
	SD	7.6
	Range	(17.6, 56.6)
Weight (kg)Self-reported. Data available from 129 patients (84.9%)	Median	92
	Mean	95.3
	SD	25.7
	Range	(45.7190)
Height (cm)Self-reported. Data available from 132 patients (86.8%)	Median	175.2
	Mean	174.4
	SD	9.8
	Range	(150,197)
Number of patients per BMI classification	Underweight (<18.5)	1 (0.7%)
	Healthy weight (18.5–24.9)	26 (17.3%)
	Overweight (25–29.9)	36 (24%)
	Obese (30–39.9)	51 (34%)
	Severely obese (>40)	12 (8%)
Sex	Male	107 (71.3%)
	Female	43 (28.7%)
Ethnicity (number of patients)	White British	47 (31%)
	White other	19 (12.67%)
	Asian or Asian British (excluding the ones below)	31 (20.67%)
	Black or Black British (excluding the ones below)	3 (2%)
	Indian	2 (1.33%)
	Pakistani	2 (1.33%)
	White or Black African	2 (1.33%)
	Chinese	1 (0.67%)
	White or Black Caribbean	5 (3.33%)
	Other	38 (25.34%)
Most common comorbidities	High blood pressure	38 (25.3%)
	Diabetes	17 (11.3%)

BMI, body mass index.

Diagnostic accuracy evaluation

The diagnostic performance of AcuPebble SA100 for the four different criteria, when comparing with the equivalent diagnosis following the gold-standard method of diagnosis at home (ie, CR-PG followed by expert manual scoring of signals), is shown in tables 2–5. The data demonstrate that AcuPebble SA100 is accurate in diagnosing OSA with specificity of 96.8%, sensitivity of 92.7%, LR+ of 29·36, LR− of 0·08 and Cohen’s Kappa of 0.90, using the current AASM diagnostic criteria. Table 6 shows the confusion matrix showing the accuracy for different disease levels. Outputs for which the indexes showed to be exactly at the border of the classification limits (eg, an index of exactly 30) would be registered in the cell corresponding to the classification of the reference test, if this was in agreement (ie, the diagonals in the table). Otherwise, they would be in the corresponding non-diagonal cell (which quantify disagreement).

Table 6

Confusion matrix for different levels of sleep apnoea diagnosis

		Diagnosis based on AHI defined by the current AASM recommended criteria17 but with the exception of having >4% as the minimum threshold for desaturation (K=0.89)				Diagnosis based on current AASM recommended AHI-based criteria17 (K=0.82)
		Gold-standard diagnosis—number of patients=150
		Normal	Mild	Moderate	Severe	Normal	Mild	Moderate	Severe
Acupebble diagnosisn=150	Normal	68	2	0	0	53	7	1	0
	Mild	1	28	2	0	2	31	3	0
	Moderate	0	3	20	2	0	3	19	2
	Severe	0	0	1	23	0	0	1	28

AcuPebble SA100 versus CR-PG (with manual expert marking) as reference gold-standard test.

Cells highlighted with green indicate complete agreement between AcuPebble and Gold Standard diagnosis.

AASM, American Academy of Sleep Medicine; CR-PG, cardiorespiratory polygraphy.

AcuPebble SA100 also gives an output of the subdivision of apnoeic versus hypopneic events. In order to evaluate the performance, the classification of events for 10 randomly chosen patients with moderate or severe sleep apnoea was individually compared with the blind classification of the scorer. The accuracy of the classification was 90·20% with 95% CI (89·11% to 91·22%).

For apnoea events, the accuracy of classification between central and obstructive apnoea was 81·7% with 95% CI 77·68% to 85·25%. This result was validated using the events from the 10 patients for whom the scorers (using the CR-PG signals) showed the maximum number of central events. The total number of events individually compared was 426, out of which 117 were central and 309 were obstructive.

Usability

One hundred and twenty-three patients out of the 150 evaluation cohort completed the usability questionnaire. The questionnaire as well as graphical representations of the answers are shown in figure 3. Patients found the AcuPebble sensor more comfortable than CR-PG, with 111 patients (90%), reporting it to be much more comfortable; 10 patients (8%) found it equally comfortable and 2 patients found it less comfortable. However, out of the two patients finding it less comfortable, one of them commented that they had been awake since 03:30, when further inspection of this particular data set demonstrated that both AcuPebble SA100 and signals from the CR-PG system highlighted apnoeic events through the night, suggesting this individual was in fact asleep. One hundred and nineteen patients (97%) agreed that AcuPebble SA100 was easier to use than CR-PG. The remaining four, neither agreed nor disagreed. Ninety-nine per cent of subjects felt confident with using the App. The individual who stated lack of confidence commented that the specific lack of confidence was of the use of the English language, implying a language barrier, rather than a usability one. One hundred and twenty patients (97·6%) claimed to have followed all the steps without assistance. Although three claimed to have needed assistance, they obtained this by themselves (ie, no one within the research team provided this assistance) and managed to complete the test.

Figure 3

Usability results. The questions posed (in the form of degrees of agreement to statements) are represented in the x axis of the figures. These were: Q1. I managed to follow all the steps on the mobile app without assistance. Q2. I understood all instructions in the phone/tablet. Q3. I felt confident using the app on the phone/tablet. Q4. It was easy to attach the sensor to my neck. Q5. I had no problem replacing the adhesive (sticky paper) on the sensor. Q6. The sensor on the neck was more comfortable than the other sensors on my body. Q7. The sensor on the neck was easier to attach than the combination of all the other sensors on my body.

In addition, the patients were given the option of writing down comments about the experience. Overall, the comments showed how patients were impressed with the device, the ease of use and very much preferred it to the current CR-PG system.

Comparative analysis of healthcare human resources in setting up CR-PG and AcuPebble SA1000

A comparative analysis of healthcare human resources associated to setting up the device for diagnosis and obtaining the diagnostic output, for AcuPebble SA100 and CR-PG, in the UK,19 is summarised in table 7. This demonstrates that using AcuPebble SA100 can result in significant time savings for HCP. This would allow for a shift of resources to improve the efficiency of the diagnostic and treatment pathways.

Table 7

Comparative analysis table summarising healthcare human resources associated to setting up the device for diagnosis and obtaining the diagnostic output, for AcuPebble SA100 with respect to CR-PG, in the UK*

	AcuPebble SA100	CR-PG
	Time	Time
Cleaning	0.5 min	2 min
Device preparation	0.5 min	10 min
Time of healthcare professional training patient on using the device	0	30 min
Analysis of signals by experts to issue a diagnosis	0	60–120 m
Cost19	~£1	£250–£500*

*This range has been calculated taking into account the variation in the time spent by different healthcare professionals analysing signals (60–120 min), as well as their cost in the UK NHS. The numbers have been obtained using the tool provided by the UK National Institute for Health Research, version 2019.19 It has been assumed that the training of the patient is done by a nurse or allied health professional and the analysis by a clinician.

CR-PG, cardiorespiratory polygraphy.

Adverse events

There were no adverse events in the study.

Discussion

This study has demonstrated that this novel device, AcuPebble SA100, can accurately and automatically diagnose OSA among patients attending a centre for home sleep testing. Furthermore, patients were able to use the device with no prior training, and patient feedback demonstrated ease of use and a very strong preference for AcuPebble SA100 compared with CR-PG.

AcuPebble SA100 provides four alternative automated diagnostic outputs, to accommodate for different diagnostic criteria followed by different sleep centres globally. In addition to the automated diagnosis, AcuPebble SA100 provides analysis of the respiratory events, heart rate and breathing rate throughout the night as well as visualisation of the processed signals and the option to add annotations. The device proved to be highly accurate for all four diagnostic criteria, in all recommended statistical metrics for assessment of diagnostic methods in sleep apnoea.15 18 In all cases, the LR+ and LR− were significantly better than those reported for other validated and regulated diagnostic methods for sleep apnoea diagnosis.15 20–24 The results were best for AHI-based diagnosis. This is not surprising, since relative reductions in oxygen saturation levels are obtained from acoustic signal processing features derived from surrogate physiological signals (including cardiac and respiratory sounds), as opposed to directly from oxygen saturation levels. Still, the accuracy for ODI-based automatic diagnosis was over 90%, the LR− lower or equal to 0·1, and the LR+ higher than 10. Furthermore, the instances where AcuPebble and CR-PG differed could be largely explained by a variety of factors. First, human error when manually scoring the cardiorespiratory traces, relating to event duration and overestimation of short respiratory, pauses not meeting diagnostic criteria. Second, the airflow quantification in both systems is based on two different anatomical locations and physical processes. Thus, it is not always the case that the same volume of air flows through the nose as it does in the lower respiratory tract. Additionally, improper placement of the CR-PG sensor in the nostril (resulting on low-quality signal) could cause a disagreement between the two systems on whether the reduction in flow was above the threshold to be classified as an event. This was compounded by the fact that there were borderline events a human marker cannot visually accurately classify as a reduction in flow below or above the threshold (whereas mathematical algorithms can). Finally, the airflow channels in the CR-PG system have electronic filters incorporated, which slow down and delay the output signal, with respect to their physiological start time. This is not usually an issue, since the exact duration of an event to subsecond accuracy is not important clinically, but it is the reason behind some of the differences between the output of AcuPebble SA100 and the reference test. AcuPebble SA100 relies on automated algorithms which, unlike the human markers, always follow the same exact rules for the determination of start and stop of events. Thus, for example, a 9·5 s event will always be calculated to have the same duration by AcuPebble SA100, whereas a human expert marker will, sometimes, mark it as 9 s ad others as 10.5 s. In addition, the sensing mechanism and electronics blocks in AcuPebble SA100 have much faster temporal responses than those in conventional polysomnography channels, which at times (depending on the signal strength) manifest with events that are shorter (but represent better the physiological processes) than those that manifest in the airflow and effort bands. Both of these factors would make a difference to the AHI, since only events larger than 10 s count. These factors explain most of the differences in diagnosis made between the two devices.

Although AcuPebble SA100 does not provide direct access to the photoplethysmography signal (ie, the physiological signal from which the oxygen saturation values are obtained) or to any absolute value of oxygen saturation, the system is able to identify drops in oxygen saturation via features of the acoustic signals. These features are representative of different physiological processes. Using this, hypopneas can be identified, since these only rely on identification of desaturations over 3% and 4% thresholds, and not on absolute values of oxygen saturation.

Within the context of the study, five patients forgot to undertake the test and three did not do it as they logged themselves out of the App (note that patients had not been given the password to log back in and had not been warned of the possibility of this happening). However, it is noteworthy that, while these scenarios could happen in real life, outside the context of the study, patients would not necessarily have to return the system the next day and could easily try again the following night. Equally, they have access to several mechanisms (such as a dedicated number) to retrieve the login code. For this study, patients were recruited consecutively and, hence, had different comorbidities. It would be, however, of interest to conduct future studies in patient populations specifically with specific comorbidities such as diabetes, Chronic Obstructive Pulmonary Disease (COPD), high blood pressure and others.

The feedback from patients revealed a clear preference for the AcuPebble SA100 system when compared with the alternative ambulatory CR-PG system. Furthermore, 100% of patients were able to use the device correctly without training, which together with the fact that no manual analysis of the signals is needed (but can be accessed if required), significantly reduces the resources that would be needed for the diagnosis of OSA. This could either lead to pathways savings or alternatively to resources being transferred towards treatment of more patients. It should be noted that although the upper age limit was set to 70 for this study, over 70s are not excluded from the intended use of AcuPebble SA100. Since a secondary endpoint of this study was to evaluate the usability on fully unattended and untrained conditions in consecutive patients, when the study was initially conceived, it was decided that a separate usability study would be done with patients over 70.

In addition to the improved patient experience, using technology with the characteristics of AcuPebble SA100 has the potential to afford other advantages: the ease of device use lends it to being issued in community-based diagnostic clinics or even being sent to patients directly. This is particularly important given the current global pandemic, sleep diagnostics have been severely affected with many trusts experiencing a severe backlog, the ability to reduce face-to-face contact for diagnostics in critically important during this time.25 This had the added advantage of further reducing the costs associated with diagnosis by reducing the number of outpatient appointments a patient is required to attend26 and allowing alternative service models to be developed. This also aligns with key aims of some national healthcare systems recent long-term plan27; and might also reduce waiting times and enable patients to have faster access to treatments by quicker diagnosis. Clearly, improving access to diagnoses is desperately needed given the BLF, McKinsey & Company and Harvard and Sleep for America estimates over a million ‘missing’ OSA cases in the UK, and in excess of 18 million in the USA.7 8 This is especially pertinent given data which support treating clinically significant OSA as cost-effective to healthcare providers.28 29 Finally, CR-PG and full PSG equipment require a large capital investment and if devices break down, this can have significant impact on service delivery. Sleep services can stock AcuPebble SA100 devices in larger quantities, thus ensuring that patient testing is never affected by breakdown of devices. Indeed, our preliminary findings demonstrate that use of the AcuPebble SA100 is comparatively inexpensive in the set up phase and these cost savings will be passed on to services purchasing devices and treatment costs. Overall, AcuPebble SA100 could result in whole systems cost savings and significantly better patients’ experience.

Limitations

The authors acknowledge that the gold-standard test for OSA is full PSG; however, we did not test against this as increasingly many sleep services in Europe are not using full PSG as their primary diagnostic test for OSA,30 and also because, although AcuPebble SA100 can be used in hospital settings, the ultimate intended use is in the home environment. Since this study was done for regulatory purposes, the real use conditions had to be replicated as close as possible and thus, from the regulatory point of view, there was no justification to carry out the study using full in-clinic PSG. Furthermore, although full at-home PSG could have been considered, since the AASM definition of events used for diagnosis did not require the use of neurological channels, those channels would have been an unnecessary burden for the patients while providing no benefit for the patients. Additionally, a less advanced research variant of the device had already been compared with PSG, although, in a pilot study.14 We chose CR-PG as this is a commonly used domiciliary sleep test30 and, therefore, represents an appropriate comparator for Acupeble SA100, which will also be a home testing device. It could be argued that oximetry devices are also relatively inexpensive and accessible. However, it is known that oximetry may underestimate the degree of sleep disordered breathing as many patients with OSA do not always have a desaturation with every respiratory event.31 Furthermore, usability results in the pilot study14 revealed that out of all of the sensors worn on the patient’s body during PSG, the clip on the finger is the one that they found more disturbing during sleep.

Currently, AcuPebble SA100 has not been validated for the diagnosis of other respiratory sleep disorders such as hypoventilation. However, for most centres, OSA will be the most common respiratory sleep disorder and AcuPebble SA100 will be sufficient for these patients. Furthermore, AcuPebble SA100 could be used to either directly diagnose patients for OSA or assist in the triaging process to determine appropriate diagnostic testing, ensuring that the more complex patients have access to more complex testing in a timely manner.

Finally, this was a single-centre study. However, in order to try to reduce any bias due to this, sleep tests were performed unattended in the home environment of patients, and manual scoring was performed independently by two expert clinicians

Conclusions

Overall, AcuPebble SA100 provides significant advantages with respect to existing gold-standard methods for home sleep testing for OSA, with excellent diagnostic accuracy, potential cost savings, reduction in face-to-face contact and clear patient preference compared with CR-PG.

Table 2

Evaluation of performance in diagnosing OSA when comparing automatic diagnosis of AcuPebble following the current AASM AHI based criteria,* with the reference test (CR-PG)

Statistic	Value	95% CI
Disease prevalence (%)	36.67	28.96 to 44.92
Sensitivity (%)	92.73	82.41 to 97.98
Specificity (%)	96.84	91.05 to 99.34%
Positive likelihood ratio	29.36	9.62 to 89.64
Negative likelihood ratio	0.08	0.03 to 0.19
Positive predictive value (%)	94.44	84.78 to 98.11
Negative predictive value (%)	95.83	89.94 to 98.34
Accuracy (%)	95.33	90.62 to 98.10
Cohen’s Kappa	0.90	0.82 to 0.97

Note also that the same results would be obtained under the upcoming 2.6 version, since the changes with respect to 2.5 do not affect this work.

*AASM v2.5.17

AASM, American Academy of Sleep Medicine; AHI, Apnoea Hypopnoea Index; CR-PG, cardiorespiratory polygraphy; OSA, obstructive sleep apnoea.

Table 3

Evaluation of performance when comparing automatic diagnosis of AcuPebble SA100, based on AHI defined by the AASM criteria* but with the exception of having ≥4% as the threshold for desaturation, with the reference test (CR-PG)

Statistic	Value	95% CI
Disease prevalence (%)	32.67	25.24 to 40.79
Sensitivity (%)	95.92	86.02 to 99.50
Specificity (%)	97.03	91.56 to 99.38
Positive likelihood ratio	32.29	10.58 to 98.59
Negative likelihood ratio	0.04	0.01 to 0.16
Positive predictive value (%)	94	83.69 to 97.96
Negative predictive value (%)	98	92.65 to 99.48
Accuracy (%)	96.67	92.39 to 98.91
Cohen’s Kappa	0.92	0.86 to 0.99

Note also that the same results would be obtained under the upcoming 2.6 version, since the changes with respect to 2.5 do not affect this work.

*AASM V.2.5.17

AASM, American Academy of Sleep Medicine; AHI, Apnoea Hypopnoea Index; CR-PG, cardiorespiratory polygraphy.

Table 4

Evaluation of performance when comparing automatic diagnosis of AcuPebble SA100, based on ODI alone considering ≥3% desaturations as the threshold for events, with the reference test (CR-PG)

Statistic	Value	95% CI
Disease prevalence (%)	52.00	43.70% to 60.22
Sensitivity (%)	91.03	82.38% to 96.32
Specificity (%)	93.06	84.53% to 97.71
Positive likelihood ratio	13.11	5.61 to 30.62
Negative likelihood ratio	0.10	0.05 to 0.20
Positive predictive value (%)	93.42	85.87 to 97.07
Negative predictive value (%)	90.54	82.48 to 95.11
Accuracy (%)	92	86.44 to 95.8
Cohen’s Kappa	0.84	0.75 to 0.93

CR-PG, cardiorespiratory polygraphy; ODI, Oxygen Desaturation Index.

Table 5

Evaluation of performance when comparing automatic diagnosis of AcuPebble SA100, based on ODI considering ≥4% desaturations as the threshold for events, with the reference test (CR-PG)

Statistic	Value	95% CI
Disease prevalence (%)	32.67	25.24 to 40.79
Sensitivity (%)	97.96	89.15 to 99.95
Specificity (%)	92.08	84.99 to 96.52
Positive likelihood ratio	12.37	6.35 to 24.08
Negative likelihood ratio	0.02	0.00 to 0.15
Positive predictive value (%)	85.71	75.5 to 92.11
Negative predictive value (%)	98.94	93.03 to 99.85
Accuracy (%)	94	88.92 to 97.22
Cohen’s Kappa	0.87	0.79 to 0.95

CR-PG, cardiorespiratory polygraphy; ODI, Oxygen Desaturation Index.