Is point-of-care testing feasible and safe in care homes in England? An exploratory usability and accuracy evaluation of a point-of-care polymerase chain reaction test for SARS-CoV-2.

INTRODUCTION: Reliable rapid testing for COVID-19 is needed in care homes to reduce the risk of outbreaks and enable timely care. This study aimed to examine the usability and test performance of a point of care polymerase chain reaction (PCR) test for detection of SARS-CoV-2 (POCKITTM Central) in care homes.
METHODS: POCKITTM Central was evaluated in a purposeful sample of four UK care homes. Test agreement with laboratory real-time PCR and usability and used errors were assessed.
RESULTS: No significant usability-related hazards emerged, and the sources of error identified were found to be amendable with minor changes in training or test workflow. POCKITTM Central has acceptable sensitivity and specificity based on RT-PCR as the reference standard, especially for symptomatic cases.Asymptomatic specimens showed 83.3% (95% confidence interval (CI): 35.9-99.6%) positive agreement and 98.7% negative agreement (95% CI: 96.2-99.7%), with overall prevalence and bias-adjusted kappa (PABAK) of 0.965 (95% CI: 0.932- 0.999). Symptomatic specimens showed 100% (95% CI: 2.5-100%) positive agreement and 100% negative agreement (95% CI: 85.8-100%), with overall PABAK of 1.Recommendations are provided to mitigate the frequency of occurrence of the residual use errors observed. Integration pathways were discussed to identify opportunities and limitations of adopting POCKIT™ Central for screening and diagnostic testing purposes.
CONCLUSIONS: Point-of-care PCR testing in care homes can be considered with appropriate preparatory steps and safeguards. Further diagnostic accuracy evaluations and in-service evaluation studies should be conducted, if the test is to be implemented more widely, to build greater certainty on this initial exploratory analysis.

Point of care tests in care homes are feasible and could increase testing capacity for the control of COVID-19.

The test of agreement between POCKIT™ Central and laboratory polymerase chain reaction for care home residents and the staff was good.

Adoption of POCKIT™ Central in care homes can be considered with appropriate preparatory steps and safeguards in place.

Repetitive errors and test malfunctioning can be mitigated with bespoke training for care home staff.

Pathways of adoption should be investigated to appreciate the high variability of the context of use.

Introduction

Care home residents have been amongst the worst affected by the COVID-19 pandemic [1]. In England and Wales, the Office of National Statistics estimates that care homes experienced 21,344 excess deaths due to COVID-19 during the first 10 months of the pandemic [2].

As the pandemic has continued, attention has turned to how SARS-CoV-2 testing might help better protect residents from COVID-19 outbreaks, whilst enabling continuity of service provision and promoting quality of life for residents, by restoring routine practices such as family visiting [3].

Point-of-care testing (POCT) describes tests that take place close to the care setting [4, 5]. We have previously described possible roles for POCT for COVID-19 in screening asymptomatic residents and staff, and diagnostic testing for symptomatic residents and staff [6] For both use cases, POCT could reduce administrative burden and test turnaround times, and reduce the chances of COVID-19 outbreaks through earlier quarantine of positive cases.

The ‘real world’ effectiveness of POCT in care homes is a function of the technical properties of the test (established in laboratory conditions where certainty and parameters are controlled) and its performance in the conditions of uncertainty that exist in homes, such as residents with complex multimorbidity, variation in testing behaviours in staff, variable infection control processes and implementation. POCT should take account of how testing can integrate into existing measures of transmission control, and the complexity involved in preparing for testing, conducting testing, and interpreting and responding to results.

Very little is known as to how well the POCTs can be integrated into care home settings and their readiness for adoption amongst potential users. If these POCTs are implemented into settings that they have not been designed for there is an inherent risk that the operator will execute the test improperly, use it in the way not intended, or even reject the test for lack of perceived benefits.

As part of the CONDOR platform [7], our study aim was to evaluate a point-of-care polymerase chain reaction (PCR) test for detection of SARS-CoV-2 in the UK care home setting. Our primary objective was to rapidly explore the usability and readiness of the technology for use in real-world settings, scope usability requirements, prioritise areas of deployment and identify any possible safety concerns and sources of error to be further investigated and mitigated. A secondary objective was to evaluate the agreement between results from the POC test with the laboratory-based Reverse Transcriptase-PCR (RT-PCR), the current benchmark standard for care homes.

Methods

The technology being evaluated was the POCKIT™ Central Nucleic Acid Analyzer (POCKIT™ Central). This is a benchtop molecular detection system, integrating magnetic bead-based nucleic acid extraction, fluorescence-based insulated isothermal PCR amplification/detection, and liquid handling technologies to offer a walk-away protocol for nucleic acid detection. The machine can run tests on up to eight specimens at once. Results are displayed on the monitor in less than 1.5 hours. POCKIT™ Central machines were delivered directly to care homes, and training provided, by the manufacturer. Only staff members who received formal training were permitted to use the machine.

This study consists of two activities: (i) scoping usability and use errors and (ii) test agreement with laboratory RT-PCR.

No formal power calculation was undertaken. Further evaluation of diagnostic accuracy is being conducted in other parts of the CONDOR platform.

Participant recruitment

The CONDOR study has recruited 28 care homes, caring for more than 1,490 residents, to our testing platform. These were recruited through publicity in a national care home WhatsApp COVID-19 peer support group [8], public-facing social media and the national care home umbrella organisations Care England and the National Care Forum. The care home database is active, and we continue to recruit. It includes nursing and residential homes; corporate chain, independent and third-sector providers; and bed capacity ranging from 20+ to 350+ in each home. From this, we selected a purposive sample of four care homes (Table 1), comprising two care homes with nursing and two care homes without nursing (also known as nursing and residential homes respectively), from two regions of the UK, two independents and two small chains, to ensure we maximised our ability to account for differences in staff training, and organisational configuration, that might impact on the implementation or use of a POCT.

Table 1

Description of care homes and participants taking part in the study

Interviewee’s role	Area	Nursing home	Residential home	Total number of beds	Identification code
Care home manager	Nottinghamshire	X		80+	CH1
Managing director	Nottinghamshire		X	140+	CH2
Project improvement officer					CH3
Chief nursing officer	Oxfordshire	X	X	350+	CH4
Managing director	Nottinghamshire	X		50+	CH5

We chose the number of participants based upon the need to understand workflows around staff and resident routine testing, whilst capturing errors that might occur on repeated use of the machine, by different staff members, on different days, under different circumstances.

Scoping usability and use errors

To scope usability areas and potential sources of error, semi-structured interviews were conducted with key stakeholders (Table 1). These interviews also enabled further insights regarding potential operational and pathway integration of the new test. Participants were directly involved with trialling POCKIT™ Central in care homes and had experience in using the device.

Stakeholders were interviewed individually and remotely by a researcher in human factors (MM) at the end of the trial. In one instance, for reasons of practicality, two interviewees who worked together were interviewed together. Interviews were semi-structured and lasted 30–60 minutes; they were audio and video recorded, with the permission of the participants. Interview schedules focused on the manufacturer’s instructions for use, how POCKIT™ Central might be integrated into the diagnostic pathway, the testing strategy and the clinical decision-making based on positive and negative results. Participants were then prompted to explore potential usability issues in the use of POCKIT™ Central (including clarity of test results, potential hazards and the disposal procedure).

Test agreement with laboratory PCR

We chose to focus primarily on routine staff testing. This is currently conducted weekly, and results from PCR laboratory tests do not always arrive within 7 days. A growing body of evidence suggests that asymptomatic transmission from staff is likely to be one of the most important factors in care home outbreaks [9]. We estimated that conducting 65 tests in each of the four care homes would allow enough repetition of the testing procedure to capture variability in practice between people and over time. The breakdown of tests between the different groups was 35 routine staff tests, 15 routine resident tests, and 15 tests for resident or staff who had become symptomatic.

Nasopharyngeal swabs were taken by care home staff, adhering to standard testing procedures recommended by the government [10], using government-issued testing kits. Use of the POCKIT™ Central device followed a Standard Operating Procedure (see Appendix I) written by laboratory staff after risk assessing the extent of viral inactivation in the sample buffer. Once the testing swab had been inserted into the viral transport medium, 200 μl was transferred into the POCKIT™ Central testing well by the care home staff member using a pre-set Gilson pipette and sterile filter tips. The remainder of the specimen was sent for formal laboratory testing as usual. Care home staff documented testing results from the POCKIT™ Central device using a results log, adding formal laboratory results when these returned. Only anonymised data were seen by the research team. Samples were not blinded at any stage.

All test results, including equivocal results and failures, were reported as per US Food and Drug Administration guidance [11]. We calculated positive and negative agreement of POCKIT™ Central with the formal laboratory RT-PCR results, the (Cohen) kappa, the Brennan and Prediger statistic (equal to the prevalence and biassed adjusted kappa (PABAK)) as well as prevalence based on laboratory RT-PCR results, with their associated 95% confidence intervals (CI). The primary analysis was based on valid results for all tests stratified by symptomatic/asymptomatic participants. These calculations were carried out in Stata/SE 16.1 using: diagt and kappaetc. We also carried out a sensitivity analysis including all POCKIT™ Central equivocal results and test failures in our calculations described above.

Results

The human factors experts reviewed all potential critical issues in the main operation tasks of POCKIT™ Central that could lead to errors and hazards affecting the test operators and the care home residents. Instructions for use and a video demonstration were analysed to highlight the expected source of errors. Four operation tasks and potential risks of use and errors were identified and are outlined in Figure 1. Each task and the probability of occurrence of these potential errors were discussed with interview participants.

Figure 1

Flowchart of the operative procedure and potential risks errors of use associated, as highlighted by human factors experts.

The overall experience with testing equipment was positive. Using the device did not cause excessive cognitive workload as users easily remembered the required steps. Handling components and inserting them into the correct location was not problematic and did not lead to errors. The test activation—which included navigating the digital interface—was considered ‘easy to manage’ (CH2–CH3) and staff members appreciated the presence of prompts and a home button for better control of the process (CH1). No results misinterpretation was observed. Results were correctly recorded. Each test was linked to a unique ID and the data automatically saved for retrieval.

The list of observed errors and unexpected situations has been divided based on the four operative procedure’s stages. Relevant quotes from participants and mitigation strategies to reduce the occurrence of errors are reported (Table 2). Quotes are selected for their relevance to inform key usability aspects and to corroborate findings.

Table 2

List of observed errors of use and mitigation strategies

	Potential use errors	Frequency of occurrence	Description and direct quotes from participants	Mitigation strategies
Task 1—Cartridge setup	No standard materials used	0 cases
	Damaged equipment	1 case	‘the various elements of the cartridge had come loose, and the lid had come off’ (CH2-CH3).	Leaflet for a rapid check of package content with visual
	Misplaced cartridges	0 cases	‘It can only fit the right [location] at the right time. So, if you have picked up the wrong cartridge at that point when it prompted you to put it in, it wouldn’t fit’ (CH1)	One-way interlocking components
	User forgets the correct sequence of steps	0 cases
	Reagents not stored properly	1 case	‘Reagents are expected to be stored in a fridge and this was not explained clearly during the training’ (CH2-CH3).	Support for operators: implementation of training and Instruction for use (e.g. dedicated video demonstration regarding frequent errors and malfunctions)
Task 2—insert the cartridge and initiate test	Foil lid not removed as indicated	2 cases	The aluminium foil did not remove one of the glue coverings: a small amount of glue and foil was left on the cassette. The prong got caught and it tipped it over slightly and took the casing off the prong (CH4).Unclear aluminium foil removal procedure – ‘One batch was processed without the foil being pulled back, but it seems to produce unchanged results. So, I’m not quite sure…And results were reported without the machine going into failure mode’ (CH4).	Design changes I: dye-coloured glue coverings to make residual visible;Design changes II: peeling-off symbol depicted on the aluminium foil surface;Procedure changes: Internal quality control from operators (e.g. cartridge surface cleaning with dedicated wipe).
	Premix container not locked into place	0 cases
	Fiddly parts not easily handled	0 cases
	Reduced dexterity with gloves	0 cases
	Misplaced transfer cartridges and extraction cartridges	0 cases
	Labels not visible	0 cases
	User forgets sequence of steps	1 case	Sample run twice: no warning message – ‘With the second batch [of tests], I ran into a problem. I put a sample from the same tube into the process, so where only seven samples, one of them was duplicated. As soon as I moved on to the next sample, I realised it was a mistake’ (CH4)
	The user inserts incorrect extraction lot number and reagent lot number	0 cases
	User not able to troubleshoot	1 case	Malfunctioning door closing - ‘We had a problem with the door closure of the equipment, which the engineer came and sorted it out’. (CH4)	Rapid troubleshooting guide on-screen display.Support for operators: implementation of training and Instruction for use (e.g. dedicated video demonstration regarding frequent errors and malfunctions)
	Interface not prompting correctly	1 case	Faulty message: the ‘remove the cartridge’ message appeared on screen although cartridges were correctly removed from the machine. Staff members reported splash marks and one stated ‘small splash marks on the cartridge holder which I gently wiped away with an antibacterial wipe. This worked and cleared the message and the machine continued to close the door and move on’ (CH2–CH3). The manufacturer was called, and he advised not to use alcohol-based wipes for this in future but that the response had otherwise been correct. The company did not foresee any risk of spillage.	Support for operators: implementation of training and Instruction for use (e.g. dedicated video demonstration regarding frequent errors and malfunctions)
	Interface (touch screen) not responsive	0 cases
Task 3—Test results	Reactions interrupted before it’s finished	0 cases
	Misinterpretation of test results	1 case	The same sample runs twice on POCKIT™ Central: a positive result first and a negative result subsequently were shown (CH2).	Decisional algorithm to be explored/implemented
	Results not saved appropriately	0 cases
	results failed to be exported on a USB drive	0 cases
Task 4—dispose of the material		0 cases

Integration pathway

Further comments covered the integration aspect of POCT in a care home setting. To run tests correctly, to safeguard operators and residents, and to protect the testing equipment from damage, a dedicated room must be set up with operators wearing fresh PPE. As CH4 explained: ‘I do think we need a dedicated room, which remains a dedicated room for long. It is because it’s a heavy machine. And you would not be transporting it from place to place (risk of damages) […] Dedicated room (that serves all the different houses) made into the testing room, separate from homes, outside the community (no access to residents)’.

Two ways in which the test results might inform practice were discussed: (i) the diagnostic test for symptomatic residents and staff and (ii) the screening test for asymptomatic residents and staff.

As a one-off test, POCKIT™ Central was considered by staff to be appropriate for testing symptomatic residents and for obtaining a rapid result to enable timely care decisions: ‘I can see a huge advantage because for those people that are symptomatic suddenly; I think it could solve a lot of situations for which you see an immediate answer, or a more immediate answer would be useful to complement it on it’ (CH1).

A rapid test would better inform isolation decisions, avoiding unnecessary segregation, with benefits to residents’ wellbeing and the wider care home community: ‘when we’ve had people who were symptomatic, rather than having to close the community we would now put in all the precautions without stopping relatives coming in […] we only have to have two suspected or two confirmed cases’ (CH4). This technology would contribute to maintaining a testing momentum in care home facilities, and it would provide a consistent diagnostic approach. Having care home managers in control would relieve the burden of managing unpredictable factors, including frequent changes of government strategy around testing and isolation: ‘And this would allow us to test people and keep the workforce in a constant continuity for the people in our care. It offers reassurance to the staff’ (CH4).

However, the main concern of staff was the test turnaround of 85 minutes, as they considered it would require extra staff capacity to test the whole home in approximately 2 days. Given the test turnaround time, we identified considerable staffing issues with asymptomatic cases, as these will require significant amounts of staff time: ‘If you’re wanting to replace testing here with something like that, the capacity of it in the time it would take to get through the number of people we need to get to. Which would be a huge problem here’ (CH1).

In big care homes (200+), it might take more than 2 days to screen all the residents if POCTs were deployed in this context. The staff in all care homes outnumbered the residents, and are currently subject to weekly testing, implying even greater workload. Consideration should be given to the resource implications for care homes. There was some disagreement amongst respondents about whether this would essentially mirror the staffing requirements for existing routine screening tests, or whether it would demand extra staff time.

Agreement with laboratory RT-PCR

In total, 278 tests were run. Tests were conducted on 252 asymptomatic participants (176 staff and 76 residents) and 26 symptomatic participants (13 staff and 13 residents) (Table 3 and Appendix II Table A1 stratified by staff/residents). Formal laboratory results were indeterminate for five specimens. POCKIT™ Central results were not available for 17 tests: 1 was a sporadic error, and the remaining 16 were compromised by sealing glue left on top of the POCKIT™ Central cartridge. This was an avoidable error, which is described in detail above. A standard for reporting of diagnostic accuracy studies (STARD) Flow diagram is shown in Figure 2.

Table 3

Full results from POCKIT and Laboratory RT-PCR

Asymptomatic cases
	Laboratory (PCR)
POCKIT	Negative	Positive	Uncertain/error	Total
Negative	222	1	4	227
Positive	3	5	0	8
Uncertain/error	17	0	0	17
Total	242	6	4	252
Symptomatic cases
	Laboratory (PCR)
POCKIT	Negative	Positive	Uncertain/error	Total
Negative	24	0	1	25
Positive	0	1	0	1
Uncertain/error	0	0	0	0
Total	24	1	1	26

Figure 2

STARD flow diagram.

Thus, agreement analysis was conducted on 256 specimens: 177 from staff, 164 asymptomatic and 13 symptomatic; 79 from residents, 67 asymptomatic and 12 symptomatic. Estimated prevalence data, positive and negative agreement, and kappa and PABAK results for asymptomatic and symptomatic participants are shown in Table 4. Equivalent statistics based on including all POCKIT™ Central equivocal and failures are reported in Appendix II Table A2.

Table 4

Agreement based on valid measuresa

Asymptomatic cases
	LAB −ve	LAB +ve	Total
POCKIT negative	222	1	223
POCKIT positive	3	5	8
Total	225	6	231
Test attributes		(95% CIs)
Prevalence (LAB measure)	2.6%	(0.96––5.57%)
Positive agreement (LAB positive as denom)	83.3%	(35.9–99.6%)
Negative agreement (LAB negative as denom)	98.7%	(96.2–99.7%)
Kappa	0.706	(0.429–0.982)
PABAK	0.965	(0.932–0.999)
Symptomatic cases
	LAB −ve	LAB +ve	Total
POCKIT negative	24	0	24
POCKIT positive	0	1	1
Total	24	1	25
Test attributes		(95% CIs)
Prevalence (LAB measure)	4%	(1–20.4%)
Positive agreement (LAB positive as denom)	100%	(2.5–100%)
Negative agreement (LAB negative as denom)	100%	(85.8–100%)
Kappa	1	(1–1)
PABAK	1	(1–1)

aAll CIs are based on standard Stata commands: For Sensitivity, Specificity, PPV and NPV (except with the prevalence option), exact binomial CIs are given, (command ci). ROC area (Receiver Operating Characteristic) uses roctab. LR+ and LR- (based on risk ratio) and odds ratio all use command cs.

Discussion

This paper reports the first exploratory in-context evaluation of point-of-care (POC) PCR for SARS-CoV-2 in the care home setting. We found no significant hazards and the sources of error identified were amenable to relatively minor changes in training or test workflow. Therefore, adopting POCKIT™ Central in a care home setting can be considered safe and feasible with appropriate preparatory steps and safeguards in place.

The positive and negative agreement reported in this study can be compared with the measures of sensitivity and specificity based on RT-PCR as the reference standard. The estimates presented here for POCKIT™ Central were acceptable, particularly for symptomatic participants.

Overall, participants reacted enthusiastically to the possibility of implementing their care home’s testing capability; the ability to promptly react on results, especially if positive, has a great effect on staff confidence and it might limit the detrimental effects of unnecessary isolations of residents. As part of the effort to gradually re-open care homes to planned visits, a POC test will contribute to the opportunity to screen care home visitors for COVID-19. However, the specific workflow or safety issues associated with testing visitors were not investigated in this study; given the atypicality of virus presentation and still unclear relations between viral load and shedding, infection and infectiousness [12], results from POCT should be used in combination, and not as a substitute for infection prevention measures. When applying the study findings to the management of visitors to care homes in England, it is important to take into consideration that the rapid testing including POCKIT™ investigated in this study is not a panacea to prevent the transmission of COVID-19 but one of the enablers to guide isolation decisions for visitors, staff and patients in care homes. Therefore, the integration of POCT in care homes should not displace any of the existing infection control measures such as wearing masks, PPE, social distancing, test-trace systems.

The technology, as expected for PCR, shows much higher accuracy, based upon the level of positive agreement seen, than the lateral flow devices recently deployed as part of the UK mass testing programme [13]. PCR is likely to be substantially more expensive in terms of up-front costs than using lateral flow tests. It does, though, potentially represent a single step testing solution, shifting expense from laboratories to the community. This would offer significant resource savings for care homes in comparison with the multi-stage testing approaches that might be needed if using currently available lateral flow tests. The wider work of the CONDOR programme involves care pathway mapping which is informing National Institute for Health and Care Excellence exploratory economic modelling of SARS-CoV-2 viral detection POCTs [14]. This will consider the cost of novel POCTs as compared with standard laboratory testing alongside the potential benefits accumulated.

An important practical consideration before any rollout of this technology is that the CE mark (Conformitè Europëenne) for the POCKIT™ Central, by which the Medicines and Healthcare Products Regulatory Agency defines its terms of use, restricts its use to ‘healthcare professionals’. Although some are registered nurses, many care home staff are not registered healthcare professionals. The CE mark would have to be modified to take this into account. This is not likely to be confined to POCKIT™ Central and is a reflection of the fact that POC tests have not hitherto been widely deployed in care homes. If such technology is to be widely deployed, then CE marks will have to take account of the competency mix of care home staff.

A key limitation of this study is that it was not statistically powered to evaluate diagnostic accuracy. The low number of positives seen explains the wide CIs reported for all estimates, particularly positive agreement. We report several measures of agreement stratified by symptomatic/asymptomatic to reflect potential different use of these tests. These statistics have been shown to have limitations and biases but can serve as an indication of overall agreement between these tests, hence our reporting of multiple measures alongside full test results. Our sensitivity analysis to explore the impact of POCKIT™ Central uncertain results and machine errors showed our results to be robust.

Another limitation is the relatively small sample of care homes involved. Although we sought to evaluate the technology in a purposive sample of homes that represented the range of training and registration across UK care homes more generally, the organisational configuration of UK care is recognised to be highly heterogeneous. Also, the ‘early adopter’ care homes that routinely engage with research or service development by academics might be atypical. If wider rollout of POCKIT™ Central in care homes were to be considered as a consequence of this evaluation, we would recommend doing so in a staged way, with further in-service evaluation studies to tailor training/training material for care homes staff and to fully assess how POCKIT™ Central can be integrated into the full range of care home workflows. A robust statistically powered diagnostic accuracy study conducted in parallel could provide conclusive evidence as to whether POCKIT™ Central compared with laboratory RT-PCR performs as well in the care home setting as it does in other contexts.

In conclusion, this evaluation of POCKIT™ Central highlights that POCT PCR for COVID-19 settings is feasible in care home settings. We have not, hitherto, identified significant safety concerns. The agreement with diagnostic benchmark laboratory PCR tests was good. If implemented at greater scale, further diagnostic accuracy evaluations alongside a further evaluation of usability and integration should be conducted in parallel. This could include evaluation of additional use cases, including extending testing to care home visitors.

Click here for additional data file.