Effectiveness evaluation of digital contact tracing for COVID-19 in New South Wales, Australia.

BACKGROUND: Digital proximity tracing apps were rolled out early in the COVID-19 pandemic in many countries to complement conventional contact tracing. Empirical evidence about their benefits for pandemic response remains scarce. We evaluated the effectiveness and usefulness of COVIDSafe, Australia's national smartphone-based proximity tracing app for COVID-19.
METHODS: In this prospective study, done in New South Wales (NSW), Australia, we included all individuals in the state who were older than 12 years with confirmed, locally acquired SARS-CoV-2 infection between May 4 and Nov 4, 2020. We used data from the NSW Notifiable Conditions Information Management System, the national COVIDSafe database, and information from case interviews, including information on app usage, the number of app-suggested contacts, and the number of app-suggested contacts determined by public health staff to be actual close contacts. We calculated the positive predictive value and sensitivity of COVIDSafe, its additional contact yield, and the number of averted public exposure events. Semi-structured interviews with public health staff were done to assess the app's perceived usefulness.
FINDINGS: There were 619 confirmed COVID-19 cases with more than 25 300 close contacts identified by conventional contact tracing during the study period. COVIDSafe was used by 137 (22%) cases and detected 205 contacts, 79 (39%) of whom met the close contact definition. Its positive predictive value was therefore 39%. 35 (15%) of the 236 close contacts who could have been expected to have been using the app during the study period were identified by the app, making its estimated sensitivity 15%. 79 (0·3%) of the estimated 25 300 contacts in NSW were app-suggested and met the close contact definition. The app detected 17 (<0·1%) additional close contacts who were not identified by conventional contact tracing. COVIDSafe generated a substantial additional perceived workload for public health staff and was not considered useful.
INTERPRETATION: The low uptake of the app among cases probably led to a reduced sensitivity estimate in our study, given that only contacts who were using the app could be detected. COVIDSafe was not sufficiently effective to make a meaningful contribution to the COVID-19 response in Australia's most populous state over a 6 month period. We provide an empirical evaluation of this digital contact tracing app that questions the potential benefits of digital contact tracing apps to the public health response to COVID-19. Effectiveness evaluations should be integrated into future implementations of proximity contact tracing systems to justify their investment. FUNDING: New South Wales Ministry of Health (Australia); National Health and Medical Research Council (Australia).

Introduction

Contact tracing is a core component of the public health response to the COVID-19 pandemic. Contact tracing aims to interrupt transmission chains by identifying people exposed to an infected individual so that those at risk of infection can be promptly quarantined, thereby reducing further transmission. Newly diagnosed individuals are interviewed by public health staff to identify contacts during their infectious period. Contacts assessed to be at high risk of infection are then notified and directed to quarantine. This conventional approach to contact identification and notification is time consuming and resource intensive, and can become rapidly overwhelmed when incidence is high.

Smartphone-based proximity tracing apps were viewed with considerable enthusiasm early in the COVID-19 pandemic as a means of overcoming limitations of conventional contact tracing, and a wide range of apps were quickly rolled out.4, 5, 6 Most digital contact tracing apps use Bluetooth technology, wherein the occurrence of a contact between two smartphone users is indicated by the duration, frequency, and transmission strength of Bluetooth signal exchanges. Information is either stored on individuals' smartphones for a determined period of time (decentralised approach) or uploaded into a common database (centralised approach). Under the decentralised model, adopted in Canada, Finland, Germany, Switzerland, the UK, and Vietnam, the matching of contact information occurs on people's smartphones without the involvement of public health authorities, and contact notification usually occurs automatically. By contrast, contact identification under the centralised approach occurs via a common database, which public health authorities can access to undertake contact risk assessment and notification. The centralised approach has been implemented by Australia, China, France, New Zealand, Singapore, and Taiwan.8, 9, 10, 11, 12

Evidence before this study

We searched PubMed, medRxiv, and Google Scholar using variations and combinations of the terms “COVID-19”, “digital contact tracing”, “mobile phone application”, “Bluetooth”, “proximity”, “smartphone”, “effectiveness”, and “usefulness” for original research and systematic reviews published in English until Nov 3, 2021, that reported on the effectiveness of digital contact tracing tools for COVID-19. The majority of screened articles were viewpoints calling for the quick roll-out of digital tracing; mathematical modelling studies predicting the potential impact of digital contact tracing in hypothetical populations; commentaries about privacy, human rights, and legal implications; and studies about the acceptance, community perception, and uptake of mobile tracing apps among the general public. Evidence about their effectiveness is scarce, as confirmed by two systematic reviews published in July and August, 2021, that found no comprehensive effectiveness evaluations of any digital contact tracing app based on empirical data. Since then, two studies, from the UK and Switzerland, have reported that apps with data storage on individual's smartphones and automatic contact notification (decentralised design) might be able to provide some benefits to a country's COVID-19 response under specific conditions. No such evidence exists to date for tracing apps for which user data are uploaded into a centralised database and accessed by public health staff for contact tracing (centralised design), as used by Australia and many other countries. The only existing report of a centralised app was of a small proof-of-concept study from Singapore from May, 2020, involving 18 doctors and their patients in a single health-care centre. The population-level effectiveness of centralised digital contact tracing apps remains unknown.

Added value of this study

We did a comprehensive, prospectively planned study of the COVIDSafe app, Australia's national smartphone-based tracing app, using data from 619 COVID-19 cases and more than 25 300 contacts in Australia's most populous state over the course of 6 months. To the best of our knowledge, this is the first study to assess the effectiveness of a centralised digital tracing app for COVID-19 compared with conventional contact tracing. We found that the COVIDSafe app did not make a quantitatively meaningful contribution to the COVID-19 response. The main reasons were that the use of the app among the population at risk of infection was much lower than in the general population, that the ability of the system to detect high-risk contacts and to correctly identify close contacts as such was poor, and that the additional yield of high-risk contacts who were missed by conventional contact tracing was minimal. At the same time, the app generated high perceived workload for public health staff, leading to considerable opportunity costs.

Implications of all the available evidence

We provide an empirical evaluation of the Australian COVIDSafe app that questions the potential benefits of digital contact tracing apps for the public health response to COVID-19. The fact that we found no evidence that COVIDSafe, a prominent example of centralised tracing apps, added substantial value to conventional contact tracing highlights the importance of supportive evidence of benefits to justify investment into this type of digital tracing. Effectiveness evaluations should be integrated into further implementations of proximity contact tracing systems and other digital tools that are intended to improve the pandemic response of public health systems.

Despite implementation in many countries, little empirical evidence exists about the effectiveness of digital tracing apps.13, 14 Although two recent evaluations from the UK and Switzerland suggest that systems based on the decentralised approach might be able to provide some benefits to a country's COVID-19 response under specific conditions,15, 16 no evidence exists to date about the population-level effectiveness of centralised digital contact tracing apps. Considering the substantial investments to develop, promote, and maintain these systems, it is important to establish how well they work, and what their added value within the existing contact tracing system is. Therefore, we assessed the effectiveness and usefulness of COVIDSafe, Australia's national smartphone-based proximity tracing app.

Methods

Study design

We did a prospective study to assess the COVIDSafe app (hereinafter the app) as an integrated tool within the existing contact tracing system in the state of New South Wales (NSW), Australia. Our objectives were to assess the effectiveness of the app to detect close contacts and prevent public exposure events, and its usefulness during the contact identification and risk assessment process (see protocol in appendix p 2).

We obtained ethics approval from the University of New South Wales Human Research Ethics Committee before the conduct of this study (reference number HC200468).

Study population and data collection

We included all individuals with confirmed, locally acquired SARS-CoV-2 infection recorded in NSW between May 4 and Nov 4, 2020. Individuals aged 12 years and younger were excluded because app use was not systematically assessed for this age group. Individuals with overseas-acquired infections were also excluded because all international arrivals into NSW were required to complete 14 days in government-managed hotel quarantine directly upon arrival. A person testing positive for SARS-CoV-2 was defined as app-using if they reported having the app installed and running on their smartphone for at least part of their infectious period. We extracted information on people who were SARS-CoV-2-positive and using the app from the NSW Notifiable Conditions Information Management System (NCIMS), which is used to record standardised information for all people with a confirmed SARS-CoV-2 infection in NSW. Additionally, we maintained a dedicated database during the study period to track app-related parameters for people using the app who were SARS-CoV-2 positive and their contacts. Information collected included whether app data were accessed by public health staff, the reason for not accessing app data where applicable, the total number of app-suggested contacts during the case's infectious period, and the number of app-suggested contacts determined by public health staff to be close contacts on the basis of the risk assessment and data reconciliation process. App use was recorded for close contacts if they were app-suggested.

Additionally, we did semi-structured interviews with public health staff in NSW to assess the app's usefulness in the contact tracing process. There were 119 public health staff at the public health unit level trained in the app, but only ten staff had practical experience in using the app during contact tracing. Staff were initially invited via a group email and subsequently followed up individually. We did six interviews with experienced staff, of which five were done through video conferencing platforms, audio recorded, and transcribed verbatim, whereas the sixth interview was done over email. Interviews took between 20 and 60 min. Details on the qualitative methods are in the appendix (p 8).

Verbal informed consent was sought from all interviewees before starting the interview.

The use of the COVIDSafe app in NSW

In Australia, public health is the responsibility of state and territory governments, with the national government providing funding and coordination. The national government launched the COVIDSafe app on April 26, 2020, with the goal of enhancing COVID-19 contact tracing nationwide, and promoted its download and use through a concerted multimedia information campaign. The app is based on the centralised approach and is intended to supplement rather than to replace conventional interview-based contact tracing. By voluntarily downloading the app, smartphone users consent to exchanging coded information with other app users' smartphones that have come within sufficient proximity to exchange Bluetooth signals. These data are automatically deleted after a rolling 21-day period. Once a person tests positive for SARS-CoV-2 (defined as a case), public health staff can seek verbal informed consent from them to access app data for contact tracing purposes. App use is ascertained as part of the standard case interview. If consent is provided, data are uploaded to the COVIDSafe National Data Store database. Data include names or user-selected aliases, telephone numbers, postcodes of residence, age ranges, and dates and times of signal exchanges between smartphones. For privacy reasons, no information is collected about the location where signal exchanges took place. The database also contains information on the strength and length of the exchanged Bluetooth signals, which allows estimation of the duration and physical proximity between smartphones during the signal exchange. An inbuilt filtering algorithm selects only those contacts whose signal exchanges with the case's smartphone meet predefined thresholds. These thresholds were set to approximate the national close contact definition in use at that time, namely face-to-face contact for at least 15 min over the course of 1 week during the case's infectious period or presence in an enclosed space with an infectious person for at least 2 h. For symptomatic cases, the infectious period for contact tracing purposes at that time was from 48 h before symptom onset until 10 days since symptom onset and absence of symptoms for at least 72 h. For asymptomatic cases, the infectious period was from date of positive SARS-CoV-2 test until 10 days since first positive test.

This list of app-suggested contacts is accessible to public health staff via a secure web portal, who then do a risk assessment and reconciliation process of app data with information obtained from the case interview and from other sources, such as attendance lists of venues or institutions. At the end of this risk assessment process, only those app-suggested contacts who are considered by public health staff to meet the close contact definition criteria are contacted and asked to self-quarantine for 14 days from the date of exposure.

The web portal access became available to NSW public health authorities on May 8, 2020. Between May and July, 2020, the national Department of Health and the Digital Transformation Agency worked together with state and territory health authorities, including NSW, to improve the use of the app and the web portal. During this period, app-related public health activities in NSW were managed centrally by the Public Health Response Branch within the NSW Ministry of Health. From August, 2020 onwards, the app was progressively rolled out to public health units in NSW, situated within each local health district of the state.

Statistical analysis

The demographic, socioeconomic, and epidemiological characteristics of cases were stratified by app usage. Comparisons between app-using and non-app-using cases were assessed using χ2 tests for categorical variables and Wilcoxon rank-sum tests for medians of continuous variables. We calculated the number and proportion of cases whose app data were accessed and used, as well as the number and proportion of close contacts generated from those cases. We also calculated the additional contact yield and the number of prevented exposure events attributable to the app. We calculated the positive predictive value of app-suggested contacts, defined as the proportion of app-suggested contacts assessed by public health staff to meet the close contact definition. We also estimated the sensitivity of the app, defined as the proportion of all close contacts identified by conventional contact tracing methods that were app-suggested. Because links between surveillance data of cases and close contacts were not documented in NCIMS before Aug 1, 2020, and because app use among contacts who were not app-suggested was not systematically ascertained, we applied the same proportion of app usage among cases to close contacts that were identified by conventional contact tracing and were registered in NCIMS between Aug 1 and Nov 4, 2020, to estimate sensitivity. All statistical analyses were done using Stata (version 15).

For the qualitative analysis, data were coded using NVivo (version 12; QSR International, Melbourne, VIC, Australia) and analysed thematically.

Role of the funding source

New South Wales Ministry of Health staff with responsibility for surveillance data provided advice on technical aspects of study design, data collection, data analysis, data interpretation, and drafting of the evaluation report. They were invited to comment on the article manuscript but had no role in the finalisation or the decision to submit for publication. The National Health and Medical Research Council of Australia had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Results

Between May 4 and Nov 4, 2020, 619 individuals recorded in NSW had confirmed SARS-CoV-2 infection acquired within Australia and were older than 12 years. More than 25 300 close contacts were identified through conventional contact tracing during the same period. 137 (22%) of the 619 cases used the app for at least part of their infectious period. Cases who used the app were less likely to live in socioeconomically disadvantaged areas, and more likely to be born in Australia than cases who did not use the app (table ). Cases who used the app were also more likely to have acquired infection from a contact outside their household or as part of a community cluster, and to have more close contacts than cases not using the app. There were no significant differences between cases using and not using the app by sex, age, or geographical remoteness.

Table

Characteristics of cases with Australian-acquired SARS-COV-2 infection in NSW by COVIDSafe app use, May 4 to Nov 4, 2020

		Cases using the app*(n=137)	Cases not using the app*(n=482)	p value
Sex		..	..	0·82
	Female	71 (52%)	255 (53%)	..
	Male	66 (48%)	227 (47%)	..
Age, years		38 (22–56)	40 (22–57)	0·64
Remoteness of local health district of residence		..	..	0·10
	Major city local health districts	133 (98%)	453 (94%)	..
	Regional local health districts	3 (2%)	27 (6%)	..
Postcode-level index of relative social disadvantage†		..	..	0·0012
	Least disadvantaged	71 (52%)	162 (34%)	..
	Less disadvantaged	40 (29%)	128 (27%)	..
	Most disadvantaged	26 (19%)	191 (39%)	..
Country of birth		..	..	0·026
	Australia	76 (55%)	209 (43%)	..
	Overseas	49 (36%)	234 (49%)	..
	Not stated	12 (9%)	39 (8%)	..
Likely source of infection		..	..	0·0050
	Household contact	33 (24%)	170 (35%)	..
	Other contact or cluster-associated	93 (68%)	246 (51%)	..
	Source not determined	9 (7%)	45 (9%)	..
	Interstate acquired	2 (1%)	20 (4%)	..
Median number of close contacts‡		5 (0–121)	2 (0–196)	0·0014

Data are n (%) or median (IQR).

Excluding missing values.

Based on the Australian Bureau of Statistics' Index of Relative Socioeconomic Disadvantage data.

For the period Aug 1 to Nov 4, 2020 only, when linkage between cases and contacts was of sufficient quality to report estimates.

App data were accessed by public health staff for 92 (67%) cases using the app (figure ). Reasons for not accessing app data were: the case being in isolation during their entire infectious period (n=31, 69%); technical issues (n=5; 11%); loss to follow-up (n=4; 9%); and declining consent (n=3; 7%). Among those whose data were accessed, the app did not record any contacts during the infectious period for 60 (65%) cases, leaving 32 with at least one app-suggested contact (5% of the total 619 cases in NSW). These 32 cases had 205 app-suggested contacts. Following the risk assessment and data reconciliation by public health staff, 79 (39%) app-suggested contacts were assessed as meeting the close contact definition and were directed to self-quarantine. Therefore, the positive predictive value of the app in correctly identifying close contacts was 39%. Common examples of app-suggested contacts who did not meet the close contact definition included: neighbours in apartment buildings; office workers in adjacent rooms; customers in neighbouring restaurants; and people waiting in separate cars at COVID-19 drive-through testing clinics. 1073 close contacts were identified via conventional contact tracing, the app, or both after Aug 1, 2020. During the same period, the app identified 35 contacts who were also detected by conventional contact tracing and were assessed to be close contacts. Applying the same 22% app use among cases to close contacts, 236 of these 1073 close contacts could have been expected to have been using the app and thus be potentially identified by the app during this period. Because 35 (15%) of these 236 close contacts were actually identified by the app, the estimated sensitivity of the app was 15% during the evaluation period.

Figure

Use of COVIDSafe app data for contact tracing among individuals with SARS-CoV-2 infection in NSW, May 4 to Nov 4, 2020

79 (0·3%) of the 25 300 close contacts in NSW during the study period were app-suggested contacts who met the close contact definition. These 79 contacts originated from 20 cases (figure). 62 (78%) of these 79 close contacts were also identified by conventional contact tracing, none of whom became SARS-CoV-2 positive. Therefore, the additional yield of close contacts identified only via the app was 17 (<0·1%) of the more than 25 300 close contacts under follow-up. These 17 contacts originated from four cases who used the app (figure), and all completed their quarantine period without testing positive for SARS-CoV-2, hence no public exposure event was prevented by the app during the study period. One or more of the following applied for all these 17 contacts: exposures had been forgotten or incorrectly recalled by cases; contact lists of venues visited by cases were unavailable or incomplete; and a broader close contact definition was sometimes applied by public health staff when multiple transmissions were known to have occurred in a specific setting.

The qualitative study component showed that the app was perceived as considerably less useful than public health staff had hoped. Its interface was not considered to be user-friendly and, especially early in the roll-out, staff at public health units required substantial assistance from staff at the central NSW Ministry of Health to access and interpret the data, resulting in inefficiencies and delaying the roll-out of the app in NSW more generally. Public health staff reported that the process of ascertaining which of the app-suggested contacts who were not already identified through the case interview were close contacts could be lengthy. This process sometimes led to a delay in notifying close contacts, particularly when the number of app-suggested contacts to review was high. According to interviewees, the app did not shorten the timeframe for identifying close contacts. Some staff were also concerned that the app did not work equally reliably on all types of smartphones, with the perception that the number of contacts on iPhones was substantially underestimated, whereas those from Android phones were overestimated. There was a general view that the app performed poorly when running in background mode. Overall, staff assessment of the impact of the app ranged from not impacting much to being an additional step that increased workload without delivering any added value. The overwhelming response was that the app was not useful in contact tracing (more details on the qualitative results are in appendix p 8).

Discussion

COVID-19 is the first epidemic in which digital proximity tracing has been widely implemented. To the best of our knowledge, this study is the first large-scale assessment of the effectiveness and usefulness of a centralised digital tracing app. Our main findings over a 6-month period in Australia's most populous state were that less than a quarter of cases were using COVIDSafe; app data from most cases who were using the app could not be used for contact tracing; the majority of app-suggested contacts were not close contacts; the majority of close contacts identified by conventional contact tracing were not detected by the app; and the additional close contacts identified by the app who would have been missed using only conventional contact tracing represented less than 0·1% of the 25 300 close contacts recorded during the study period.

We identified three broad issues that prevented the app from making a meaningful contribution to COVID-19 contact tracing in NSW. First, the uptake of the app among the population at risk of infection was lower than expected. Although no data on the overall uptake of the app have been released by the government for any jurisdiction in Australia, approximately 44% of a representative sample of the adult Australian population reported having downloaded the COVIDSafe app in June, 2020. In our study, this proportion was less than a quarter (137 [22%] of 619) among those who actually contracted infection. Given that uptake greater than 50% is considered necessary for proximity tracing apps to help contain SARS-CoV-2, it is likely that the low uptake of COVIDSafe among the population at risk of infection alone was an important factor preventing the app from making a substantial contribution to the COVID-19 response. Data security concerns, namely lack of privacy and fearing the normalisation of governmental tracking; functionality issues, in particular the negative effect on smartphone performance through increased battery usage; and privacy considerations due to the need for a centralised database, which is an integral design feature of tracing apps based on the centralised approach, were identified early as potential barriers for acceptance of the COVIDSafe app.19, 20, 21, 22, 23

Second, with a positive predictive value of 39% and an estimated sensitivity of 15%, the diagnostic performance of the app was not sufficiently high to add value for COVID-19 contact tracing in NSW. Given that the app is intended to complement rather than to replace conventional contact tracing, a modest sensitivity might have been acceptable, particularly because some exposure settings were judged to be high risk and broader close contact criteria were applied. The low uptake of the app among cases (137 [22%] of 619) probably led to a reduced sensitivity estimate in our study, given that only contacts who were using the app could be detected by the app. However, this does not apply to the positive predictive value, which is calculated among app users only. The app's positive predictive value to correctly identify close contacts remained low despite filtering for Bluetooth signal exchange patterns that approximated the close contact definition. This meant that individual risk assessment by public health staff of all app-suggested contacts was necessary, as the imposition of unwarranted 14-day quarantine for the 61% of all app-suggested contacts who were not deemed close contacts would have been unacceptable in a relatively low-transmission setting such as NSW in 2020. Investigators of a small COVID-19 cluster that occurred in Sydney in March and April, 2020 before the introduction of the COVIDSafe app, speculated that the app might have helped to speed up the identification of contacts and links between cases. However, the poor diagnostic accuracy parameters established in our study suggest otherwise. The potential effect of the COVIDSafe app was also overestimated in an earlier modelling study that projected a 25% reduction in cases if 27% of the population (ie, slightly more than the proportion among cases in our study) had downloaded it. The only other empirical evaluation of a proximity tracing app of a design similar to COVIDSafe was a small proof-of-concept study of Singapore's TraceTogether app among staff and patients in a clinical setting, which found performance parameters even lower than in our study.

Third, the app had a low perceived usefulness by public health staff while generating high workload. In particular, interviewed staff flagged the perceived discrepancy between iPhones and Android phones in their ability to detect contacts. The failure to register contacts on iPhones was also noted in a pilot study of a similar app in the UK, which was later abandoned in part due to this problem. Another technical flaw of the app that affected its usefulness according to interviewed staff was its poor ability to register contacts unless the app was open. Problems with the COVIDSafe app picking up Bluetooth signals from locked phones or when running in the background had been noted early, but were not sufficiently resolved. At the same time, the app did pick up a high proportion of contacts who were not deemed close contacts, which caused high workload for public health staff during the risk assessment process. This could have easily overwhelmed public health staff had incidence been substantially higher during the evaluation period, leading paradoxically to a reduction in usefulness of the app when it would be needed the most.

There are several limitations to our study. First, although app usage among cases in our study was only half of that estimated in the general population in Australia, we did not investigate the reasons for non-uptake among cases. Second, we could not determine to what extent the absence of registered contacts during the entire infectious period of some cases who were using the app was real or a result of technical problems of the app as already described. Third, before August, 2020, the quality of surveillance data was insufficient to establish a reliable count of close contacts per case, and app use among contacts who were not suggested by the app was not ascertained. As a result, we had to assume the same levels of app uptake among cases and contacts to estimate sensitivity. Due to the observational nature of our study, integrated as implementation research within the routine activities of a rapidly evolving COVID-19 response, we were not able to report either the number of cases among contacts who were identified by the app but missed by conventional contact tracing, or the number of contacts who used the app and were identified by conventional contact tracing but were missed by the app. Fourth, contrary to our original plans (see protocol in the appendix p 2) it was not feasible to quantify the overall workload generated by the app for public health staff, in particular for the risk assessment of app-suggested contacts who did not meet the close contact definition, and to compare the time needed to trace contacts identified by the app and by conventional contact tracing. Fifth, although we had to exclude cases aged 12 years or younger, due to unavailability of data, we could not do the same among contacts who were identified by conventional contact tracing, which might have led to an underestimation of sensitivity in our analysis. Sixth, we could not assess if any under-reporting of app use occurred by cases during the interview with public health staff. Seventh, the particular epidemiological context, characterised by relatively low incidence and no uncontrolled community transmission during most of the evaluation period, together with the strong public health capacity in NSW, make the extrapolation of our results to other contexts difficult. Eighth, we could not quantify some anecdotal observations by public health staff about flawed app reporting—eg, registering people as close contacts despite never having been in the same room with a case, or under-reporting of contacts on iPhones. Ninth, the absence of infections among the 17 contacts only identified by the app should be interpreted with great caution given the highly heterogenous infectiousness of SARS-CoV-2 cases.

Although the app still exists, anecdotal reports suggest that it is no longer in widespread use among the general public. Key implications of our findings for future use of the COVIDSafe app are that conventional case interviews should remain the primary source of contact tracing information, and that all potential contacts, whether app-suggested or not, should undergo the same risk assessment by public health staff to establish their infection risk. The app might have shown greater benefits if uptake among the population at risk of SARS-CoV-2 infection had been higher. The app might also have been more effective had it been used in a more targeted manner—eg, for cases unable to give a reliable history, or for high-risk exposure venues with incomplete attendance record keeping. It should be noted here that the close contact definition at the time of the study did not weight airborne spread as a major driver of transmission and would thus now be considered too conservative.

Results from the only other two empirical studies on the effectiveness of digital contact tracing from the UK and Switzerland are difficult to compare with our findings. Those settings had much higher case numbers and different evaluation methodologies were used. In addition, their tracing apps are based on the decentralised system architecture. Both studies conclude positively about the performance of the apps. The main advantage of decentralised systems with regards to the problems encountered with the COVIDSafe app is that no involvement of public health staff is needed to inform contacts using the app about their exposure to a case who was using the app, as this happens automatically through the app once the infection status of an app user gets changed from negative to positive. This reduces workload and makes scalability easier than with the centralised system.

Shortfalls during the implementation of innovative interventions are to be expected during pandemic responses, and their value needs to be considered alongside other response measures, many of which also have limited effectiveness. Our evaluation highlights the importance of piloting technical innovations in real-life settings to detect and amend technological flaws early, and to systematically seek input during the developmental stage from end-users, in this case public health practitioners tasked with contact tracing and smartphone users of the general public. Even rapid piloting might increase acceptance and usefulness of such technology. We understand that neither happened in a meaningful way before the implementation of the COVIDSafe app.

The costs for developing and operating the COVIDSafe app have been estimated at AU$6·75 million up to January, 2021, with ongoing monthly maintenance costs of approximately AU$100 000. These costs, coupled with the substantial additional workload required for public health staff to identify a very small number of additional close contacts, call for a formal cost–benefit assessment of the COVIDSafe app, and of digital tools in the public health response to COVID-19 more generally.

In conclusion, the COVIDSafe app was not sufficiently effective to make a meaningful contribution to COVID-19 contact tracing in Australia's most populous state during a 6 month period in 2020. Key issues were low uptake of the app among the population at risk of infection, its poor positive predictive value and sensitivity, and difficulties for public health staff in accessing app-derived data. The additional contact yield was minimal and did not prevent any public SARS-CoV-2 exposures. At the same time, the app generated substantial workload for public health staff, leading to high opportunity costs. To the best of our knowledge, this is the first empirical study that questions the potential benefits of a digital contact tracing app for the public health response to COVID-19, calling for supportive evidence of benefits to justify investment into apps like COVIDSafe. Real-world piloting, input from users during the development phase, and comprehensive effectiveness evaluations after roll-out are important elements to ensure added value of digital tracing technology for public health.

Data sharing

Data underlying this research will be made available upon individual, reasonable request to the corresponding author by other researchers.

Declaration of interests

We declare no competing interests.