Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settings.

BACKGROUND: COVID-19 is an ongoing public health pandemic regardless of the countless efforts made by various actors. Quality diagnostic tests are important for early detection and control. Notably, several commercially available one step RT-PCR based assays have been recommended by the WHO. Yet, their analytic and diagnostic performances have not been well documented in resource-limited settings. Hence, this study aimed to evaluate the diagnostic sensitivities and specificities of three commercially available one step reverse transcriptase-polymerase chain reaction (RT-PCR) assays in Ethiopia in clinical setting.
METHODS: A cross-sectional study was conducted from April to June, 2021 on 279 respiratory swabs originating from community surveillance, contact cases and suspect cases. RNA was extracted using manual extraction method. Master-mix preparation, amplification and result interpretation was done as per the respective manufacturer. Agreements between RT-PCRs were analyzed using kappa values. Bayesian latent class models (BLCM) were fitted to obtain reliable estimates of diagnostic sensitivities, specificities of the three assays and prevalence in the absence of a true gold standard.
RESULTS: Among the 279 respiratory samples, 50(18%), 59(21.2%), and 69(24.7%) were tested positive by TIB, Da An, and BGI assays, respectively. Moderate to substantial level of agreement was reported among the three assays with kappa value between 0 .55 and 0.72. Based on the BLCM relatively high specificities (95% CI) of 0.991(0.973-1.000), 0.961(0.930-0.991) and 0.916(0.875-0.952) and considerably lower sensitivities with 0.813(0.658-0.938), 0.836(0.712-0.940) and 0.810(0.687-0.920) for TIB MOLBIOL, Da An and BGI respectively were found.
CONCLUSIONS: While all the three RT-PCR assays displayed comparable sensitivities, the specificities of TIB MOLBIOL and Da An were considerably higher than BGI. These results help adjust the apparent prevalence determined by the three RT-PCRs and thus support public health decisions in resource limited settings and consider alternatives as per their prioritization matrix.

Introduction

Coronavirus disease 2019 (COVID-19) is still an ongoing global pandemic and a public health emergency resulting in more than 4,891,942 deaths from 240,061,637 documented cases as of 13 October, 2021 the first reported case in China, Wuhan in December 2019 [1,2].

The global community had three epidemic surges of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) causing COVID-19. This has imposed an immediate increasing demand on daily diagnostic screening, with the aim to shorten the epidemic curve and to prevent the occurrence of new variants of concern and interests, which could lead the next wave as has, occurred in some parts of the globe [3,4]. It is a pushing factor and time for designing and a critical need for accurate and rapid diagnostic assays to prompt clinical and public health interventions. This can help to perpetuate over the upcoming time, due to the recurrence of outbreaks [5]. One step reverse transcriptase PCR is a popular diagnostic technique for novel Corona virus ever since the first case of COVID-19 reported elsewhere [6,7].

In Africa, including Ethiopia, COVID-19 laboratory testing facilities are centralized, that have been managed in a referral linkage of a limited access to laboratory facilities, by which there is a possibility of overloading and flooding of the already overwhelmed reference laboratories during emergency conditions. Availability, low cost and user friendliness are highly desirable in Ethiopia. While high sensitivity is of paramount importance not to miss any COVID-19 case and stop virus transmission [8], diagnostic specificity is highly relevant when tests are used as confirmatory and to avoid the costs of false positive results. In most COVID-19 diagnostic tests accuracy studies, RT-PCR is considered as the gold standard, i.e. having 100% diagnostic sensitivity and specificity. Despite the tremendous worldwide research activities related to COVID-19 diagnostic tests, very few diagnostic test accuracy studies were conducted to adjust imperfect tests [9-13]. So far, very few attempts were made to assess the diagnostic performance of RT-PCRs in Ethiopia in a clinical setting.

COVID-19 pandemic in Ethiopia

Ethiopia is the second most populous country in Africa. It is one of the 220 COVID-19 affected countries and territories, since the first confirmed COVID-19 was reported on March 13, 2020 [14]. Presently case numbers are increasing rapidly. There were two peaks of kurtosis of the epidemic; first wave from August to September 2020, and second wave occurred during February to March 2021. There are reports of Delta variants in parts of Africa including Ethiopia which might have resulted in the third wave [15,16]. As of 13 October 2021 there are 356,772 confirmed cases which make Ethiopia the 4th most affected country in Africa [15] with 6,103 documented deaths. A total of 3329 health professionals have contracted the disease and 36 of them have died [15,17].

PCR-based laboratory testing is indispensable in identifying the infected cases and also curbs the transmission epidemic. In Ethiopia, so far there are 3,574,218 total tests using RT-PCR, which equates to 30,136 tests per million population, since the first COVID-19 case was reported and the subsequent declaration of global pandemic [15,17]. Even though the country has come a long way in increasing the testing capacity through redirecting and mobilizing resources and organizing testing campaigns, the tests per the total population are still very low and need due attention [18,19].

Available diagnostic platforms for the diagnosis of COVID-19

Since the occurrence of the current pandemic: Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2) infection on December, 2019 in Wuhan, China, Scientist have been developing diagnostic tools with different diagnostic tests, relying on different biological principles [20].

Detection methods based on nucleic acid amplification tests (NAAT) are popular and preferable for novel Coronavirus in many countries. The test demonstrated highest sensitivity when it performed at the earliest time point in the acute phase of infection. The current reference or gold standard for diagnosing COVID-19 depends up on detection of the viral genetic material (RNA) in a nasopharyngeal swab or sputum sample., it requires PCR, a technology that amplifies the amount of genetic material to detectable levels and takes several hours to perform [21].

One-step reverse transcriptase-polymerase chain reaction (RT-PCR) is the established reference standard test for the diagnosis of COVID-19 because of its performance in identifying SARS-CoV-2 infection. This technique is sensitive and can detect the virus earlier in the infection [21]. Currently, there are different RT-PCR assays with quite different analytical and diagnostic sensitivity and specificity have been developed and recommended by the WHO and used for the diagnoses of SARS-CoV-2 [22,23].

Evidence indicated that, in many countries all available test kits have been used including rapid diagnostic tests that detects the virus either an antibody or an antigen based, whereas Ethiopia uses both RT-PCR and Ag RDT for the diagnosis of SARS CoV- 2 [8,18,24]. Indeed, evidence based information on the diagnostic performance of the commercially available test assays in the country and currently used by most COVID-19 testing laboratories are critical and timely for the program managers and have its impact in the containment strategy of the virus through quality test kit selection on their prioritization and continuous quality improvements on its development as part of the control of the pandemic [9].

In view of this, many pharmaceutical companies have accomplished different activities across the world as part of the pandemic response, which can benefit synergetic modality. However, so far because of its analytical and diagnostic presentation, RT-PCR is the reference method for the detection of SARS-CoV-2, irrespective of their variability in performance [25]. As of September 11, 2021 there are 1129 SARS-CoV-2 tests that are commercially available or in development for the diagnosis of COVID-19 and evaluated through FIND in collaboration with WHO and partners. FIND has been conducting independent evaluation of the available assays and summarized data for emergency use authorization (EUA) [23].

Testing for SARS–CoV-2 is becoming increasingly available in high-income countries and though still scant in resource limited countries, and even when available, it is highly centralized and limited to more specialized institutions with greater supply chain bottlenecks [26].

Reverse transcriptase-PCR test is usually done to identify positive cases in a population to prevent transmission to others and monitoring patient outcome. Likewise, it is critical importance for the mitigation of the pandemic. This is very much pondering if and only if the laboratory used better comparable quality diagnostic assays [27-30].

In Ethiopia, there are considerable numbers of one step RT-PCR assays commercially available for the diagnosis of COVID-19 either by donation or direct procurement by government. Among the available assays, three of them Sarbeco virus SARS-CoV-2 TIB MOLBIOL (hereafter named as TIB), Da An and BGI are widely used for the diagnosis of SARS CoV-2 at different time. TIB assay has two targets, E and RdRp involving two steps. The first step is screening by the E gene and the second step is confirming all positive cases by the RdRp gene. Da An assay works by multiplexing and targets two viral genes (nucleocapsid (N) and the ORF 1ab). The BGI assay targets only one viral gene the ORF 1ab. Each assay has its own procedural and result interpretation steps. However, their performance in a clinical setting in Ethiopia has not been assessed.

As PCR is not considered as gold standard our main scientific hypothesis is the assessment of comparing the ct levels and then the demographic data, and finally assessing agreement between the PCRs and obtaining robust estimates for diagnostic sensitivities and specificities in the less developed countries.Therefore, the current study was aimed to evaluate the diagnostic sensitivities and specificities of three commercially available one step reverse transcriptase assays for the detection of SARS-CoV-2 in resource limited settings, Ethiopia using Bayesian latent class model (BLCM).

Materials and methods

Study design, period and settings

A health facility based cross sectional study was conducted at Parasitology COVID-19 testing laboratory of Ethiopian Public Health Institute (EPHI) from April to June 2021. Nasopharyngeal swabs were collected from 279 individuals referred for COVID-19 testing from health facilities of Addis Ababa. The laboratory is amongst the first established national COVID-19 testing laboratory with qualified and competent performance which has been serving COVID -19 testing for samples come from referring health facilities of Addis Ababa through the national emergency operation center (EOC) [18].

Flow of study participants

In this study a total of 468 eligible participants who fulfill the WHO criteria for COVID-19 suspected cases were screened [2]. Of these a total of 279 nasopharyngeal swabs collected by trained health professionals who encompass from community surveillance consider them as population one (32.62%), contacts of confirmed cases, consider as population two) (17.2%), and suspects, consider them as population 3 (50.18%). The detail is depicted at diagram 1 in S1 Annex.

Test kits selection

Reverse transcriptase-PCR test has been designed based by targeting some parts of SARS-COV-2: spike (S), open reading frame1a/b (ORF1a/b), ORF1b-nuclear shuttle protein14 (ORF1b–nsp14)), envelope (E), RNA-dependent RNA polymerase (RdRp) or nucleocapsid (N) genes from the structural and nonstructural part of the gene [27]. Accordingly, the current study evaluated test kits employed E gene, RdRp, N gene and ORF as illustrated in Table 1. However, their respective pharmaceutical companies had not had any involvement with the research methodology design, analysis and write up of the research manuscript.

Table 1

RT-PCR test kits with their corresponding primers and probes sequences used in this study, Addis Ababa, Ethiopia, 2021.

S#	Assay name and company	# of targets	Target Gene	Primers and probes sequence information (5’-3’)	Length(bases)	Ref
1	TIB MOLBIOL, Germany *	2	E gene (Screening)	Forward primer: ACAGGTACGTTAATAGTTAATAGCGTReverse primer: ATATTGCAGCAGTACGCACACAProbe: FAM ACACTAGCCATCCTTACTGCGCTTCG-BBQ	145 bp	[31,32]
			RdRp gene (Confirmatory)	Forward primer: ATGAGCTTAGTCCTGTTGReverse primer: CTCCCTTTGTTGTGTTGTProbe: FAM-AGATGTCTTGTGCTGCCGGTA [5’]Hex [3’]BHQ-1	196 bp	[32]
2	Da An Gene Co., Ltd. of Sun Yat-sen University, China	2	N gen	Forward primer: TAATCAGACAAGGAACTGATTAReverse primer: CGAAGGTGTGACTTCCATGProbe: FAM/ZEN-GCAAATTGTGCAATTTGCGG-IBFQ	323 bp	[32,33]
			ORF1ab	Forward: AGAAGATTGGTTAGATGATGATAGTReverse: TTCCATCTCTAATTGAGGTTGAACCProbe:FAM TCCTCACTGCCGTCTTGTTGACCA-BHQ1	588 bp	[32,34]
3	BGI	1	ORF1ab	Forward primer: AGAAGATTGGTTAGATGATGATAGTReverse primer: TTCCATCTCTAATTGAGGTTGAACCProbe:FAM-TCCTCACTGCCGTCTTGTTGACCA-BHQ1	588 bp	[32,34]

Note:-FAM: 6-carboxyfluorescein; BBQ: blackberry quencher; BHQ-1: Black Hole Quencher-1;. W is A/T; R is G/A; M is A/C; S is G/C, RdRp: RNA-dependent RNA polymerase

*we have used WHO TIB reagent. First line screening assay: E gene assay, Confirmatory assay: RdRp gene assay.

Laboratory procedures

RNA purification. Respiratory specimens (throat/nasopharyngeal) collected in 2 mL VTM (China, Miraclean Technology Co.,Ltd., www.mantacc.com) by well trained professionals were used for RNA purification. Briefly, 200 μL of respiratory samples were transferred into 1.5 ml eppendorf tube. Then, 50 μl of proteinase K and 200 μl of lysis buffer were added after brief centrifugation and incubation of the tube at 72°C for 10 minutes. 250 μl of absolute alcohol was added and the whole mixture was transferred to spine column. The remaining procedures were performed as per the Da An RNA/DNA purification kit insert (Da An Gene Co., Ltd. of Sun Yat-sen University). In all extraction procedures, positive and negative controls were included.

Detection of SARS-CoV-2 by RT-PCR

Following the Viral RNA extraction the extracted samples were detected by the three (TIB, Da An and BGI) commercially available RT-PCR assays against their respective manufacturer recommendations and instructions on a Quant Studio 5 DX real-time PCR system (catalog number A34322, Thermo Fisher Scientific), and data were analyzed and computed. Master Mix preparation and cycling conditions for respective assays presented in Table 2.

Table 2

RT-PCR test kits with their corresponding master mix protocols used in this study, Addis Ababa, Ethiopia, 2021.

Types of assay	Master mix	Volume of reagent used per reaction	Cycling conditions
TIB (EAV/RdRp)	Nuclease free water	Nx3.3 μl	Initial RT step 50°C for 30 minutes
	Combined primer/Probe mix	N x 0.5 μl	95°C 2 minutes
	Internal control	N x 0.5 μl
	qScript One-Step enzyme	N x 0.7 μl	95°C for 15 seconds	X45 cycles
	2X one Step master mix	N x 10 μl	55°C for 30 seconds
	Total volume	N x 15 μl
	Template per each reaction tube	5 μl
Da An	Solution A	N x 17 μl	Initial RT step 50°C for 15 minutes
	Solution B	N x 3 μl	95°C 15 minutes
	Total volume	N x 20 μl	95°C for 15 seconds	X45 cycles
	Template per each reaction tube	5 μl	55°C for 30 seconds
BGI	Reaction Mix	N x 18.5 μl	Initial RT step 50°C for 15 minutes
	Enzyme mix	N x 1.5 μl	95°C 15 minutes
	Total volume	N x 20 μl	95°C for 15 seconds	X40 cycles
	Template per each reaction tube	10 μl
			55°C for 30 seconds

Determination of results

For the TIB assay, a 76 bp long fragment from the E gene is amplified with specific primers and detected with a FAM label hydrolysis probe. The assay detects SARS and 2019-nCoV pneumonia virus (bat-associated SARS related Sarbeco virus has its own control reaction of 70 bp fragment from Equine Arteritis Virus detected with an Atto647 labeled probe. It’s RNA positive control contains all diagnostic targets E gene, N gene and RdRP. Mainly it had FAM Color Module 1 and Cy5: Color module 5 (IC) by RdRp also FAM: Color Module 1 and Cy5: Color module 5 (IC) by E gene. It has two steps to declare positive: screening test done by E gene and if any clinical sample positive by the E-gene assays are subject to repeat and confirmatory tests done by the RdRp gene-specific to SAR-CoV-2 [35].

For Da An assay, if the test sample has no amplification curve in the FAM and VIC channels, but amplification curve in the Cy5 channel, the result can be judged as there is no 2019 Novel Coronavirus (2019-nCoV) RNA in the sample; If the test sample has obvious amplification curve in the FAM and VIC channels and Ct values ≤40, the result can be judged as the sample is positive for 2019 novel coronavirus; if the test sample only has the ct value ≤40 in a single channel of FAM or VIC and there is amplification curve in Cy5 channel, the results need to be re-tested. If the re-test results are consistent, the sample can be judged as positive for 2019 Novel Coronavirus (2019-nCoV). If the re-test results are negative, it can be judged that no 2019 Novel Coronavirus (2019-nCoV) RNA has been detected. The ROC curve method is used to determine both the reference Ct value of the kit and the internal standard reference values are 40 for positive value judgment [36].

Interpretation of test results. in each test procedure, negative, positive and no template controls were included. Only when the controls meet the requirements can the test results be determined. When the FAM and VIC detection channels are positive, the result from the Cy5 channel (internal standard channel) may be negative due to the competition of the system and when the internal standard result is negative, if the FAM and VC detection channels of the test tube are also negative, it indicates that the system is inhibited or the operation is wrong, the test is invalid. Therefore, the sample needs to be re-tested [36].

In case of BGI, the controls for the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV are evaluated using the nucleic acid amplification curve and Ct values generated by the RT-PCR system. Ct cut-off values were determined using the receiver operator characteristic curves of the tested clinical samples. The Ct value in the FAM channel for a valid no template (negative) control should be “0” and there should be no sigmoidal amplification curve. Experimental analysis found that Ct value for positive SARS-CoV-2 samples should be no higher than 38. Thus, the Ct value in the FAM channel for a valid positive control should be no higher than 38 and there should be a sigmoidal amplification curve. Experimental analysis found that Ct value for the internal positive control samples should be no higher than 35. Thus, the Ct value in the VIC/HEX channel for a valid internal positive control should be no higher than 35 and there should be a sigmoidal amplification curve [37].

Data quality assurance

Sample and data was collected by well trained professionals. All laboratory procedures were performed as per the documented standard operating procedures and according to specific manufacturing recommendations. The quality of each reagent like lot number, expire date, storage conditions, physical leak proof and not breakages, etc were cheeked before the actual laboratory analysis. Samples and reagents were stored at appropriate temperature as indicted on the manufacturer inserts. Internal and external quality controls were run as required during laboratory analysis.

Data was double checked manually for completeness and consistency before data entry. We have used a respective recommended manufacturer protocols. The above requirements must be met at the same time in each experiment; otherwise, the experiment is invalid and needs to be carried out again.

Data analysis

For the best narrative, only results findings by three RT-PCR tests were analyzed and considered for our further analysis and interpretation, the rest were trimmed from the final analysis. Results were interpreted as positive and negative based on the cut-off Ct values of each manufacturer. The binomial 95% confidence intervals (95% CI) were calculated using MedCalc (MedCalc Software Ltd.) and P-values <0.05 considered statistical significant using Vassarstats and SPSS v.23 statistical software Cohen’s kappa statistic among the three assays. A value of 1 implies almost perfect agreement and values less than 1 imply less than perfect agreement, with a range of values between 0 and 1 [38].

Diagnostic sensitivity, specificity, predictive and kappa values were computed. Positive and negative predictive values were computed following.

Bayesian latent class model (BLCM)

With the aim to obtain diagnostic tests accuracies in the absence of a perfect gold standard, a Bayesian latent class model (BLCM) was fit to the data following the approach from Hui and Walter for three tests and three populations using MCMC (Markov chain Monte Carlo) simulation to construct posteriors in JAGS version 4.3.0 [39] using the runjags package [40].

The frequencies of the eight combinations of dichotomized RT-PCR results (+++; ++-; +-+;+—;-++,-+-,—+,—) in the three populations, respectively, were modeled with a multinomial distribution. To allow for potential conditional dependencies, pair wise covariances between sensitivities and specificities of all RT-PCRs were included in separate models. Model selection, i.e. in- or exclusion of conditional dependencies was based on the 95% credibility intervals (including 0 or not) and on Bayesian p-values. The model code (S1 File) was obtained with the function “auto huiwalter” of the runjags package [41], with three chains of 50 000 iterations each, a burn-in of 5000 iterations, and a thinning of 10 iterations. Minimally informative priors (beta (1,1)) were used for the sensitivities and prevalence’s. In contrast, for the specificities informative priors (beta (34.17, 1.33)) were included. The shape parameters were obtained with beta buster assuming “to be 95% sure that the specificity is greater than 90% with a mode at 99%”. Convergence was assessed by visual inspection of the trace plots and the potential scale reduction factor (Gelman Rubin statistic) being below 1.1. A sensitivity analysis was performed by using different combinations of minimally (dbeta(1,1)) or weakly informative priors (dbeta(2,1)).

Ethical consideration

This research has been accomplished after getting ethical approval from EPHI and AAU (EPHI–IRB-259-2020 and 029/20/Lab (AAU, CHS) respectively). All participants’ identifiers were removed and only codes were used throughout the study to keep confidentiality.

During data and sample collection the data collectors inform each study participant about the purpose and anticipate benefits of the research project and also informed on their full right to refuse, withdraw or completely reject partly or all of their part in the study. Then, we obtained written informed consent from adult study participants and parents or legal guardians of study participants under the age of 18 years to participate in the study and to use their files and records for the purpose of this study.

Furthermore, this work was done accordance with the principles of Helsinki declaration.

Results

Laboratory results of study participants by the three assays

A total of 279 respiratory samples were analyzed using by three assays. The age distribution of the participants ranges from 10 to 85 years with majorities (40%) of the participants were 21–30 years and 144(51.6%) of the participants were female. We have three different populations, of which half of the participants samples 140(50.18%) were from suspect group followed by community surveillance 91(33%) and the remaining from close contacts, Table 3.

Table 3

Socio demographic characteristics of study participants against the three assays in Addis Ababa, Ethiopia, 2021(N = 279), 2021.

Variables	Assay
	TIB		DaAn		BGI
	Neg	Pos	Neg	Pos	Neg	Pos
Sex
Female	115	29	110	34	106	38
Male	114	21	110	25	104	31
Age in Year
<10	6	4	7	3	5	5
11–20	14	13	15	12	13	14
21–30	64	25	69	20	62	27
31–40	51	16	49	18	43	23
41–50	20	12	21	11	18	14
51–60	24	5	22	7	19	10
>60	20	5	18	7	18	7
Reason for testing
Community surveillance	89	2	87	4	84	7
Contact of confirmed case	35	13	32	16	32	16
Suspect	104	34	100	38	93	45

Results with threshold cycle (Ct) values >40 or those that remained undetected during the 45 cycles of the experiment work and a probability of false negatives using these three test assays with a positivity rate of from 18 to 25%. There were 2 more cases were positive by E gene (the screening test of TIB) while become negative by RdRp (the confirmatory one). The results were evaluated in comparison with those of the TIB™, Daan™ COVID-19 Direct assay and the BGI COVID-19 assay as indicated Fig 1 and Table 4.

Fig 1

Comparative analytical sensitivity of the three RT-PCR assays based on Ct values, Addis Ababa, Ethiopia, 2021.

Table 4

Overall results of 279 samples tested by three molecular SARS-CoV-2 detection assays value with corresponding Ct values, Addis Ababa, Ethiopia, 2021.

Variables	TIB MOLBIOL	DaAn	BGI	mean (± SD) Ct value
Positive results
All(ct value)	52 (17.9–39.4)	48 (17.5–39.2)	55 (18.7–39.3)	30.1±5.8
E gen and RdRp
Egenes(ct value)	52(9–38.7)			28.81±6.4
RdRp(ct value)	50(12.4–37.6)			29.54±6.1
N gen and ORF 1
N gen(ct value)		59(13.3–39.2)		31.95±5.6
ORF1(ct value)		60(16.6–39.4)		34.20±5.6
ORF(ct value)			69(10.9–35.5)	28.78±5.7
Negative result	229	219	210
No. of concordant results	50	60	NA
# of discordant results	2	1	NA

NOS nasal/oropharyngeal swab, C threshold cycle, E envelope, RdRP RNA-dependent RNA polymerase, N nucleocapsid, S spike, ORF open reading frame.

The performance of each assay was evaluated and Ct values of 2019-nCoV positive samples ranged from 12.4 to 37.6 and 13.3–38.7 for RdRP and E genes of TIB, 13.3–39.22,16.65–38.91 for N genes and ORF of DaAn and from 10.9–35.5 of ORF of BGI. The value for the ORF gene of BGI (28.78±5.7), which makes the lowest ct value among the average of the positive samples.

The raw data, comprising the frequencies of the eight possible combinations of dichotomized test results of the three RT-PCRS in the three populations are displayed in Table 5A.

Table 5

a. Possible frequencies combination of dichotomized test results of the three assays among three populations in Addis Ababa, Ethiopia, 2021.

b. Measure of Cohen’s kappa coefficient (κ) Value among the three tests kits, Addis Ababa, Ethiopia, 2021.

TIB	Da An	BGI	Population 1	Population 2	Population 3
-	-	-	83	26	81
-	-	+	4	4	14
-	+	-	1	2	8
-	+	+	1	3	2
+	-	-	0	1	2
+	-	+	0	1	4
+	+	-	0	3	3
+	+	+	2	8	26
			Total: 91	Total: 48	Total: 140

We perform a Cohen’s kappa coefficient(k) to see the inter test agreement beyond chance of the three assay using 279 COVID-19 suspected individuals and we have a moderate to substantial level of agreement finding report among the three RT-PCR test kits with a kappa value of 0 .55 to 0.72 as the detail illustrated in Table 5B.

Thus, since none of the currently available COVID-19 tests can be considered as a perfect gold standard with both a perfect sensitivity and specificity, we employed a Bayesian latent class model (BLCM) using Markov Chain Monte Carlo sampling with Just another Gibbs sampler (JAGS) to the clients’ data using three tests in three populations.

Posteriors means and corresponding 95% credible intervals (CrI) of models without any covariance or one covariance between either two sensitivities or two specificities are displayed in Table 6. There was no evidence, based on visual inspection of the histograms and on 95% credible intervals (not including 0), that adding a covariance term led to a better model fit. Convergence was checked by visual inspection of the trace plots and the Gelman-Rubin statistic with all potential scale reduction factors (psrf) being below 1.05. Adding a covariance term between the sensitivities if Tib Molbiol and Da An led to a slight decrease in the sensitivities of all RT-PCRs, whereas adding covariance’s between Tib Molbiol and BGI or between Da An and BGI did not substantially alter the posteriors. Adding covariance’s between the specificities, mildly affected the sensitivities, but the specificities were not affected considerably.

Table 6

Posteriors means and corresponding 95% credible intervals (CrI) of BLCM models with different pair wise covariance’s added, Addis Ababa, Ethiopia,2021.

	Models
	no	Cov.se12	Cov.se13	Cov.se23	Cov.sp12	Cov.sp13	Cov.sp23
Posteriors
Se_TIB	81.2 [65.8;93.8]	79.7 [60.1;94.4]	81.0 [65.4;94.4]	81.0 [66.2;94.2]	78.9 [63.0;93.0]	79.3 [64.0;92.7]	79.9 [63.6;93.5]
Se_Da An	83.6 [71.2;94.0]	82.1 [65.0;94.5]	83.6 [69.9;95.2]	84.0 [71.1;94.3]	80.9 [68.3;92.3]	83.2 [70.8;94.3]	82.7 [70.3;93.6]
Se_BGI	81.0 [68.7;92.0]	80.3 [65.3;92.9]	80.2 [66.1;92.0]	81.0 [68.4;92.0]	80.6 [67.7;91.7]	78.9 [66.4;90.1]	79.3 [66.4;90.7]
Sp_TIB	99.1 [97.3;1]	98.9 [96.8;1]	98.8 [96.8;1]	98.9 [97.0;1]	99.3 [97.6;1]	99.3 [97.7;1]	99.1 [97.4;1]
Sp_Da An	96.1 [93.0;99.1]	95.8 [92.3;99.0]	96.3 [92.9;99.5]	96.3 [92.9;99.3]	95.9 [92.5;99.0]	96.5 [93.2;99.5]	96.6 [93.5;99.7]
SP_BGI	91.6 [87.5;95.2]	92.0 [87.4;96.8]	91.2 [86.9;95.1]	91.7 [87.4;95.6]	92 [87.8;96.0]	91.3 [87.3;95.1]	91.5 [87.4;95.3]
Cov	-	0.008 [-0.023; 0.056]	0.006 [-0.024; 0.045]	-0.006 [-0.035; 0.024]	0.002 [-0.001; 0.008]	0.002 [-0.001; 0.008]	0.001 [-0.003; 0.007]
Prev 1	3.6 [0.6;8.3]	3.7 [0.4;8.6]	3.7 [0.5;8.3]	3.7 [0.5;8.2]	4.3 [0.7;9.4]	4.3 [0.8;9.2]	4.2 [0.8;8.9]
Prev 2	35.4 [21.7;50.7]	35.8 [20.6;51.4]	34.9 [21.2;50.1]	35.1 [21.2;49.9]	36.3 [22.2;51.7]	36.4 [22.3;51.0]	35.9 [21.9;51.3]
Prev 3	28.7 [20.2;38.1]	29.2 [20.0;40.5]	28.6 [20.1;38.0]	28.9 [20.2;38.2]	29.5 [21.2;39.5]	29.3 [20.7;38.5]	29.5 [20.7;39.2]
deviance	80.16	81.32	80.74	80.73	80.36	79.99	80.68
bpv	-	0 [0;1]	0 [0;1]	0 [0;1]	0 [0;1]	0 [0;1]	0 [0;1]

Based on the obtained posteriors for the sensitivities, specificities and prevalence, positive and negative predictive values were determined for each test and in each population as depicted at Table 7.

Table 7

Predictive values of three tests comparing with three different populations for SARS-COV-2 infection in Addis Ababa, Ethiopia, 2021.

Population	Predictive value	TIB Mol Biol	Da An	BGI
Population 1	PPV	0.765	0.450	0.266
Population 2	PPV	0.979	0.923	0.841
Population 3	PPV	0.972	0.897	0.795
Population 1	NPV	0.993	0.994	0.992
Population 2	NPV	0.905	0.915	0.898
Population 3	NPV	0.929	0.936	0.923

*PPV-positive predictive value, NPV- negative Predictive value.

Discussion

High-throughput quality SARS-CoV-2 diagnostic testing is the pivotal factor and has an indispensable role in the control of the current global pandemic [42]. The pandemic is not over. Thus, accurate, reliable and timely laboratory supported testing and management of the pandemic is very much important to curve the infection through swiftly to test and care for patients and to trace their contacts for brining to care. However, this efforts might be hampered by a lack or limited information on the test kit quality diagnostic performance by which have great impact on the testing and containment of the virus [9,10,17].

Accordingly, the current study aimed to evaluate three commercially available one step RT-PCR assays in Ethiopia with an overall diagnostic sensitivity of 0.813(0.658–0.938), 0.836(0.712–0.940) & 0.810(0.687–0.920), and a specificity of 0.991(0.973–1.000), 0.961(0.930–0.991) & 0.916(0.875–0.952) of TIB MOLBIOL, Da An and BGI respectively, which is slightly lower performance than a study done by public health England using BGI test kits of sensitivity 97.4% (86.5 to 99.9%; 95% CI and by Corman et al., as sensitivity of 97.9% [92.8–99.7 by TIB MOLBIOL, Berlin, Germany using E-gene [42,44] and its superior findings than SARS-COV-2 R-GENE (BioMérieux) with 60.2% [49.8–70.0] to 66.3% [56.1–75.6]) and combined with those for N or the RdRp genes (71.4% [61.4–80.1] and 69.4% [59.3–78.3], respectively) reported by Alcoba et al. and it’s concordance finding with in positive case of BGI of SARS-CoV-2 in Australia showed high diagnostic power in detecting SARS-CoV-2 with a positive percent agreement of 88.89% (95% CI: 83.4%, 94.3%). the main difference might be study time and sampling of the presumptive cases too. More over all of these studies try to present their finding against the classical “gold standard” approach. However, we used an innovative approach with the Bayesian latent class model (BLCM), which is recently recommended to see the diagnostic tests accuracies in the absence of a perfect gold standard [43-45].

The current results showed a moderate to substantial level of agreement of the inter test agreement beyond chance of the three assays with an average Cohen’s kappa value of 0 .55 to 0.72. Which is an inter performance agreement a measure of 0.72(95% CI(0.61–0.82)) among TIB and DaAn assays, 0.61 (95% CI(0.49-.0.72)) b/n TIB and BGI with substantial agreement where as a moderate agreement were reported among DaAn and BGI assays with a kappa value of 0.55(95% CI (0.44–0.67)) via 279 COVID-19 suspected individuals, which was consistent with similar studies(27,51,52) and lower finding compared with Altamimi, Asmaa M., et. al., the difference might be the defined study population [46].

From the current study, we learnt that, TIB and Da An assays correctly identified all negative samples while two assays each failed to correctly identify one different negative sample. GI has been presented in low specificity. The consistent detection of positive samples at different Ct values gives an indication of when to repeat testing and/or establish more stringent in-house cut-off value. The varied performance of different diagnostic assays, mostly with emergency use approvals, for a novel virus is expected. This comparative assay’ performance findings may guide country laboratories to determine both their repeat testing Ct range and/or cut-off value [43,47].

Notably, all professional and pharmaceutical, diagnostic developer should take their maximum capacities to get the best quality diagnostic assay to flatten the curve of COVID-19. Likewise, RT-PCR is the current established gold standard approaches for the diagnosis of COVID-19, but it has its own limitation in high false negativity rate. This is more or less concordance with our study findings. Thus, in light of this, further action should be considered to get the most sensitive and specific test kits, that can minimize the false positive and false negative cases in to insignificant level [43,47,48].

We noted that the positivity proportion of these three assays were higher among the age group of 21-30years of the presumptive, and there is no gender specific difference among the participants, which is similar and concordance with prior studies [25,33]. Likewise, the evaluated assays sensitivity diagnostic performances have been improved its levels of sensitivity improved to those of all other kits when one primer-probe combined with other targeted genes, but it takes more time to repeat for confirming the screening test with more sensitive. Thus, one touch step assays of combining different targets should be considered to have more competitive with cost effectiveness and benefit from combining the results of more than one primer probe set [22,27,31].

In the current study, we are at most the highest level of care and rightfully controlled procedure from the pre analytical to post analytical, which will be the contributing factors for the quality performance difference. Given that, we used similar analysis, similar testing time, similar personnel, sample collection strategy.Moreover we try to use similar written standard procedure, which is all assay solutions were uniformly affected. Thus, the differences in diagnostic sensitivity among the evaluated test kits were purely related to their different components. This finding will give a highlight which target gen is more sensitive while selecting a gene target, quality control, primers and give a clue for pharmaceuticals how to choose alternative test kits for the diagnosis of COVID-19 and that minimizes misdiagnoses of an active SARS-CoV-2 infection [29,49].

The current global pandemic is in a way of highly genomic variation and changing its primary nature that might have a diagnostic variability performance and RT-PCR assay have not yet documented gold standards for their comparative analysis. Accordingly, we followed the STARD-BLCM guidelines by which specificity robust estimates independent of conditional dependencies, fewer data to estimate sensitivity which leads less robust. Prevalence estimates are in line with having a higher prevalence in contact cases and suspects as of the study populations, the predictive values were found influenced by prevalence’s. In case of TIB MOLBIOL; there were 2 discordant cases of the primary screening E gene based targeted test compared with a confirmatory of RdRp and also its time taking and resource intensive while repeating those discordant results. On top of this, there might be more false negative clients with Da An and TIB MOLBIOL comparing to BGI; which might be a source of infection in the community as false negative diagnosis [43,44,47].

We report a higher ct value in case of DaAn (31.95±5.6 (in case of N gen & 34.20±5.6 in case of ORF1) and lower ct value in BGI with a mean (± SD) Ct value of 28.78±5.7 by ORF1 of BGI among the positive cases. Higher ct value mean it needs more resources and time, while smaller ct value mean, positive result with in shorter turnaround time(TAT) and cycle of PCR reaction that can be more efficient than the others. In case of DaAn, in case of inconclusive test results and situations like ct value of the test sample is NMT 40 only in one channel b/n FAM and VIC, and no amplification curve in the other channel, the manufacturer recommend to repeat the test.if the result of the retest consist with the original, it can be determined as positive for SARS-CoV-2; if the result of the retest is negative, it can be determined as negative. This leads more strenuous and more tiring among the professional working in this lab that might be pushing looking for and searching more recommend user friendly and single touch/step test kits in the era of highly demanding laboratory results with in short around time [20,32,33,36].

Among the evaluated assays the laboratory try to perform the SARS-CoV-2 diagnosis based on the availability of the test kits. Among these TIB MOLBIOL urge the test to run two times as a manner of screening by using Egen and confirmatory based RdRp, the final verdict of the test judge based on the RdRp findings and in case of DaAn the final diagnostic result should be based on the agreement of Ngen and ORF. While in case of BGI the final decision of the presumptive to have positive or negative based on only and only one test using ORF based. From these we understand that the former two tests need more time and resources to finalize the result with much lever intensive and not user friendly, which makes significant difference and BGI be more upper weigh and be superior, that concur with different similar studies [35,36,46,47].

In the other hand, we get a chance of feedback about the supply chain management system from the professionals and try to test out the reagent supply in light of sustainable laboratory reagents and consumables have been a pillar in the global response to the COVID-19 pandemic. In this regard, we learnt that the testing laboratories were suffering a lot with erratic supply chain management system.However, this should be expatiated sustainable supplies will go on with to be a key tool for controlling COVID-19 and regress effort by supply chain managers, policy makers and laboratory professionals [31,43,46].

Limitation of the study

We noted that possibly not enough data to estimate conditional dependencies between sensitivities which is minor differences in posteriors. Due to high estimates for specificities, conditional dependencies between specificities are implausible. Thus, readers should take in to consideration while inferring our findings.

Conclusion and recommendation

Based on the current research findings, we wind up as Tib Mol Biol performs best, followed by DaAn: the sensitivity of all tests are rather similar, but there are differences with regard to specificity. So for issues and costs associated with false positive, one should go for TIb Mol Biol.; and we understand that these assays are helpful tools for the routine diagnosis of COVID-19 in resource limited settings and could be implemented accordingly considering their prioritization contexts.

Despite the incredible efforts have been made to control the pandemic a global crisis has been aggressively going on. To contain this calamity, based on the recent study finding, the following points should be considered. Large scale study should be conducted and there might be the genetic mutation of the circulating virus during our sampling that might be the source of the performance variation, possibly false negative. Thus, the research team highly recommends the whole genome sequencing to know the circulating variant/strain and to come up a conclusive justification.

Supplementary material used for the BLCM code for analysis, Addis Ababa, Ethiopia, 2021.

(R)

Click here for additional data file.

(DOCX)

Click here for additional data file.

26 Nov 2021

PONE-D-21-35232Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settingsPLOS ONE

Dear Mr Abay Sisay,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by 10 December 2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Basant Giri, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3.  We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

4. Thank you for stating in your Funding Statement:

(This work was partially financed by Addis Ababa University through adaptive research and problem solving project, Ref #-PR/5.15/590/12/20. However, the University had not any role in the research methodology design, data collection, analysis and write up of the manuscript. The contents are purely the responsibilities of the authors and did not represent and reflect the view of the funder.)

Please provide an amended statement that declares *all* the funding or sources of support (whether external or internal to your organization) received during this study, as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now.  Please also include the statement “There was no additional external funding received for this study.” in your updated Funding Statement.

Please include your amended Funding Statement within your cover letter. We will change the online submission form on your behalf.

5.  Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

6.  We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 7 in your text; if accepted, production will need this reference to link the reader to the Table.

7. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

8. Upon re-submitting your revised manuscript, please upload your Supplementary file as Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover study’s minimal underlying data set as letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a decent study that evaluates the diagnostic performance of 3 available RT-PCR assays in resource-limited setting in Ethiopia. Experiments and statistical analyses are performed to a high technical standard and are described well. However, major concern with this research is the sample size – is it sufficient enough to draw conclusions on their performance evaluation?

Below are some other comments that must be addressed before it can be published.

Major comments:

1. There are other assays available in Ethiopia. Why are only three of them selected in this study? There should a separate sub-section under Experiments which should clearly explain the rational behind the selection of those assays.

2. In the data quality assurance section, it is mentioned that the assays were performed by following standard operating procedures and is against specific manufacturing recommendations. What does it mean? Does it mean that, all the assays were performed by standard operating procedure and not the manufacturers protocol? If so, what is the rationale behind doing this? If specific manufacturers protocol is not followed, how can we say that the result is from that particular assay? Is the result now trustworthy?

3. The details about the assays must be mentioned at least in SI in detail. The details of assay components, for example, lot#, expiry date, date of analysis etc. must be given.

It is also mentioned that the quality of assay reagents was checked? How was it done; it needs to be mentioned.

4. From figure 1, the threshold cycle for BGI assay is very less compared to others. Significance test must be done between these three. From this data, can we say that BGI assay is sensitive over others? But the BLCM analysis looks contradicting – BGI being the least sensitive. It is important to discuss figure 1 in the main text.

5. More importantly, the conclusions/recommendation about the usage of assays is missing in this research. The authors, after this research, what do they recommend for the reliable SARS-CoV-2 detection? Rather than mentioning the problems only and suggesting the alternative, it is important to recommend something out of this research, otherwise it is ineffective.

Minor comments:

1. Line 42: efforts made

2. Line 45 and 47: not well documented in resource-limited setting, then why is it performed in clinical setting? These two settings do not really line-up well here.

3. Line 178 and 179: inconsistencies in numbering population

4. Line 268: cut-off

5. Table 3 is difficult to follow.

Reviewer #2: Summary of the research and overall impression:

This study is really admirable for the effective diagnosis and control prospective in this current pandemic situation in the resource constraint setting as we are also engaging in similar COVID-19 diagnosis activities despite bearing countless challenges and biohazard risks.

This manuscript aimed to investigate different diagnostic assays available in Ethiopia’s clinical setting using Bayesian latent class model, as PCR is not considered as gold standard. Although, research gap is explained thoroughly, the scientific hypothesis has not been addressed. Manual RNA extraction was conducted, master-mix preparation, amplification and results were analysed as indicated by the manufacturer. The technical aspects of the paper are adequately conveyed and the BLCM method is appropriate to answer the question.

Specific areas for improvement:

Major issue:

1. Please clarify what the author wants to claim in the fourth paragraph. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 9]

2. Perhaps, explain what FIND does to provide more insight, as it is used to evaluate diagnostic tests commercially available. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 23, Fifth paragraph]

3. No clear indication on critical, timely and containment strategy explanation. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 26, Sixth paragraph]

4. TIB targets E and RdRp genes, Da An targets N and ORF1ab genes, and BGI targets ORF 1ab gene. It has clear indication that these three different assays target different genes, please mention how these can be correlated to compare and predict its specificity. As various publications have already claimed that different target genes have different expression levels and detectable range. [Section: Available diagnostic platform for the diagnosis of COVID-19, Last paragraph]

5. The sentence, “The quality of each reagent was checked before the actual laboratory analysis.”, needs a better explanation. What does the word ‘quality’ indicate? How was the quality checked? [Section: Data Quality Assurance, First paragraph]

6. As BLCM assigns individuals to classes based on their probability of being in classes given the pattern of scores they have on indicator variables. Proper class assignment is not guaranteed. Also, because class assignment is based on probabilities, the exact number or percentage of sample members within each class cannot be determined. Please, briefly explain how this study assigns samples to different classes. [Section: Bayesian latent class model (BLCM)]

7. The discussion is well-written, although it could benefit from including other relevant studies/prior works to support your results. Likewise, few parts of discussion are already stated in the introduction, so it would be better if care be taken for making it concise and readable. Also, if possible please break down the reference and include it in-text. (provide reference at the end of sentence rather than at the end of each paragraph) [Section: Discussion]

8. Conclusion is relatively weak. Please, consider revising it and sufficiently summarizing the methods and results.

9. While the study appears to be sound, the language is unclear, making it difficult to follow. I advise the authors to work with a writing coach or copyeditor to improve the flow and readability of the text.

Minor issues:

1. The authors have acquired ethical approval from their respective boards but haven’t mentioned if they have also acquired the informed (verbal or written) consent from the patients themselves.

2. The manuscript should rigorously undergo for plagiarism check.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

4 Dec 2021

Date: 04 December, 2021

To: PLOS ONE Journal editorial office

Dear Editor:

We are very much glad to write this response that our research output manuscript entitled “Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settings, with a manuscript reference number PONE-D-21-35232” has been possible considered for publication in PLOS ONE. We are pleased to have an opportunity to make our manuscript revised and we have greatly appreciated for the reviewers’ high level comments, and suggestions were very helpful for the overall improvement of the manuscript. In revising the manuscript, we have carefully considered reviewers’ comments, academic editor and suggestions on our revised submission.

Moreover, this work was financed by Addis Ababa University through adaptive research and problem solving project, Ref #-PR/5.15/590/12/20. However, the funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The contents are purely the responsibilities of the authors and did not represent and reflect the view of the founder. There was no additional external funding received for this study.

As instructed, we have attempted to succinctly explain changes made in reaction to all comments and reply to each comment in point-by-point fashion as appended here

Response to Academic Editor Comments/Journal requirements

1. Please ensure that your manuscript meets PLOS ONE's style requirements

Response-Noted

2. About Ethics and Consents

Response- This research has been accomplished after getting ethical approval from EPHI and AAU (EPHI–IRB-259-2020 and 029/20/Lab (AAU, CHS) respectively). All participants’ identifiers were removed and only codes were used throughout the study to keep confidentiality.

During data and sample collection the data collectors inform each study participant about the purpose and anticipate benefits of the research project and also informed on their full right to refuse, withdraw or completely reject partly or all of their part in the study. Then, we obtained written informed consent from adult study participants and parents or legal guardians of study participants under the age of 18 years to participate in the study and to use their files and records for the purpose of this study.

Furthermore, this work was done accordance with the principles of Helsinki declaration.

3. Grant information------

Response- Revised accordingly, as

This work was financed by Addis Ababa University through adaptive research and problem solving project, Ref #-PR/5.15/590/12/20. However, the funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The contents are purely the responsibilities of the authors and did not represent and reflect the view of the founder. There was no additional external funding received for this study.

6. about ethics statement –

Response- revised and done accordingly. Please have a look the revised manuscript.

7. about table 7-

Response- revised and done accordingly. Please have a look the revised manuscript.

8. Supporting Information files-

Response- revised and done accordingly. Please have a look the revised manuscript.

Reviewer #1

XXXX-However, major concern with this research is the sample size – is it sufficient enough to draw conclusions on their performance evaluation?

Response- Sample size is very much important for inferring the result for the general population. Accordingly, we have tried to perform our best. We run 3 plates of 96 reagents with their quality control. And there are many published and recommended of similar studies for this type of experiments, even less than this sample size. Most of the experiments recommended sample size is greater than 100. Please refer the following published works and CLSI protocols, as an example…Clin Chem. 1999 Jun; 45(6 Pt 1):882-94. PMID: 10351998, CLSI guide CLSI EP12-A2.

1. There are other assays available in Ethiopia. Why are only three of them selected in this study? There should a separate sub-section under Experiments which should clearly explain the rationale behind the selection of those assays.

Response- Of course, there were others, but during our study these were the widely available and used commercial available one step RT-PCR tests in the country , as already explained page 6 line number 155 of the submitted manuscript.

2. In the data quality assurance section, it is mentioned that the assays were performed by following standard operating procedures and is against specific manufacturing recommendations. What does it mean? Does it mean that, all the assays were performed by standard operating procedure and not the manufacturer’s protocol? If so, what is the rationale behind doing this? If specific manufacturer’s protocol is not followed, how can we say that the result is from that particular assay? Is the result now trustworthy?

Response- In the ISO standards working on the quality aspects..” Against the …” mean as per the recommendation protocols, which means comparing and contrasting that particular recommendations. It has a synonyms of compared to. In our cases we used the specific recommended SOPs and manufacture recommendations and protocols, then we have compare and contrast accordingly.

Just to avoid misleading and misinterpretation we have rephrased it as … according to…. Please have a look the revised manuscript line 270.

3. The details about the assays must be mentioned at least in SI in detail. The details of assay components, for example, lot#, expiry date, date of analysis etc. must be given.

It is also mentioned that the quality of assay reagents was checked? How was it done; it needs to be mentioned.

Response-for this comments, we understand that, It’s already existed at the table 1, if any one of the readers need to understand about each assays, the can cross refer the listed references at the column of the table. I don’t know, if we exostively list those things, the document could be not attractive or the fellow readers.

4. From figure 1, the threshold cycle for BGI assay is very less compared to others. Significance test must be done between these three. From this data, can we say that BGI assay is sensitive over others? But the BLCM analysis looks contradicting – BGI being the least sensitive. It is important to discuss figure 1 in the main text.

Response-The assays or the three different RT- PCRs have different company production with different targets, and that their CT values were also different and their interpretation is quite different as per their experimental recommendation of respective manufacturer. Thus, they should not be the same. That’s why we run the BLCM.

5. More importantly, the conclusions/recommendation about the usage of assays is missing in this research. The authors, after this research, what do they recommend for the reliable SARS-CoV-2 detection? Rather than mentioning the problems only and suggesting the alternative, it is important to recommend something out of this research, otherwise it is ineffective.

Response- We agree with the reviewer’s comment. The ultimate goal of our analysis is to propose the most sensitive and specific test for the detection of SARS Cov2. In addition, based on a population level, to the results of the BLCM, i.e., having a robust estimation of the performance of the tests under field conditions, can be used to adjust the apparent prevalence by for example the Rogan Gladon approach. Accordingly, we revised our manuscript, and please have a look it, at line 507.

6. Line 42: efforts made

Response- done accordingly

7. Line 45 and 47: not well documented in resource-limited setting, then why is it performed in clinical setting? These two settings do not really line-up well here.

Response- clinical setting Vs resource. These two settings have not line up.

Our study was done in clinical settings of resource limited settings of less/developing countries. Thus, resource limited settings means which is more related to the scarcity of economy. Or it mean as resource poor or constrained setting ,whereas clinical settings means a hospital, department, outpatient facility, or clinic, or laboratory, public health institutes … etc which health cares were rendered.

8. Line 178 and 179: inconsistencies in numbering population

Response: the three study populations were selected conveniently and categorized as where they come until the final proposed sample sizes were full filled.

9. Line 268: cut-off

Response- done accordingly

10. Table 3 is difficult to follow.

Response-it’s just giving an additional information on the demographic of the study participants considering our readers get in-depth understanding on it.

Reviewer #2

Xxxx”… the scientific hypothesis has not been addressed…”

Response: Our main scientific hypothesis is the assessment of comparing the ct levels and then the demographic data, and finally assessing agreement between the PCRs and obtaining robust estimates for diagnostic sensitivities and specificities in the less developed countries.

Revised accordingly and please have a look the revised manuscript at line 165-171.

1. Please clarify what the author wants to claim in the fourth paragraph. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 9]

Response- We are here for trying to explaining countries specific testing strategies: some countries’ recommend and implement COVID-19 testing, for instance in our country, Ethiopia the recommended test for the diagnosis of COVID-19 is RT-PCR.

2. Perhaps, explain what FIND does to provide more insight, as it is used to evaluate diagnostic tests commercially available. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 23, Fifth paragraph]

Response- In the recent pandemic, COVID-19, FIND has been accomplishing a milestone and commendable activities by independent evaluations of assays for SARS-CoV-2 diagnosis in collaboration with WHO and summarizes data for emergency Use Authorization (EUA) and the recommend specific country has to be perform validation/verification accordingly prior to use it.

Addressed accordingly- and this sentences were included in the revised manuscripts. Please have a look it, line 145.

3. No clear indication on critical, timely and containment strategy explanation. [Section: Available diagnostic platform for the diagnosis of COVID-19, Reference no.: 26, Sixth paragraph]

Response- I think, this information has been already explained in the same section, line 133-138, no need of repetition.

4. TIB targets E and RdRp genes, Da An targets N and ORF1ab genes, and BGI targets ORF 1ab gene. It has clear indication that these three different assays target different genes, please mention how these can be correlated to compare and predict its specificity. As various publications have already claimed that different target genes have different expression levels and detectable range. [Section: Available diagnostic platform for the diagnosis of COVID-19, Last paragraph]

Response- Sure!, they have different targets and different ranges of detection. However these assays used in Ethiopia and other countries in these states .Thus, this is one of the rational of doing this study to come up for some scientific justification of their performance specifications. Accordingly, we use BLCM as a tiebreaker and a bridge to minimize the gaps among the assays

5. The sentence, “The quality of each reagent was checked before the actual laboratory analysis.”, needs a better explanation. What does the word ‘quality’ indicate? How was the quality checked? [Section: Data Quality Assurance, First paragraph]

Response- The quality of each reagent: lot number, expire date, storage conditions, physical leak proof ness and not breakages, etc were checked before the actual laboratory analysis., and also the positive and negative controls were incorporated in all runs.” This sentences is incorporated in the revised manuscripts for a better explanation for our readers, line 269-274.

6. As BLCM assigns individuals to classes based on their probability of being in classes given the pattern of scores they have on indicator variables. Proper class assignment is not guaranteed. Also, because class assignment is based on probabilities, the exact number or percentage of sample members within each class cannot be determined. Please, briefly explain how this study assigns samples to different classes. [Section: Bayesian latent class model (BLCM)]

Response-: In our analysis we followed the Hui Walter approach, thus we modelled the eight different combinations of dichotomized test results (+++, ++-, +-+, +--, -++, -+-, --+, ---) in the three different populations with a multinomial distribution. Here, we assume that there are two classes (negatives and positives).

Please have a look at the following to illustrate the principle of BLCM (for simplicity just for two tests, thus with four possible combination of dichotomized test results).

T1+T2+: P1*Se1*Se2+(1-P1)*(1-Sp1)*(1-Sp2)

T1+T2-: P1*Se1*(1-Se2)+(1-P1)*(1-Sp1)*Sp2

T1-T2+: P1*(1-Se1)*Se2+(1-P1)*Sp1*(1-Sp2)

T1-T2-: P1*(1-Se1)*(1-Se2)+(1-P1)*Sp1*Sp2

Thus, with P1 being the prevalence, Se1 and Se2, the sensitivities of both tests, Sp1 and Sp2, the two specificities and T1+/- and T2+/- the dichotomized test results, we assume that there are two “classes” the true positives and the true negatives. For example, among the samples which tested positive for both tests (T1+T2+) we have the true positives (TP), equal to the product of the prevalence and both sensitivities, and the false positives (FP), equal to the product of (1-P) times (1-Sp test 1) and (1-Sp test2). Here we have five unknowns (P and both sensitivities and specificities) but just four equations (and if the total number of n submitted to two tests is known, the fourth frequency of the two possible test combination becomes known, so we just have three degrees of freedom). In this situation the model is not identifiable. Hui and Walter (Hui SL, Walter SD. Estimating the error rates of diagnostic tests. Biometrics. 1980 Mar;36(1):167-71. PMID: 7370371) proposed a solution, by adding a second population from which samples are also analysed by two tests, assuming that the two tests have the same sensitivities and specificities in both populations. This then would add one additional unknown parameter (the prevalence of the second population (so in total six unknown parameters), but also three additional degrees of freedom. So in total the model becomes identifiable. Hui and Walter have shown that if the following condition for S (the number of populations), and T (the number of tests is fulfilled), the model is identifiable

In our case we have three tests and three populations. The aim is to obtain reliable estimates of se and sp, the aim is not to determine class membership of each individual (being truly positive or negative).

This type of Bayesian latent class analysis (also called no gold standard models) is different from the latent class approach where - based on specific patterns of the data or the underlying latent structure) individuals are assigned to classes and class membership could then be used as a variable for further analysis. In this context one would looks with specific metrics if the data are best describes by 1 to n classes).

In our approach of BLCM we have performed a sensitivity analysis with different priors to assess the importance of the prior information.,

7. The discussion is well-written, although it could benefit from including other relevant studies/prior works to support your results. Likewise, few parts of discussion are already stated in the introduction, so it would be better if care be taken for making it concise and readable. Also, if possible please break down the reference and include it in-text. (provide reference at the end of sentence rather than at the end of each paragraph) [Section: Discussion].

Response-Done and revised accordingly. Please have a look the revised manuscript.

8. Conclusion is relatively weak. Please, consider revising it and sufficiently summarizing the methods and results.

Response- Addressed and revised accordingly

9. While the study appears to be sound, the language is unclear, making it difficult to follow. I advise the authors to work with a writing coach or copyeditor to improve the flow and readability of the text.

Response-To be honest, we tried our best, in this regard, and we did grammatical review in the revised manuscripts. The comment is accepted positively and revised accordingly, it’s an opportunity for the suitability and readability of our manuscript. Please have a look the revised manuscript.

Minor issues:

1. The authors have acquired ethical approval from their respective boards but haven’t mentioned if they have also acquired the informed (verbal or written) consent from the patients themselves.

Response-We did this work after getting the approval of institutional ethics board and we have getting written consent and assent of the study participants….It’s already addressed in the editor comments section.

2. The manuscript should rigorously undergo for plagiarism check.

Response- Thank you very much for your detailed and rigorous review of the manuscript, it helps for the improvement of the manuscript. And we did and write our manuscript using our own words. Hope the revised manuscript get better improvement accordingly.

Looking forward to hearing from you and thank you very much again for your consideration!

Sincerely,

Abay

Submitted filename: Abay Response for the Reviewer_F4.docx

Click here for additional data file.

19 Dec 2021

Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settings

PONE-D-21-35232R1

Dear Dr. Sisay,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Basant Giri, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All the comments have been addressed. This article can be considered for publication after minor revisions.

Reviewer #2: Went through all the rebuttals provided by authors in the revised manuscript, I found that the authors have given justifiable reasons and explanations for each major and minor issues. Moreover, they have made necessary revisions to the manuscript and are considerable.

Though, the conclusion is still comparatively weaker compared to the respective result/discussion sections and hence recommend to make it scientifically strong. Also, provide the take home message and contributions in the frontiers of molecular diagnostics.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

23 Dec 2021

PONE-D-21-35232R1

Diagnostic accuracy of three commercially available one step RT-PCR assays for the detection of SARS-CoV-2 in resource limited settings

Dear Dr. Sisay:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Basant Giri

Academic Editor

PLOS ONE