Am I Positive? Improving Human Immunodeficiency Virus Testing in the Era of Preexposure Prophylaxis and Immediate Antiretroviral Therapy Using Machine Learning.

Background: Human immunodeficiency virus (HIV) testing is the first step in the HIV prevention cascade. The Centers for Disease Control and Prevention HIV laboratory diagnostic testing algorithm was developed before preexposure prophylaxis (PrEP) and immediate antiretroviral therapy (iART) became standards of care. PrEP and iART have been shown to delay antibody development and affect the performance of screening HIV assays. Quantitative results from fourth-generation HIV testing may be helpful to disambiguate HIV testing.
Methods: We retrospectively reviewed 38 850 results obtained at an urban, academic medical center. We assessed signal-to-cutoff (s/co) distribution among positive and negative tests, in patients engaged and not engaged in an HIV prevention program, and evaluated changes in patients with multiple results. Classification and regression tree (CART) analysis was used to determine a threshold cutoff, and logistic regression was used to identify predictors of true positive tests.
Results: Ninety-seven percent of patients with a negative HIV test had a result that was ≤0.2 s/co. For patients tested more than once, we found differences in s/co values did not exceed 0.2 s/co for 99.2% of results. CART identified an s/co value, 38.78, that in logistic regression on a unique validation cohort remained associated with the likelihood of a true-positive HIV result (odds ratio, 2.49). Conclusions: Machine-learning methods may be used to improve HIV screening by automating and improving interpretations, incorporating them into robust algorithms, and improving disease prediction. Further investigation is warranted to confirm if s/co values combined with a patient's risk profile will allow for better clinical decision making for individuals on PrEP or eligible for iART.

Human immunodeficiency virus (HIV) testing is the critical first step in the HIV prevention cascade [1]. HIV testing is required for both primary prevention with preexposure prophylaxis (PrEP) and secondary prevention with treatment as prevention, both of which are central to efforts to end the HIV epidemic [2]. Current HIV treatment guidelines recommend immediate antiretroviral therapy (iART; same-day antiretroviral therapy [ART] and more broadly rapid ART) or starting as soon as possible after an HIV diagnosis [3]. Current HIV prevention guidelines require regular HIV testing. The Centers for Disease Control and Prevention’s (CDC) updated recommendations for laboratory testing for the diagnosis of HIV infection algorithm recommends HIV screening with an initial fourth-generation HIV antigen/antibody (Ag/Ab) combination immunoassay, followed by HIV-1/2 differentiation immunoassay if reactive [4]. A nucleic acid amplification test is recommended if the differentiation assay is indeterminate or inconclusive [4]. However, the CDC developed the current HIV laboratory diagnostic testing algorithm before the widespread use of iART and PrEP [5]. The recently released “Preexposure Prophylaxis for the Prevention of HIV Infection In the United States—2021 Update” clinical practice guideline now recommends HIV RNA testing, in addition to HIV Ag/Ab testing, for some patients starting and all patients taking antiretrovirals for PrEP [6]. This recommendation comes with implementation challenges and at a significantly increased cost. A comprehensive evaluation of how iART and biomedical HIV prevention could impact the current HIV testing algorithm is needed.

Fiebig et al described the natural history of HIV infection and immunopathogenesis as 6 discrete stages [7]. Contemporary HIV testing was designed around this natural history, taking advantage of the p24 positivity that occurs with the rise in viral load seen in stage II and early HIV antibodies seen in stage III, about 10 and 14 days after infection. A commonly used fourth-generation HIV test is the Abbott Architect HIV Ag/Ab Combo, a chemiluminescent microparticle immunoassay. The platform measures light intensity (in relative units) for which a relationship exists between the amount of HIV antigen and antibodies in the sample. The result is determined by comparing the chemiluminescent signal in the reaction to a cutoff signal determined during calibration. Samples with a signal-to-cutoff ratio (s/co) >1.0 are considered reactive.

The early initiation of ART during acute HIV has been shown to lead to HIV-specific antibodies failing to develop or disappearing. Similarly, multiple case reports have demonstrated that patients who acquire HIV while adherent to PrEP may have low viral loads and ambiguous HIV test results [8-14]. In a study of rhesus macaques utilizing a simian HIV challenge, macaques receiving PrEP had lower peak viral loads and delayed antibody maturation but no change with regard to the timing of seroconversion [15]. In the HIV Prevention Trials Network ADAPT study (Use of Emtricitabine and Tenofovir Disoproxil Fumarate for Pre-Exposure Prophylaxis [PrEP] [ADAPT]), 50% of patients with acute infection at the first visit had a viral load below the limit of quantification. Furthermore, in cases where PrEP was continued for 3–4 months after infection, RNA levels dropped below the level of detection, and s/co ratios were low [16]. Additional studies show that the use of PrEP can delay the maturation of HIV antibody responses [17]. The Association of Public Health Laboratories reporting language acknowledges that “there is insufficient data regarding the performance of the algorithm and any potential effects of preexposure prophylaxis” [5].

To better understand the potential utility of using quantitative s/co as part of HIV testing, we performed a retrospective analysis of HIV testing done at an urban academic medical center to compare the distribution of s/co ratios for those engaged and not engaged in a comprehensive HIV prevention programs and describe how s/co ratios change over time. We used machine learning to stratify disease predictors and select cutoffs that would identify true-positive patients with greater likelihood. These results have potential implications for HIV testing for individuals on PrEP as well as the application of iART to broad populations during efforts to end the HIV epidemic.

METHODS

Study Design

We conducted a cross-sectional study to assess the association between s/co and the primary outcome of being HIV positive as confirmed by confirmatory HIV testing.

Setting and Patient Population

This study was performed at a large urban academic medical center consisting of 3 inpatient hospitals and multiple ambulatory care service sites in northern Manhattan, New York City. The program serves an area with an increased HIV incidence (31 per 100 000 people) and economic disadvantage (20% of households below the federal poverty line, 17% going without needed medical care). The institution also supports a robust sexual health and HIV prevention program (HPP). Patients engaged in the program are prescribed PrEP based on New York State and CDC guidelines and, after starting, have a 1-month follow-up visit with subsequent visits every 3 months. As patients may take PrEP on demand and frequent starts and stops are common, adherence to PrEP is not measured. However, engagement at the first follow-up visit is 68%, dropping to 35% by the third follow-up visit. While these numbers are low, at least one-third of patients reengage at a later date [18].

Training Cohort

We included all patients receiving a fourth-generation HIV Ab/Ag screen test during 1 January–26 November 2018; test results from 24 May through 15 August 2018 were unavailable.

Validation Cohort

We included all patients receiving a fourth-generation HIV Ab/Ag screening test from 1 November 2021 through 31 January 2022.

Data Collection

Data for eligible patients were extracted retrospectively from the electronic medical record (EMR) and augmented with manual chart review to resolve inconsistencies or complete missing data. Electronically extracted EMR data included age, sex at birth as recorded in the EMR, date of birth, HIV test dates, and HIV test results. The s/co ratios were obtained from the clinical laboratory. Engagement in the HPP was obtained from the HPP database and filtered for patients who started biomedical HIV prevention. All data were merged and cleaned using RStudio version 1.4.1106 software.

Outcomes

True-positive tests were tests in which the HIV Ab/Ag test was positive, and either the HIV-1/2 differentiation assay or HIV RNA assay was positive [5]. False-positive tests were tests in which the HIV Ab/Ag test was positive, but both the HIV-1/2 differentiation assay and HIV viral load assay were nonpositive (HIV-1/2 differentiation assay results may be negative or indeterminate). In addition to ancillary testing, the medical records of all patients with an s/co >0.5 were manually reviewed. No suspected false-negative results were identified, and all true-positive results were confirmed.

Statistical Analysis

We initially performed a descriptive analysis to describe the distribution of s/co ratios and change in s/co ratio over time for individuals with multiple HIV tests. To assess changes over time, we identified individuals for whom we had >1 sample collected and tested in this period and calculated days between visits and changes in s/co.

We used classification and regression tree (CART) analysis, a nonparametric supervised machine-learning approach, to estimate a threshold level for s/co. The full tree was grown using information from the Gini index in the training cohort. Regression tree analyses were conducted with complete set and missing values assigned using the “popular node” option to split nodes. Because the tree split at the first node and the frequency in the terminal one of the leaves was <5 (ie, the smallest tree), we, pre-pruned by conducting cross-validation and chose the tree with the model with complexity parameter corresponding to the lowest misclassification error to avoid overfitting [19]. Continuous (age, s/co) and categorical (sex, participation in the HPP) variables were included as predictor pruning via cost complexity criterion was not required.

Univariate regression with true positivity as the primary outcome was performed on the training and validation cohorts. Features with P values ≤ .05 in the univariate analysis were included in a multivariate logistic model performed on both the combined cohort (training and validation) and the validation cohort. All analyses were conducted using R version 4.1.2 with RStudio version 1.4.1106 software [20]. The rpart package was used for CART analysis, the gtsummary package for univariate analysis, and the stats package (glm) for logistic regression.

The institutional review board at Columbia University Irving Medical Center approved this protocol under an expedited review without a requirement for individual patient consent.

RESULTS

Description of Clinical Characteristics in the Training and Validation Cohorts

A total of 38 850 test results from 35 500 patients were reviewed; 25 297 (71%) females were tested, and 1105 patients were engaged in HIV prevention services. A total of 38 559 results were true negatives, and no false negatives were identified (sensitivity, 100%; negative predictive value [NPV], 100%). Two hundred twenty-nine results from 209 patients were true-positive results, while 62 (21%) results from 48 patients were false-positive results (positive predictive value [PPV], 79%; specificity, 99.8%). Compared to the training cohort, the validation cohort was older (median age, 32 vs 34 years) and less likely to be engaged in the HPP (4.8% vs 3.5%) (Table 1).

Table 1.

Demographics of the Training and Validation Cohorts

Characteristic	Patients in Training and Validation Cohorts				Tests in Training and Validation Cohorts				Positive Tests in Training and Validation Cohorts
	Overall(n = 35 500)	Training Cohort(n = 24 830)	Validation Cohort(n = 10 670)	P Value[a]	Overall(n = 38 850)	Training Cohort(n = 27 594)	Validation Cohort(n = 11 256)	P Value[a]	Overall(n = 291)	Training Cohort(n = 1941)	Validation Cohort(n = 971)	P Value[b]
Age, y, median (IQR)	33 (25–44)	32 (25–43)	34 (27–46)	<.001	32 (25–43)	32 (25–42)	34 (26–46)	<.001	36 (29–54)	35 (29–53)	40 (31–56)	.2
Sex recorded at birth				.2				.3				>.9
Female	25 297 (71)	17 739 (71)	7558 (71)		27 376 (70)	19 488 (71)	7888 (70)		89 (31)	59 (30)	30 (31)
Male	10 203 (29)	7091 (29)	3112 (29)		11 474 (30)	8106 (29)	3368 (30)		202 (69)	135 (70)	67 (69)
s/co category				.4				.3				.6
s/co <1	35 243 (99)	2 ,660 (99)	10 583 (99)		38 559 (99)	27 400 (99)	11 159 (99)		0 (0)	0 (0)	0 (0)
s/co 1–10	45 (0.1)	27 (0.1)	18 (0.2)		58 (0.1)	36 (0.1)	22 (0.2)		58 (20)	36 (19)	22 (23)
s/co 10–100	17 (<0.1)	11 (<0.1)	6 (<0.1)		23 (<0.1)	15 (<0.1)	8 (<0.1)		23 (7.9)	15 (7.7)	8 (8.2)
s/co >100	195 (0.5)	132 (0.5)	63 (0.6)		210 (0.5)	143 (0.5)	67 (0.6)		210 (72)	143 (74)	67 (69)
Engagement in HPP				.6				<.001				>.9
Engaged in comprehensive HPP	1105 (3.1)	781 (3.1)	324 (3.0)		1725 (4.4)	1325 (4.8)	400 (3.6)		9 (3.1)	6 (3.1)	3 (3.1)
Not engaged in comprehensive HPP	34 395 (97)	24 049 (97)	10 346 (97)		37 125 (96)	26 269 (95)	10 856 (96)		282 (97)	188 (97)	94 (97)
Final test interpretation				.3				.2				.5
False positive	48 (0.1)	29 (0.1)	19 (0.2)		62 (0.2)	39 (0.1)	23 (0.2)		62 (21)	39 (20)	23 (24)
True negative	35 243 (99)	24 660 (99)	10 583 (99)		38 559 (99)	27 400 (99)	11 159 (99)		0 (0)	0 (0)	0 (0)
True positive	209 (0.6)	141 (0.6)	68 (0.6)		229 (0.6)	155 (0.6)	74 (0.7)		229 (79)	155 (80)	74 (76)

Data are presented as No. (%) unless otherwise indicated.

Values in bold indicate significant P-values.

Abbreviations: HPP, human immunodeficiency virus prevention program; IQR, interquartile range; s/co ratio, signal-to-cutoff.

Wilcoxon rank-sum test; Pearson χ2 test.

Wilcoxon rank-sum test; Pearson χ2 test when expected cell counts were ≥5, and Fisher exact test when expected cell counts were <5.

Description of s/co Distribution in the Training and Validation Cohorts

A log-plotted view of all test results demonstrates that the s/co of most cases is around 0.2 or >100 (Supplementary Figure 1). The majority of negative results for those engaged and not engaged in comprehensive HIV prevention services had an s/co of <0.2 (97% overall), with 2.4% from 0.2 s/co to 1.0 s/co. Individuals engaged in the HPP were more likely to have an s/co >0.1 and <1 (Table 2). Most positive results were >100 s/co (72%). The distribution of positive tests demonstrated that 90% (55) of those with an s/co of 1 to 10 were false positives compared with 9.8% (6) of those with s/co of 10–100. All results with s/co >100 (211) were true positives (Figure 1).

Table 2.

Distribution of Signal-to-Cutoff Ratio Among Individuals Engaged and Not Engaged in a Comprehensive Human Immunodeficiency Virus Prevention Program

s/co Category	Overall[a](N = 38 850)	Engaged in a Comprehensive HPP (n = 1725)	Not Engaged in a Comprehensive HPP (n = 37 125)	P Value[b]
s/co ≤ 0.1	23 508 (61)	981 (57)	22 527 (61)	.002
s/co >0.1 and ≤ 0.2	14 126 (36)	681 (39)	13 445 (36)	.006
s/co >0.2 and < 1	925 (2.4)	54 (3.1)	871 (2.3)	.037
s/co 1–10	58 (0.1)	6 (0.3)	52 (0.1)	.043
s/co 10–100	23 (<0.1)	3 (0.2)	20 (<0.1)	.08
s/co >100	210 (0.5)	0 (0)	210 (0.6)	.002

Data are presented as No. (%) unless otherwise indicated.

Values in bold indicate significant P-values.

Abbreviations: HPP, human immunodeficiency virus prevention program; s/co ratio, signal-to-cutoff.

Contains both training and validation cohorts.

Pearson χ2 test when expected cell counts were ≥5, and Fisher exact test when expected cell counts were <5.

Figure 1.

Distribution of signal-to-cutoff ratio for patients with a positive human immunodeficiency virus (HIV) antigen/antibody test.

Description of s/co Differences Among Patients With Multiple Test Results in the Training and Validation Cohorts

A total of 2754 patients had multiple tests (median, 2 [range, 2–6]), resulting in 3146 test pairs. Of repeat tests, 97.5% and 99.2% demonstrated an absolute s/co change of 0.1 or 0.2 or less, respectively. A total of 25 results, 7 from patients engaged in HIV prevention services and 18 from nonengaged patients, showed a >0.2 s/co difference between repeat tests (Figure 2).

Figure 2.

Change in negative tests over time among individuals engaged and those not engaged in comprehensive human immunodeficiency virus (HIV) prevention services.

Classification and Regression Tree Analysis

We used CART analysis on the training cohort to automate the identification of clinically meaningful thresholds for the likelihood of our primary outcome (true-positive HIV result). The model included the patient’s age, sex, engagement in a comprehensive HIV program, and s/co result. However, in the CART model, the s/co was the only predictor of a true-positive result with a cutoff of 38.78. Eight (11.6%) of those with s/co <38.78 were true positives, while all patients with an s/co >38.78 had a true positive result. In a univariate model of patients in both the training and validation cohorts, age, sex, engagement in an HPP, and s/co ≤38.78 met our specified cutoff of P <.05 and were included in the multivariate logistic regression model. In the multivariate logistic regression model using data from only the validation cohort, both engagement in the HPP (odds ratio [OR], 1.72) and s/co (OR, 2.49) were associated with a true-positive result (Table 3). When the selected cutoff was applied to the 11 256 observations In the validation cohort, the sensitivity decreased from 100% to 95%, while the NPV also decreased from 100% to 99.9%. Concurrently the specificity and PPV increased from 99.8% to 100% and 79% to 100%, respectively.

Table 3.

Univariate and Multivariate Analysis

Characteristic	Summary(Training and Validation Cohorts)	Univariate Regression(Training and Validation Cohorts)(n = 291)			Multivariate Regression(Training and Validation Cohorts)(n = 291)			Multivariate Regression(Validation Cohort Only)(n = 97)
	(N = 291)	OR	(95% CI)	P Value[a]	OR	(95% CI)	P Value[a]	OR	(95% CI)	P Value[a]
Age, y, median (IQR)	36 (29–54)	1	(1–1.01)	.042	1	(1.00–1.00)	.3	1	(1.00–1.00)	.2
Sex recorded at birth
Female	89 (31)	…	…		…	…		…	…
Male	202 (69)	1.23	(1.12–1.36)	<.001	1.02	(.98–1.07)	.3	1.05	(.98–1.13)	.2
Engagement in an HPP
Engaged in a comprehensive HPP	9 (3.1)	0.56	(.43–.73)	<.001	1.11	(1–1.24)	.61	1.72	(1.41–2.10)	<.001
Not engaged in a comprehensive HPP	282 (97)	…	…		…	…		…	…
s/co ≥ 38.78	221 (76)	2.42	(2.32–2.53)	<.001	2.45	(2.34–2.57)	<.001	2.49	(2.30–2.69)	<.001

Values in bold indicate significant P-values.

Abbreviations: CI, confidence interval; HPP, human immunodeficiency virus prevention program; IQR, interquartile range; OR; odds ratio; s/co ratio, signal-to-cutoff.

Wilcoxon rank-sum test for continuous variables; Pearson χ2 test when expected cell counts were ≥5, and Fisher exact test when expected cell counts were <5.

DISCUSSION

The goal of this study was to use an unbiased machine-learning approach in the retrospective review of clinical and laboratory data from a large cohort for whom HIV test results were available. We sought to define the range in s/co values for patients who are HIV positive and HIV negative and understand the potential future use of s/co to inform clinical decision making. Multiple studies have demonstrated a change in viral load and time to seroconversion for individuals receiving PrEP [15, 16, 21–24]. This has contributed to the change in the HIV prevention guidance recommending HIV RNA assays, in addition to HIV Ag/Ab as part of routine HIV prevention care [6]. Additionally, data have also revealed that early initiation of ART during acute HIV led to HIV-negative HIV-specific antibodies failing to develop or disappearing [13, 14]. Thus, for those who are starting iART, qualitative HIV testing may be challenging to interpret. Furthermore, 1 consequence of increased HIV testing and decreasing HIV incidence will be an increasing percentage of false positives as PPV decreases while NPV increases in settings of low prevalence. In the era of iART, this could result in an increase in exposure to ART with associated financial and psychological costs for many individuals who are HIV negative. Our data suggest that quantitative s/co could improve the accuracy of HIV testing processes.

In this large study of s/co ratios, only 0.25% of patients had s/co values of 0.5 -1. Using personalized pretest probabilities of HIV risk could allow for individualized s/co thresholds lowering the threshold to, for example, 0.5. Theoretically, in high-prevalence settings, such as a PrEP cohort, where patients receive frequent testing and HIV viral load could be suppressed due to medications, lowering the s/co threshold value could increase sensitivity and NPV without significant loss of specificity and PPV.

In addition, it is also essential to understand and define intraindividual variation in s/co values over time. Figure 2 demonstrates that for most patients, there is minimal variation in 2 test results: 99.3% with an s/co change of <0.2. These data suggest that a difference in results >0.2 for 2 samples from an at-risk patient, such as a patient receiving HIV prevention services, over a period of time, may flag a potential change in clinical status. Such a tailored flag can raise the alarm and prompt the clinician to request additional testing.

This study was conducted in the era of daily tenofovir disoproxil fumarate (TDF)–based PrEP. However, evaluating the role of HIV testing with current oral PrEP may provide insight into HIV testing for other HIV prevention modalities, including daily tenofovir alafenamide and on-demand dosing strategies. Long-acting injectable PrEP, long-acting oral medications, and the potential for implantable prevention can provide prolonged therapeutic levels. However, with these newer modalities, there could potentially exist prolonged periods of subtherapeutic levels when the drug is discontinued. Due to these medications’ long-acting nature, stopping PrEP to address ambiguous test results may not be possible in the short term [12]. Identifying potential infections using quantitative HIV test results may allow for earlier identification and appropriate treatment with ART before the development of resistance.

It is exactly this concern that led to the recent CDC recommendation to perform HIV RNA testing as part of routine PrEP care. However, HIV RNA testing is more expensive than HIV Ag/Ab testing, and the current recommendations may not be feasible for implementation at all care sites. Using the quantitative s/co reflex, HIV RNA testing could be done for individuals with s/co greater than a prespecified cutoff (example: 0.2 or 0.5) or with a significant intraindividual variation (example: >0.2 change), reducing the number of RNA tests required and making PrEP provision viable at more sites.

As we close in on ending the epidemic and HIV prevalence decreases, false positives with routine testing will become more dominant, and physicians may be hesitant to continue sending routine HIV testing. Additionally, in the era of iART, quantitative s/co reporting could allow for more informed choices on whether to bring patients in immediately to start ART or wait for the results of confirmatory testing. In low-prevalence settings, raising the s/co has been explored to increase specificity and PPV without losing sensitivity [25].

Our model identified a single cut-point. All patients with s/co >38.78 were true positives, creating a cohort for which bringing the patient back to start iART would be indisputably appropriate. Most patients with s/co of <38.78 were false positives, creating a cohort for which waiting for confirmatory testing before iART could be considered appropriate without prior exposure to ART. As many risk factors are not well captured in structured data, only demographic data were used for this analysis [26-28]. Known risk factors for true positives, such as a history of condomless sex with new partners, and false-positive results like pregnancy, autoimmune conditions, and others may allow for better risk stratification when combined with s/co. The ability to incorporate this risk factor data in the future may allow for improved models. While the current recommendations are to start patients on ART, given our data, it may make sense to wait for supplemental testing, particularly in those with a lower pretest probability, s/co <38.78, and not taking PrEP. Adding this parameter would potentially decrease not only drug exposure and drug cost but also psychological distress and mistrust in the healthcare system. For patients with an s/co between 1.0 and 38.78, these data suggest that in the absence of significant clinical risk factors, there is equipoise, and a shared decision-making approach between patient and provider is needed.

Although these observations are intriguing, they require further investigation. These data are limited as they come from a single center, using a single assay in the era of daily TDF-based PrEP. Currently, the HIV Ab/Ag Combo assay is not Food and Drug Administration approved as a quantitative assay, but one where s/co values are interpreted as positive or negative based on reaching a cutoff of 1.0 s/co. Although we reviewed >38 000 HIV Ag/Ab results, there was a limited number of false-positive tests, no false negatives identified, and no patients who converted while taking HIV prevention medications in this data set.

Our preliminary data suggest that further prospective investigation is needed. The assay could be explored for use as a quantitative test, combined with an individualized risk profile, leading to improved clinical decision making as PrEP and iART alter the underlying biology of acute HIV infection for many.

Click here for additional data file.