Application of Health Technology Assessment (HTA) to Evaluate New Laboratory Tests in a Health System: A Case Study of Bladder Cancer Testing.

We describe the methods and decision from a health technology assessment of a new molecular test for bladder cancer (Cxbladder), which was proposed for adoption to our send-out test menu by urology providers. The Cxbladder health technology assessment report contained mixed evidence; predominant concerns were related to the test's low specificity and high cost. The low specificity indicated a high false-positive rate, which our laboratory formulary committee concluded would result in unnecessary confirmatory testing and follow-up. Our committee voted unanimously to not adopt the test system-wide for use for the initial diagnosis of bladder cancer but supported a pilot study for bladder cancer recurrence surveillance. The pilot study used real-world data from patient management in the scenario in which a patient is evaluated for possible recurrent bladder cancer after a finding of atypical cytopathology in the urine. We evaluated the type and number of follow-up tests conducted including urine cytopathology, imaging studies, repeat cystoscopy evaluation, biopsy, and repeat Cxbladder and their test results. The pilot identified ordering challenges and suggested potential use cases in which the results of Cxbladder affected a change in management. Our health technology assessment provided an objective process to efficiently review test performance and guide new test adoption. Based on our pilot, there were real-world data indicating improved clinician decision-making among select patients who underwent Cxbladder testing.

Introduction

Hospital clinical laboratories are under significant pressure from clinicians and vendors who are promoting the use of new test modalities, more recently often focused on molecular diagnostic techniques. Such new laboratory tests frequently claim to be integral parts of a precision medicine approach, to improve diagnostics, and to be less invasive, more accurate, and offer earlier diagnoses. Such tests are typically send-out tests performed at nonlocal laboratories which can incur substantial institutional costs for a hospital. A subset of these tests have uncertain sensitivity and specificity and lack data supporting broad use.

With the rapid growth in available laboratory tests, clinical laboratories are faced with a challenge of balancing 2 types of fundamental error: (1) making new laboratory tests available that are inaccurate or (2) denying access to cutting-edge diagnostics that are clinically beneficial or more precise compared to current diagnostic methods.[1] The fiscal risk of adopting new tests can also be significant. Most new tests are initially offered at 3 times the cost of current practices. The cost of these tests can be fiscally challenging if they are not reimbursed by the Centers for Medicare and Medicaid Services (CMS) or commercial payers (eg, Regence, Premera Blue Cross, Kaiser). This can lead to uncompensated costs for the hospital.

With the implementation of a hospital-based health technology assessment (HTA) program, Smart Innovation,[2] and development of a Laboratory Formulary Committee, our institution has been able to integrate HTA practices into laboratory medicine, aiming to steward precious health care resources by adopting for use only those laboratory tests with strong evidence.

Bladder cancer is diagnosed in more than 70 000 patients in the United States every year and in over 430 000 patients globally, making it the 4th most common cancer in men and the 11th most common cancer in women.[3] Localized bladder cancer can be categorized into 2 main groups: muscle invasive and nonmuscle invasive. Those with nonmuscle invasive bladder cancer comprise 70% of incident bladder cancer cases. These tumors are characterized by a high rate of recurrence in the bladder necessitating frequent surveillance.[4] UW Medicine conducts approximately 500 bladder cancer screens (cystoscopies) per year. Surveilling recurrent urothelial carcinoma (UC) of the bladder requires frequent cystoscopy, which involves the insertion of a cystoscope to the bladder via the urethra, and urine cytopathology. Urine cytopathology has fair interobserver concordance (kappa = 0.32, average chance-corrected agreement including low- and high-grade tumors) suggesting the need for a more objective urine-based test.[5] These tests can be expensive and time-consuming for patients; indeed, bladder cancer has higher per-lifetime, per-patient cost than most forms of cancer.[6]

The Laboratory Formulary Committee was approached by a urologist interested in utilizing a newly available send-out molecular bladder cancer test for initial diagnosis and for surveillance of recurrent disease. The proposed test (Cxbladder; Pacific Edge Cancer Diagnostics) is a urine-based molecular test and therefore offers a less invasive screening option for patients compared to the current gold standard, cystoscopy and cytopathology. The Cxbladder test uses quantitative polymerase chain reaction techniques to analyze messenger ribonucleic acid expression levels of 4 genes associated with urothelial cancer and 1 gene involved in inflammatory responses. When utilized for surveillance, the test combines the urine-based gene expression findings with patient clinical data to help determine the probability of the presence of urothelial cancer.[7]

Methods

Following the physician’s request to have the use of Cxbladder reviewed by the Laboratory Formulary Committee, the Laboratory Medicine Department contacted Smart Innovation staff to determine if the new bladder cancer test should go through Smart Innovation’s decision framework and evidence review. The Laboratory Formulary Committee is made up of 24 voting members and 5 nonvoting members. Among the voting members, there were 4 who were laboratory medicine directors and 20 who were nonlaboratory medicine clinicians. The nonvoting members include physicians and Smart Innovation staff, and the chair of the committee is the Chair of Laboratory Medicine. It was decided that the test met the program inclusion criteria and Smart Innovation began the process to lead the technology assessment of the new test for adoption consideration.[2] Smart Innovation’s decision framework consists of 5 formal steps[2] and included 9 dimensions of evidence.[8] The program engages with a clinical committee as well as a hospital executive committee for formal decision-making (Figure 1).

Figure 1.

Smart Innovation decision framework. (1) Smart Innovation begins with a new technology request from a clinician or department. (2) Smart Innovation reviews each request to determine it meets inclusion criteria. (3) The HTA report includes 9 dimensions of evidence and is reviewed by a clinical expert for accuracy. (4) A clinical committee specific to the technology reviews the HTA report and makes an adoption recommendation to the clinical committee (if the clinical committee votes to not to adopt, it is not forwarded to the executive committee). (5) The executive committee reviews the new technology and clinical committee recommendations and makes the final adoption decision.

For the review of Cxbladder, Smart Innovation utilized a set of decision-making processes to evaluate the new test and engaged the technology sponsor (the clinical laboratory/urologist) as well as UW Medicine’s multidisciplinary Laboratory Formulary Committee, finance, administration, physicians, and physician leadership.[2] The Committee approved its use on a limited basis by a single urologist for cases meeting a narrow definition: negative cystoscopy and atypical/suspicious urine cytology. Following a decision by the Committee, a case series review was performed for 12 patients: 6 who had received the test in the subsequent year following the adoption decision and 6 patients evaluated for suspicion of recurrent bladder cancer prior to the adoption decision. A pathologist at our institution summarized the physician notes and patient charts by categorizing the outcome variables and described the observed differences in clinician decision-making and patient outcomes. Outcome variables included the type and number of follow-up tests conducted, their results, and results of subsequent cystoscopy, cytopathology, imaging studies, biopsy procedures, and, if applicable, repeat Cxbladder testing. Test turnaround time was calculated as the duration (in days) between the date of sample collection and the date the result was received from the outside laboratory.

In December 2019, Smart Innovation collaborated with Laboratory Medicine to conduct a postdecision analysis on the Cxbladder pilot study. We selected 12 patients of a single urologist at our institution. The 12 patients each had a negative cystoscopy and an atypical or suspicious urine cytology result. Group 1 included 6 patients tested with Cxbladder between January 2018 and December 2019 (Table 1). For all 6 patients, a negative cystoscopy result and atypical or suspicious cells on 1 or 2 successive urine cytopathology tests prompted the physician’s decision to send the Cxbladder test. Search functions of our institution’s pathology database were utilized to identify a similar group of patients (group 2), comprised of patients seen by the same urologist who were found to have atypical or suspicious urine cytopathology but between 2016 and 2017 when Cxbladder was not available. The search was then narrowed to those noted to have negative cystoscopy at the time of urine collection, similar to the group tested with Cxbladder, yielding 6 patients for group 2 in the case series (Table 2).

Table 1.

Clinical Data From Cases With Cxbladder Testing (Group 1).

Age	Tumor type	Cytopathology	Cxbladder test	TAT	Result	Management	Outcome
				(days)
76	Noninvasive, high grade	Suspicious cells	MONITOR	9	Score: 7.4; clinician-directed protocol	Biopsy	Positive for recurrence
69	Noninvasive, low grade	Atypical cells	DETECT	9	Score: 0.04; normal gene expression score, 97% NPV	Surveillance cystoscopy/cytology q 6 months + BCG	No recurrence identified × 1.5 years
66	Invasive, low grade	Highly atypical cells × 2	MONITOR	13	Score: 4.3; clinician-directed protocol	Surveillance cystoscopy/cytology q 6 months	No recurrence identified × 1 year
60	Invasive, high grade	Rare atypical cells × 2	MONITOR	10	Score: 3.0; low probability, 97% NPV	Cystoscopy in 1 year	Cystoscopy at 1 year negative
83	Invasive, high grade	Atypical cells	DETECT	6	Score: 0.07; normal gene expression score, 97% NPV	Surveillance cystoscopy/cytology q 3 months + BCG	No recurrence identified × 1.5 years
69	Invasive, high grade	Suspicious cells	DETECT	6	Score: 0.18; elevated gene expression score	Surveillance cystoscopy/cytology q 6 months + BCG	6-month cytology positive for carcinoma
		Suspicious cells	DETECT	7	0.72; high gene expression score	Biopsy	Positive for recurrence

Abbreviations: BCG, Bacillus Calmette Guerin; NPV, negative predictive value; Q, every; TAT, turnaround time.

Table 2.

Clinical Data From Cases Without Cxbladder Testing (Group 2).

Age	Tumor type	Cytopathology	Cxbladder test	TAT	Result	Management	Outcome
				(days)
74	Noninvasive, high grade	Highly atypical cells				Biopsy	Positive for recurrence
56	Noninvasive, high grade	Rare atypical cells				Surveillance cystoscopy/cytology q 3 months + BCG	Suspicious cystoscopy at 1 year, biopsy = negative, no recurrence identified × 4 years (death by other causes)
76	Noninvasive, high grade	Suspicious cells				Surveillance cystoscopy/cytology q 3 months + mitomycin C	Suspicious cystoscopy at 1 year, biopsy = positive for recurrence
83	Noninvasive, low grade	Rare atypical cells				Surveillance cystoscopy/cytology q 3 months + valrubicin	Positive cystoscopy at 2.5 years, biopsy = positive for recurrence
56	Invasive, focal high grade	Rare atypical cells				Surveillance cystoscopy/cytology q 3 months + BCG	No recurrence identified × 4 years
82	Invasive, high grade	Rare atypical cells				Surveillance cystoscopy/cytology q 3 months + mitomycin C	No recurrence identified × 3 years (death by other causes)

Abbreviations: BCG, Bacillus Calmette Guerin; NPV, negative predictive value; Q, every; TAT, turnaround time.

We also evaluated whether Cxbladder testing was associated with a difference in the number of tests ordered (cytologies, cystoscopies, biopsy) between groups 1 and 2. Patients were followed for at least 12 months following an atypical urine cytology, and the counts of tests between the 2 groups were compared using the nonparametric analysis of variance Kruskal-Wallis test.

Results of the Health Technology Assessment

Published Evidence

The published clinical data available for Cxbladder contained mixed evidence. One vendor-funded published study demonstrated a sensitivity of 91% and a specificity of 60% for Cxbladder compared to the gold standard (cystoscopy and biopsy).[9] The study compared urine-based testing to cystoscopy and used a total of 1036 urine samples collected from 803 patients undergoing surveillance for urothelial cancer. An Agency for Healthcare Research and Quality (AHRQ) funded systematic review that evaluated urinary biomarkers for the diagnosis of bladder cancer, including Cxbladder, found a high rate of false positives, and that accuracy is poor for low-stage and low-grade tumors.[10] The systematic review also presented one study that was published in 2012 and funded by the vendor in which the accuracy of Cxbladder indicated medium-risk of bias with a reported sensitivity of 0.82 (CI, 0.70-0.90) and a specificity of 0.85 (CI, 0.81-0.88) for evaluation of symptoms, for a positive likelihood ratio (LR) of 5.53 (CI, 4.28-7.15), and for a negative LR of 0.21 (CI, 0.13-0.36).

Conclusions from the AHRQ’s systematic review included: “Urinary biomarkers miss a substantial proportion of patients with bladder cancer and are subject to false-positive results in others.” They also specified there’s not enough current medical evidence to show that using tumor marker tests compared to standard tests to look for bladder cancer leads to more health benefits and that more medical studies are needed. Results from AHRQ’s analysis of Cxbladder included the following: (1) demonstrated that strength of evidence = as low, (2) reporting bias = not detected, (3) precision = indicated imprecise, and (4) consistency = could not be determined.

Clinical Guidelines for Bladder Cancer Screening

At the time of the review, there were no national clinical guidelines that recommended using Cxbladder for bladder cancer screening. The American Urological Association published guidelines on the evaluation of microscopic hematuria in adults. They recommended cystoscopy for adults older than 40 years, as well as for adults younger than 40 if they had risk factors for developing bladder cancer.[11] The National Institute for Health and Care Excellence 2015 bladder cancer diagnosis and management guidelines did not indicate using urine-based testing (including Cxbladder) for diagnosing and monitoring bladder cancer. They recommend using cystoscopy as the gold standard.[12] The National Comprehensive Cancer Network does not address Cxbladder specifically but recommended consideration of urine biomarker tests for monitoring disease recurrence.[13] Major national health insurers were assessed for coverage options and Premera Blue Cross, a primary carrier in our region indicated a noncoverage policy for Cxbladder, considering it experimental.[14] At the time of the assessment, there was not a US National Coverage Decision, and the CMS did not cover Cxbladder.

Other HTA Reports

During our review in 2017, we searched for other HTA reports and found one conducted by Emergency Care Research Institute (ECRI). There were no HTA reports by the Washington State Healthcare Authority Health Technology Assessment Program. The ECRI report reviewed Cxbladder for the indication of monitoring patients with a confirmed bladder cancer diagnosis. Two key conclusions from the ECRI HTA report included: (1) “Cxbladder use does not completely obviate the need for routine cystoscopy” and (2) “additional studies of Cxbladder that focus on extended follow-up are required.” The ECRI report recommended that future studies should quantify the proportion of patients receiving false-negative results and be appropriately compared to cystoscopy to achieve a better balance between patient quality of life (decreasing invasive cystoscopy) and ensuring cancer recurrence is identified prior to any serious health concerns.[15]

UW Medicine, Fiscal Impact Analysis

The fiscal impact of adopting Cxbladder at our institution, in addition to, but not replacing, the current cystoscopy with and without biopsy exams, was estimated at US $1.5 million dollars annually. When assessing our average cost to screen and monitor for bladder cancer: US $900 per cystoscopy (with and without biopsy) and total cost per year US $450 000 to the additional cost of Cxbladder: US $3000 per test and total cost per year US $1.5 million, Cxbladder was estimated to be more than 3 times the cost of cystoscopy and urine cytology (approximately 500 tests per year). Suggestive or positive Cxbladder tests would need to be followed up with cystoscopy, so it was determined that “replacement” of cystoscopy would not entirely occur, therefore not offering economic or clinical efficiency, and the true cost would include current testing methods (cystoscopy, biopsy, cytology, imaging) as well as Cxbladder. There are other costs associated with Cxbladder testing that were not detailed in our fiscal report that add to the overall cost. These include unnecessary office visits and other laboratory testing costs.

Adoption Decision

The Laboratory Formulary Committee reviewed the available evidence with the requesting clinician and determined that given the low specificity of the test, the fiscal impact, and lack of coverage by primary health insurers in the United States, the hospital would be incurring the cost of the test without likely reimbursement at low yield for patients. Therefore, Smart Innovation and the Laboratory Formulary Chair recommended that additional published data were necessary before the test would be approved for use for bladder cancer monitoring or surveillance.

After reviewing the evidence, members of the Laboratory Formulary Committee voted to not adopt Cxbladder system-wide; however, to support a pilot study for the requesting urologist for the stated indication: “an equivocal visualized lesion with atypical urine cytology,” which was estimated to be approximately 10 patients per year. This recommendation was unanimously approved by the clinical committee, was reviewed and approved by our executive committee, and the policy went into effect in January 2018.

Pilot Study

Patients in both groups were in a similar age range (group 1: range: 60-83 years, median: 69 years; group 2: range: 56-83 years, median: 75 years). All 12 patients had a history of papillary UC, high-grade or low-grade, invasive or noninvasive, in both groups as indicated in Tables 1 and 2. All patients had received prior rounds of Bacillus Calmette Guerin (BCG) therapy with or without resection.

The Cxbladder test was sent 9 times for a total of 6 patients in group 1. One test was rejected due to kit expiration and 1 test was rejected due to sample stability. One patient was tested twice. The provider/clinic variably ordered the Cxbladder MONITOR test or the Cxbladder DETECT test, due to the availability of kits placed in the clinic by the manufacturer, despite the tests’ indications for suspected recurrence, which would suggest the use of the MONITOR test. The 7 Cxbladder test results varied for each pilot case with values ranging from 0.04 to 7.4, and results of normal (n = 2), low probability (n = 1), elevated gene expression score (n = 1), high gene expression (n = 1), and clinician-directed protocol (n = 2). Both patients with a normal score received surveillance cystoscopy and cytopathology (1 every 3 months, 1 every 6 months) with BCG and did not receive a biopsy.

In group 1, recurrence was identified by the time of conclusion of this study in 2 of the 6 patients (median follow-up of patients without recurrence was 22 months at the conclusion of this study [range: 16-27 months]). One of the patients with a low probability score was followed with cystoscopy in 1 year, which was negative. The 2 patients with “clinician-directed protocol” scores were treated differently based on clinician judgment: the first with a history of high-grade bladder cancer and the higher score (7.4 with a range of clinician-directed protocol from 3.5 to 10.0) underwent a biopsy which was positive for recurrence; the second patient, with a history of low-grade noninvasive bladder cancer and a lower score (4.3), underwent surveillance cystoscopy/cytopathology at 6-month intervals, with no recurrence identified by the time of conclusion of this study. The patient tested twice initially received an elevated gene expression score which gave a “low probability of urothelial cell carcinoma” and was followed with surveillance cystoscopy and cytopathology at 6-month intervals with BCG; however, the 6-month cytopathology was positive for carcinoma. At that time, a second Cxbladder was sent, which demonstrated a high gene expression score. A biopsy demonstrated the recurrence of the patient’s carcinoma. For the 2 patients in group 1 who were found to have recurrent carcinoma within the study time frame, the time to detection of recurrence was 64 and 308 days. In total, the patients in group 1 received 10 urine cytologies, 9 cystoscopies, and 1 biopsy in the first year following each patient’s initial atypical cytology result.

In group 2, the group not receiving Cxbladder, 5 of 6 patients were managed with surveillance cystoscopy and urine cytopathology every 3 months with variable therapies including BCG, mitomycin C, and valrubicin. For 1 patient with highly atypical cells on cytopathology, a biopsy was performed 1 month after the initial cytology result and was positive for recurrence. For the remaining 5 patients, if at any point cystoscopy returned positive, biopsy was performed. In total, 3 of the 6 patients in group 2 were found to have recurrent carcinoma within the study time frame, with time to detection of recurrence of 37, 343, and 1014 days. The median follow-up of patients without recurrence was 35 months at the conclusion of this study (range: 31-52 months). In total, the 6 patients in group 2 received 14 urine cytologies, 13 cystoscopies, and 3 biopsies in the first year following each patient’s initial atypical cytology result.

The counts of follow-up tests for both groups are displayed in Table 3. There was not a statistically significant between-group difference in follow-up tests ordered (P = .15).

Table 3.

Number of Tests Ordered Based on Group in 1 Year.*

Group	# of Cytologies	# of Cystoscopies	# of Biopsies
Group 1	10	9	1
Group 2	14	13	3

* Data limited, in both groups, to the first 12 months following atypical cytology.

Discussion

Making HTA decisions can be difficult when a new technology has yet to develop a large body of evidence including real-world data. This leaves decision makers with a level of uncertainty that can only be addressed prospectively: (1) by waiting for newly published evidence (by other institutions) or (2) by conducting pilot studies. Therefore, HTA committees tend to lean more on the conservative side of adoption decisions to abate the risk of poor patient outcomes and unnecessary costs from adopting new and uncertain technologies.

The HTA decision process for Cxbladder was able to weigh the financial risk and the potential added improvement for managing certain patients with bladder cancer. This is where the difficulty can arise in a clinical committee and challenges the competing interests of clinicians who seek new advantages when managing complex patients and HTA experts who are concerned with the fiscal impacts and are without the needed evidence to make a clear recommendation. In our clinical committee, the discussion addressed these issues and eventually landed on not adopting system-wide, but to pilot Cxbladder.

What we learned from this new approach was if we would have made a safe and low-risk decision to not adopt and not pilot, we would have missed an opportunity to identify a certain patient profile that can benefit from Cxbladder testing. For patients with normal cystoscopy and atypical cytology, use of Cxbladder led to fewer follow-up cystoscopies, cytologies, and biopsies with the same proportion of patients with recurrence detected within 1 year (2 of 6 in each group). We also learned that following up on HTA decisions is important and collaborating with our clinicians derives a true partnership as well as improved insights for clinical guideline development. Smart Innovation will continue to consider this type of HTA decision pathway (do not adopt—pilot), and our leaders and staff now have the confidence and understanding about how to better balance competing interests among clinicians, fiscal impacts, and HTA.

There is research interest in better understanding the real-world data related to using Cxbladder for bladder cancer screening and monitoring. A recently published study[16] used 443 Cxbladder MONITOR tests among 309 patients at 3 hospitals in New Zealand. The authors indicated that Cxbladder MONITOR was able to identify 77.8% who were safely managed with 1 cystoscopy per year. The authors concluded, “Including Cxbladder MONITOR in the protocol for patient surveillance provided clinical utility by reducing the average number of annual cystoscopies by approximately 39%.” The nonvendor-funded study may offer insights for urologists who are monitoring patients with known UC; however, the generalizability may not be representative to all US hospitals because of the difference between single and pluralistic payer systems, and Cxbladder is not widely reimbursed in the United States.

The HTA evaluation of Cxbladder demonstrated that adding an alternative HTA decision pathway (do not adopt—pilot) provided the ability to pilot a new molecular test that in traditional HTA programs would not have been implemented. The pilot study offered our HTA body, the Laboratory Formulary Committee, and clinicians at our institution the benefit of collecting real-world data following our HTA decision. Cxbladder performed better than cytology and other noninvasive tests, as well as trended to a reduced number of cytologies, cystoscopies, and biopsies among certain patients, although the difference did not reach statistical significance.

The case series demonstrated a relatively consistent surveillance management of patients with atypical or suspicious cytology prior to the limited use of Cxbladder. After piloting Cxbladder for this subset of patients, high scores on Cxbladder, with suspicious cytopathology, appeared to move the provider to perform a biopsy sooner than in the provider’s standard of care practice; 2 patients had intermediate score recommending “clinician directed protocol” and the management differed in each based on the provider judgment, one undergoing biopsy with cancer recurrence detected and the other monitored with surveillance and no recurrence was identified within the follow-up period. Some patients with negative and low scores of Cxbladder received less frequent follow-up surveillance than had been the provider’s usual care based on the evaluation of management in group 2.

Conclusions

We found the HTA process was more efficient and less resource intensive than conducting a broad-based study with a large sample size and provided policy decisions that reflects the published evidence, clinical guidelines, and actual clinical practice. Our HTA decision regarding Cxbladder will continue to be monitored to ensure patients are afforded accurate and early diagnosis of bladder cancer by ongoing review of emerging research, coverage, as well as clinical guidelines. The results of the pilot study indicated that among a narrow patient profile selected, Cxbladder assisted the pilot urologist in ordering confirmatory testing and a reduction in the number of follow-up cystoscopies, cytologies, and biopsies within 1 year.