Who is responsible for evaluating the safety and effectiveness of medical devices? The role of independent technology assessment.

INTRODUCTION: The global medical technology industry brings thousands of devices to market every year. However, significant gaps persist in the scientific literature, in the medical device approval process, and in the realm of postmarketing surveillance. Although thousands of drugs obtain approval only after review in randomized controlled trials, relatively few new medical devices are subject to comparable scrutiny.
OBJECTIVE: To improve health outcomes, we must enhance our scrutiny of medical devices, and, without simply deferring to the Food and Drug Administration, we must ask ourselves: Who is responsible for evaluating the safety and effectiveness of medical devices?
CONCLUSIONS: Technology assessments by independent organizations are a part of the solution to this challenge and may motivate further research focused on patient outcomes.

INTRODUCTION

New technology has been defined as “anything that doesn’t quite work yet.”1 While in some instances accurate, this definition would justifiably elicit public outcry if applied to the latest medical technology. And yet, as will be described below, many new medical devices (i.e., “medical technology”) make it to market without the kind of rigorous evidence expected for drug approval. In this paper, we review the existing process of medical device approval, and we discuss how independent, systematic assessments of newly approved medical devices can evaluate the extent and quality of the evidence for medical devices, thereby supporting the safe and effective use of emerging medical technology.

THE DRUG/DEVICE SPLIT: A HISTORICAL ARTIFACT?

Medical technology, regulated as devices, includes items as mundane as bedpans and latex gloves, and as complex as magnetic resonance imaging scanners and hybrid devices such as drug-eluting stents. The explosive growth of medical technology has paved the way for fields as disparate as organ transplantation and preimplantation genetic diagnosis.2 By the late 1990s, the US Food and Drug Administration (FDA) had approved about 500,000 medical device models produced by approximately 23,000 different manufacturers. It has been estimated that 4% of the population in the USA has at least 1 implanted medical device.3,4

In spite of the ubiquity and impact of emerging medical technology, few physicians, and even fewer patients, understand the process by which new medical devices are evaluated and approved by the FDA. Although there have been some serious and well-publicized problems (e.g., rofecoxib), the process for evaluating pharmaceuticals before their release into the market is much more thorough than the analogous process for new medical devices.5–9 Whereas all new drugs must pass rigorous premarketing approval testing with randomized clinical trials, such testing is required for relatively few new devices. Devices are defined according to the Federal Food, Drug and Cosmetic Act, Section 201, as “an instrument, apparatus, implement, machine, contrivance, implant, in vitro reagent, or other similar article that is intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease.”3,10 This drug/device split in testing extends throughout the health care system to health plans and hospitals, as formulary committees for drug coverage have few device–coverage correlates. The somewhat artificial distinction between devices and drugs is a historical and legislative artifact. In contrast, in the UK, the National Institute for Health and Clinical Excellence (NICE) evaluates both drugs and devices according to the same protocol. Recently, there has been a call to address the issue in the USA with a “center for comparative effectiveness reviews.”11

FDA APPROVAL: PROBLEMS AND PROSPECTS

Within the FDA, the Center for Devices and Radiological Health (CDRH) has primary responsibility for the premarket assessment of new medical technology. The CDRH not only considers the risks of new devices, but also monitors them via a nationwide postmarket surveillance system. New devices are classified as low (I), moderate (II), or high risk (III). Among the thousands of device applications submitted annually, fewer than 100—those considered to be novel and high-risk—undergo a premarket approval application (PMA) analogous to the rigorous scrutiny required for new drugs.12 The bulk of new device approval applications are instead initiated via a premarket notification application [510(k)], an FDA process based on the assumption that most devices are essentially equivalent to those already approved. For example, via the 510(k) pathway, the manufacturer of an approved laser may not be required to provide safety and efficacy data to employ a second-generation laser for treating conditions beyond those listed in the initial PMA approval.13 Computed tomographic (CT) colonography (virtual colonoscopy), which received 510(k) FDA clearance in May of 2003, exemplifies an application of CT for which neither validity nor clinical benefit had to be demonstrated before marketing and clinical application.14,15

This pathway to approval does not usually require clinical data derived from randomized trials regarding the effectiveness or safety of a device for a given use or population of patients.3,16–18 For the fiscal year 2005, the Office of Device Evaluation reported receiving 3,130 510(k) applications (ca. 250 per month), whereas only 43 original PMAs requiring rigorous study were submitted.12 The 510(k) process leads to the imprimatur “FDA approved” on many devices for which there may be only limited data in support of clinical effectiveness or safety for that specific clinical indication (see Fig. 1 for an overview of the FDA approval process for medical devices).3

Figure 1

Overview of the Medical Device Approval Process.3

There are concerns about inadequate scrutiny of devices at the FDA, resulting in part from limited resources. In fact, the FDA issued a report stating that “medical device program resources have been reduced in recent years, and there have been indications that review performance has begun to decline.”19 The Institute of Medicine (IOM) has described all of the FDA evaluation programs as “severely underfunded,” which has led to a potential increase in public risk and growing reliance on industry-paid user fees. These fees will account in 2008 for more than 40% of the drug regulation resources and almost a quarter of the resources needed for the review of medical devices.9,19–21 Some have expressed concern that user fees render the FDA more accountable to the very drug and device manufacturers it is supposed to be regulating and that the manufacturers are replacing the public as the FDA’s primary clients.21 In addition, the FDA has limited ability and authority to conduct postmarketing surveillance for both devices and pharmaceuticals; it falls on the manufacturers and academic and clinical investigators to initiate the bulk of recalls and failure reports.3,16,22–25 This situation leaves ample room for those with potential and real financial conflicts of interest to exert undue influence on the regulatory process. In a recent initiative to reduce conflicts of interest among members serving on device regulatory advisory committees, guidelines have been drafted that suggest an eligibility cutoff of US $50,000 as the maximum financial interest of any potential committee member under FDA consideration.26

Furthermore, the FDA explicitly does not regulate physician behavior or approve particular clinical procedures.27 Once a technology is approved, it can be used in clinical scenarios that fall outside the patient- and clinical-inclusion criteria of the pivotal trials. In the recent controversy over the long-term safety of drug-eluting stents, it was revealed that up to 55% of these devices are implanted for such “off-label” use.28,29 The stent uproar has been said to represent “both a success and a failure” of the FDA medical-device regulatory system: While a panel was rapidly assembled to respond to concerns, much still remains unknown about the long-term safety of drug-eluting stents, and millions of these devices have already been implanted.25

EVIDENCE-BASED MEDICINE AND TECHNOLOGY ASSESSMENT

Recognizing that all health care sectors need to cooperate to critically assess the available evidence to ensure the safety and effectiveness of medical care, the Institute of Medicine (IOM) addressed the “quality chasm” by initiating a roundtable on evidence-based medicine based on the findings of numerous studies showing that

... far too much of our health care spending is devoted to activities that do not improve health, and far too little investment is devoted to better understanding the relative advantages among various intervention choices. This gap in knowledge about what approaches deliver the best results will only be compounded as the pace of technology development quickens...30,31.

One means of addressing the problem of our lack of information about the relative efficacy and safety of various medical devices is to use an evidence-based medicine (EBM) approach that encourages the uniform application of consistent scientific methodology to inform and tailor care.32–34 EBM-based assessments of new medical technology have the potential to enhance safety and efficacy by advocating that clinical practice be informed by analyses of well-designed, randomized, controlled trials optimally conducted at more than 1 center; these features of EBM may increase transparency and reduce opportunities for bias and conflicts of interest. However, in some circumstances well-designed observational studies may suffice for evaluation of diagnostic tests.35

BEYOND THE FDA: APPLYING EBM TO MEDICAL DEVICES

Medical technology assessment may amplify conflicts of interest related to investments, reimbursement for clinical services, research support, political and lobbying influences, and other factors affecting all stakeholders: manufacturers, stockholders, physicians, patients, insurance carriers, and even the FDA itself. Fortunately, several existing medical technology assessment groups provide complementary critical analyses of the efficacy and safety of new medical devices (Tables 1, 2, and 3). Among the government-sponsored entities other than the FDA that have contributed to the research and dissemination of technology assessments, the most prominent is the Centers for Medicare and Medicaid Services (CMS), which often initiates coverage of medical devices with an evidence-development process ultimately aimed at synthesizing large amounts of data from various clinical experiences.36 Other groups include the IOM, the National Institutes of Health, the US Preventive Services Task Force, and the Agency for Healthcare Research and Quality (AHRQ), which itself includes a network for Developing Evidence to Inform Decisions about Effectiveness (DeCIDE), Centers for Education and Research on Therapeutics (CERTs), and several evidence-based practice centers (EPCs) funded by the AHRQ that not only review relative effectiveness and safety, but also identify areas requiring further study. A number of foreign federal and private technology assessment agencies (e.g., NICE in the UK) also make important contributions.

Table 1

Major Evidence-Based Sources of Information on Medical Technology

Center	Website	Services	Comments
Food and Drug Administration–Center for Devices and Radiological Health Databases (FDA–CDRH)	http://www.fda.gov/cdrh/databases.html	Full listing of medical devices in commercial distribution by both domestic and foreign manufacturers, as well as medical devices at various stages of premarket approval, and those devices which have been recalled from the market. The Manufacturer and User Facility Device Experience (MAUDE) database provides information on reported adverse events for individual devices.	FDA–CDRH reports are lengthy and dense; they often do not provide a synthesis of the data or the information. MAUDE is cumbersome to use. The CDRH also assures that non-medical radiation-emitting products such as cell phones, lasers, and TV sets meet radiation safety standards
	Addresses topics such as FDA approval, use and purchase of specific medical devices and diagnostic tests, manufacturer contact information, device instructions, post-approval alterations of products, and information about devices applied in the treatment and diagnosis of specific diseases
Centers for Medicare and Medicaid (CMS)	http://www.cms.hhs.gov/mcd/index_list.asp?list_type=tech	Information on Medicare coverage decisions including the rationale. Often provides cost analysis	Medicare Advisory Committee Technology Assessments: decisions may be influenced by political pressures, process is not always transparent, limited number of topics reviewed
The Cochrane Library Health Technology Assessment Database	http://www.mrw.interscience.wiley.com/cochrane/cochrane_clhta_articles_fs.html	Extensive number of topics covered. Access to assessments from other English-speaking countries, including Canada, UK and Australia. Usually systematic reviews and meta-analyses. Cost often included in the assessments	Reviews lengthy and do not all use the same criteria/standards; many non-technology topics are covered. Not all trial reports are eventually published. Reviews are referenced in the literature, but not as primary documents for medical technology evaluations; thus do not directly influence policy
	Requires a subscription to access the database, though reviews are often available for download

Table 2

Selection of Technology Assessment Entities

Center	Website	Services	Comments
Agency for Healthcare Research and Quality (AHRQ)	http://www.ahrq.gov	Focus on evidence, high standards, and technology’s impact on health outcomes. Input from an expert panel; collaboration with Kaiser Permanente. TEC is an evidence-based practice center (EPC) designated by the AHRQ.	Often lengthy assessments, but with a brief summary. Process informs BCBS coverage decisions. Serves clients such as Kaiser and CMS. There is a public route for submitting input to AHRQ, although it remains unclear how this input is incorporated into the decision-making process. 36
Technology Evaluation Center (TEC) of the Blue Cross–Blue Shield Association	http://www.bcbs.com/tec/whatistec.html
California Technology Assessment Forum-Blue Shield of California Foundation (CTAF)	http://www.ctaf.org	Evidence-based reviews with expert and community input; focus on identifying medical technologies that will improve health	No cost analyses. Assessments often lengthy. Stated goal is to educate the public, but assessments are often highly technical
Hayes	http://www.hayesinc.com	Independent, professional reviews; consultation about specific health technology topics	Must subscribe in order to read assessments
Institute of Clinical Systems Improvement (ICSI)	http://www.icsi.org	Independent non-profit organization funded by all 6 of the health payers in Minnesota	Focus on health care services for people who live and work in the state of Minnesota and in adjacent areas of surrounding states
ECRI Institute	http://www.ecri.org/Pages/default.aspx	Non-profit health research services agency that also operates an international medical device problem reporting system and a journal, Health Devices. Medical Devices Safety Reports (MDSR) are available	MDSR are selected and edited such that they provide general information. ECRI does not serve as an alerting service, but weekly alerts of medical device hazards and recalls are available through membership
	http://www.mdsr.ecri.org/information/about.aspx

Table 3

Selection of Professional Societies

Center	Web site	Services	Comments
American Society for Bariatric Surgery (ASBS)	http://www.asbs.org/	Specialist panels of individuals with experience in a particular field are asked to weigh in on the approval of a new device in their field. Members and specialist sites also provide information about post-approval application of new devices	Financial conflicts of interest and/or other professional bias must be taken into consideration, as such conflicts could reduce objectivity
American College of Cardiology (ACC)	http://www.acc.org/
American Rhinologic Society (ARS)	http://www.american-rhinologic.org/index.phtml
American Society for Therapeutic Radiology and Oncology (ASTRO)	http://www.astro.org/
American College of Physicians (ACP)	http://www.acponline.org
American Academy of Family Physicians (AAFP)	http://www.aafp.org
American Academy of Pediatrics (AAP)	http://www.aap.org
American Osteopathic Association (AOA)	http://www.do-online.org
American Thoracic Society (ATS)	http://www.thoracic.org

Private Payers and Technology Assessment

Private payers rely on a mix of both proprietary and publicly available technology assessments for their policy development; some have their own proprietary technology assessment process or purchase proprietary technology assessments produced by independent, often for-profit, companies. An example of an independent, nonprofit organization that publishes technology assessments is the Institute of Clinical Systems Improvement (ICSI), with 56 members funded by all 6 of the health plans in Minnesota. The Blue Cross/Blue Shield Association (BCBSA) produces technology assessments through an EPC, the Technology Evaluation Center (TEC). TEC assessments are available to the public at their website and are distributed to all of the Blues plans as a common starting point for medical policy coverage development. Technology assessment organizations, whether developed by specific payers or independent companies, variably engage specialty societies or local physicians for their input. Payers typically convene a committee to consider either privately or publicly the findings of an assessment; consumers, specialists, and practicing physicians may serve as contributors to the assessment itself, as members of an assessment committee, or as majority or minority voting members on a technology assessment. There is some concern that private payers are vulnerable to financial pressures. A pointed summary from the 2004 Consensus Conference of the American Society for Bariatric Surgery (ASBS) suggested that insurance providers are “unwilling to invest in expensive treatments that yield no profits until after 4 years because 4 years is the median time for an individual to remain with an insurance company.”37 Whether such views are substantiated, it remains important to acknowledge the potential for vested interests.

Professional Societies

Professional societies oversee the utilization of new technologies that fall under their domain, and they frequently provide checks and balances on other decision-making entities. Their Web sites can serve as a forum for hotly debated, complex topics that affect patient safety (Table 3).38,39 Indeed, professional societies have often taken the lead in endorsing evidence-based reviews. For example, the American College of Physicians (ACP) announced its collaboration with the American Academy of Family Physicians (AAFP), the American Academy of Pediatrics (AAP), and the American Osteopathic Association (AOA) to release joint principles for a “Patient-Centered Medical Home,” a program in which evidence-based medicine serves clinical decision making.40 The American Thoracic Society (ATS) has been a strong advocate of applying and refining standard criteria for conducting systematic reviews with the aim of developing practice guidelines.41,42 Professional societies not only provide a format for discussion, they also work toward improving the overall quality of debate. However, as with other evaluative entities, they are not immune to pressures and financial conflicts of interest that may influence their recommendations.43,44

Two examples of professional society recommendation serve to highlight the potential for member self-interest to influence the evaluation of novel medical devices. In May of 2006, an article in the lay press questioned patient testimonials on a manufacturer’s Web site extolling a new procedure, balloon dilation of the sinuses, a minimally invasive alternative to sinus surgery. This challenge was officially echoed in a position statement by the American Rhinologic Society (ARS), claiming that in spite of FDA approval, “... the scientific literature has no data on the technology for long-term safety, indications, efficacy and outcomes.”45–47 What the ARS did not say is that some of their members might experience a decline in income if the balloon dilation procedure were established as the standard of care over surgery. Likewise, intensity-modulated radiation therapy (IMRT), an alternative to conventional 3D-conformal radiation therapy (3D-CRT), has been broadly embraced by the radiation oncology professional organizations for treatment of prostate cancer despite the lack of any comparative studies in humans and no FDA review process of specific applications of IMRT for prostate cancer. The embrace of IMRT may have been in part driven by its favorable reimbursement compared to that of standard therapies for prostate cancer.48

The safety and effectiveness of medical devices are not yet comprehensively evaluated by the FDA, other federal entities, payers, or professional societies. Therefore, a stance of critical self-analysis will remain crucial while advocating innovation. Technology assessments by objective, independent, not-for-profit entities are necessary so that deficiencies in the clinical evidence and potential threats to patient safety are recognized in a manner that counterbalances market forces driving diffusion and demand.

A Case Study: The California Technology Assessment Forum

The California Technology Assessment Forum (CTAF) is an independent, nonprofit entity whose mission is to provide transparent, objective reviews of new medical devices to better educate the public, clinicians, and policymakers (see http://www.ctaf.org). The CTAF reviews are intended to complement the role played by professional societies and the FDA. Recent topics have included full-field digital mammography for breast cancer screening, gene expression profiling for the diagnosis of heart transplant rejection, wireless capsule endoscopy, and device-controlled breathing as a treatment for hypertension.

CTAF contracts with general internists at academic medical centers to provide about 15 systematic reviews of new and emerging medical technologies per year. Topics are selected through a consensus process based on impact of the technology and availability of relevant clinical data by an ad hoc committee consisting of representatives from the CTAF panel (see below), the generalist physician consultants and the Blue Shield of CA health plan. Potential topics may be brought to the attention of the committee by all potential stakeholders: the health plan, industry, professional societies, consumer groups, or CTAF staff. Although the subject of these reviews is often highly specialized and technical, CTAF believes that generalist physicians are better suited to provide objective, evidence-based assessments than are subspecialists who, in spite of their superior technical knowledge, may be more likely to have preexisting biases or opinions. Three times per year, CTAF invites manufacturers, patients, clinicians, and payers to openly debate the safety and effectiveness of 4 to 5 new devices; these discussions typically include input ranging from anecdotal patient experience to expert testimony from the principal investigators of the pivotal randomized controlled trials.

CTAF reviews are systematic analyses of the peer-reviewed literature that synthesize the results of all published clinical trials, case series, and randomized clinical trials, with an emphasis on comparative trials. The MEDLINE database, Cochrane clinical trials database, Cochrane reviews database, and the Database of Abstracts of Reviews of Effects (DARE) are searched using relevant key words from 1966 up to the present. In addition, the bibliographies of systematic reviews and key articles are manually searched for additional references. Abstracts of citations are reviewed and all relevant articles reviewed in full. Each assessment is researched and written by the lead consultant with consultation and input from the other consultants as needed and are, generally, 25,000 to 30,000 words in length.

The consultant must determine whether each new technology selected for review meets 5 criteria (see Table 4) according to a consistent, reproducible approach with a visible line of reasoning. The technology assessment should specify critical patient-centered outcomes, as well as identify which data are available that will ideally evaluate these outcomes, and where there is insufficient data. CTAF reviews are sent to subspecialty experts and to professional societies for critical appraisal and comment. Members from both of these groups are invited to the CTAF meetings to contribute to the discussion in a public forum. Other interested parties from the government (e.g., CMS), health care organizations, academic centers, and the community are also invited to attend and participate in the discussion. Efforts are also made to invite those who are advocates of a particular device (such as manufacturers and inventors), as well as others who may be against it. The open meeting is a unique feature of the CTAF assessment protocol.

Table 4

California Technology Assessment Forum Assessment Criteria

Number	Description	Comments
TA 1	The technology must have final approval from the appropriate government regulatory bodies	Approval via 510(k) process is streamlined for new devices that are similar to older devices
TA 2	The scientific evidence must permit conclusions concerning the effectiveness of the technology regarding health outcomes	Range: Levels 1–5, e.g., level 1, randomized, controlled trials powered to demonstrate clinically significant outcomes; level 5, case series without controls
TA 3	The technology must improve net health outcomes	Diagnostic tests must change management in ways that benefit patients
TA 4	The technology must be as beneficial as any established alternatives	It remains unclear how to evaluate diagnostic/prognostic tests lacking a gold-standard reference
TA 5	The improvement must be attainable outside the investigational setting	By definition, trials are conducted within an investigational setting, and therefore this criterion implies a speculation about potential, and not actually studied, applications

The voting panel includes experts from a variety of disciplines: ethicists, consumer advocates, practicing clinicians, methodologists, and others. The consultants present a brief overview of the peer-reviewed literature to the panel and then recommend that the technology either meets or does not meet the 5 criteria for approval. If the recommendation cannot be accepted as stated, the panel then entertains and votes on alternatives. Decisions are made by a simple majority vote; consensus is not required. Approximately 25% of new technologies are approved (i.e., found to meet all 5 CTAF criteria for efficacy and safety). There is no formal appeal process, but technologies will be assessed again when more data become available and frequently are approved at that time. CTAF does not have the ability to formally track the costs or impact of new technology it has reviewed.

LIMITATIONS OF EVIDENCE-BASED TECHNOLOGY ASSESSMENT

A general limitation of technology assessment is the lack of high-quality data derived from randomized clinical trials. There are often no RCTs to provide an ideal level of evidence for decision making. As described above, the 510(k) process permits most devices to be marketed in the absence of high-quality studies. Moreover, manufacturers, researchers, clinicians, and other stakeholders frequently have little incentive to conduct the kind of studies needed to answer the most important or relevant clinical questions. The device industry and professional societies often exert significant influence on which devices are evaluated and how they are used. However critical, assessment of the evidence by these groups may be of limited value because of real or potential conflicts of interest. The ties to industry funding also vary among the assessment groups listed in Tables 1, 2, and 3, and in some cases, assessments are provided on a for-profit basis. In short, no immunity from conflicts of interest is conferred upon any of the stakeholders in this process.

Discrepancies between assessment groups’ conclusions can and do occur due in part to different values used to weight the evidence and different methodologies (e.g., CTAF relies solely on published peer-reviewed literature only for their reviews, whereas another group may include unpublished data). However, the occasional discrepancy can stimulate useful debate. Perhaps more problematic is the fact that the available evidence-based technology assessments remain poorly disseminated. All stakeholders—physicians, patients, policy makers, FDA advisory committee members, and manufacturers—stand to benefit from better access to assessments like those provided by groups such as TEC and CTAF, and from well-designed multicenter RCTs that can inform practice.

CONCLUSION

As a result of the proliferation, complexity, and magnitude of impact of new and emerging medical devices, there is an urgent need to monitor new technology ever more closely in terms of effectiveness and safety. In spite of this need, significant gaps persist in the scientific literature, in the medical device approval process, and in the realm of postmarketing surveillance. While most pharmaceuticals obtain FDA approval only after analysis of randomized clinical trials, relatively few new medical devices are subject to comparable scrutiny. To improve health outcomes, we must enhance our understanding of new medical devices and, by not simply deferring to the FDA, support independent, evidence-based technology assessments. However, given the current process that allows for the vast majority of new devices to enter the market under a 510(k), once a device is introduced and used, the ability to eliminate inappropriate use, even with rigorous TA, is nearly impossible. In spite of this limitation, technology assessments by independent organizations after FDA approval can help identify those medical technologies that are truly beneficial and safe. Furthermore, such assessments may motivate research designed to answer remaining questions about emerging technologies and can serve to educate the public and health professionals about the potential promise and pitfalls of new technology.