Challenges Associated with the Safety Signal Detection Process for Medical Devices.

BACKGROUND: Previous safety issues involving medical devices have stressed the need for better safety signal detection. Various European Union (EU) national competent authorities have started to focus on strengthening the analysis of vigilance data. Consequently, article 90 of the new EU regulation states that the European Commission shall put in place systems and processes to actively monitor medical device safety signals.
METHODS: A systematic literature review was conducted to synthesize the current state of knowledge and investigate the present tools used for medical device safety signal detection. An electronic literature search was performed in Embase, Medline, Cochrane, Web of science, and Google scholar from inception until January 2017. Articles that included terms related to medical devices and terms associated with safety were selected. A further selection was based on the abstract review. A full review of the remaining articles was conducted to decide on which articles finally to consider relevant for this review. Completeness was assessed based on the content of the articles.
RESULTS: Our search resulted in a total of 20,819 articles, of which 24 met the inclusion criteria and were subject to data extraction and completeness scoring. A wide range of data sources, especially spontaneous reporting systems and registries, used for the detection and assessment of product problems and patient harms associated with the use of medical devices, were studied. Coding is remarkably heterogeneous, no agreement on the preferred methods for signal detection exists, and no gold standard for signal detection has been established thus far.
CONCLUSION: Data source harmonization, the development of gold standard signal detection methodologies and the standardization of coding dictionaries are amongst the recommendations to support the implementation of a new proactive approach to signal detection. The new safety surveillance system will be able to use real-world evidence to support regulatory decision-making across all jurisdictions.

Introduction

Signal detection is defined by the International Medical Device Regulators Forum (IMDRF) as “The process of determining patterns of association or unexpected occurrences that have the potential to impact patient management decisions and/or alter the known benefit-risk profile of a device.1”

The aim of safety signal detection for medical devices is to promptly identify risks associated with the use of a product.2 Signals can be production related (eg, a defective batch or a released series of batches) or linked to the design and/or use. Signals can be identified during the pre-market surveillance phase using clinical trial data, or during the post-market surveillance phase using post-market data sources. The decision of whether a finding represents a “signal” and whether such finding is subject to further investigation can be challenging.

For medicinal products, quantitative safety signal detection is followed by a signal validation process during which the signal is verified to be real or not. This process is often performed through careful case evaluation. Thereafter, signal prioritization is completed depending on the strength of the signal, whether or not the signal represents a new finding, the clinical importance and potential public health implications, and the availability of preventive measures to mitigate the adverse public health impact.3

After prioritization, the marketing authorization holder together with the regulators has to decide whether additional risk minimization measures are needed to address this safety issue.2

Although the signal management for medicinal products and medical devices are conceptually equivalent, the legislation requirements are better described in the pharmaceutical regulation4–6 than in the medical devices regulations. For the latter, the guidelines defining requirements and giving practical advice on signal management are yet to be developed.

Recent safety issues involving medical devices have highlighted the need to improve signal detection.7 Various European Union (EU) national competent authorities have started to focus on strengthening the analysis of vigilance data of medical devices. As a consequence, the new EU medical device regulation was published; namely, article 90 that states that the European Commission shall put in place systems and processes to actively monitor the data available in order to identify trends, patterns or signals that may reveal new risks or safety concerns.8

In this paper, we aim to describe aspects that influence signal detection of safety issues related to medical devices in order to identify gaps and provide recommendations for optimizing signal detection approaches.

Methods

We performed a systematic literature review to identify articles describing different aspects associated with safety signal detection for medical devices (see Table 1). We searched Embase, Medline, Cochrane, Web of science, and Google scholar using terms that included “medical device” and terms attributable to safety (“signal detection” or ”post-marketing surveillance” or “risk management”), following Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for systematic reviews. outlines the strings that have been used.

Table 1

Systematic Literature Review Methodology

Step	Inclusion Criteria	Exclusion Criteria	Number of Articles
Electronic Search	Embase, Medline, Cochrane, Web of science, and Google scholar using terms that included “medical device” and terms attributable to safety (“signal detection” or “post-marketing surveillance” or “risk management”).	Articles that did not include “medical device” or terms attributable to safety (“signal detection” or “post-marketing surveillance” or “risk management”).	20,819
Restrictive Query Search	Any abstracts from the list of selected articles (20,819) with years ranging 2004–2017 (Jan-2004 through Jan-2017) and containing any of the following terms: “Signal” OR “Adverse reaction” OR “Adverse event” OR “Injury” OR “Malfunction” OR “Product problem”.	Any abstracts from the list of selected articles (20,819) that did not include any of the following terms: “Signal” OR “Adverse reaction” OR “Adverse event” OR “Injury” OR “Malfunction” OR “Product problem”.	996
Abstract Review	Articles were included if the abstract review (996) contained any of the following items: “post-market safety data sources in medical devices” OR “signal detection methodologies for medical devices” OR “medical device event coding dictionaries”.	Articles were excluded if the abstract review (996) did not include any of the following items: “post-market safety data sources in medical devices” OR “signal detection methodologies for medical devices” OR “medical device event coding dictionaries”.	45
Full-Text Review	Articles (45) were reviewed and selected if the article included any information related to “medical device Post-Market Surveillance (PMS) data sources” OR “Methodologies used for signal detection for medical devices” OR “Coding dictionaries for medical devices”.	Articles (45) were excluded if the article did not include any information related to “medical device Post-Market Surveillance (PMS) data sources” OR “Methodologies used for signal detection for medical devices” OR “Coding dictionaries for medical devices”.	24

Due to the high number of articles that resulted from the initial search, we decided to implement a restrictive query search on the abstracts from the list of selected articles in order to narrow down the selection. This restrictive query consisted of: any abstract with date range year 2004–2017 (Jan-2004 through Jan-2017) containing any of the following terms: “Signal” OR “Adverse reaction” OR “Adverse event” OR “Injury” OR “Malfunction” OR “Product Problem”.

Review of Articles

Following the query, all remaining abstracts were reviewed. Articles were excluded if the abstract review did not include any of the following items: “post-market safety data sources in medical devices”, “signal detection methodologies for medical devices” or “medical device event coding dictionaries”. Articles containing the latter terms were included in the further study. Subsequently, a full-text review was conducted for each of the remaining articles. Articles were excluded if they did not include any information related to “medical device Post-Market Surveillance (PMS) data sources” OR “Methodologies used for signal detection for medical devices” OR “Coding dictionaries for medical devices”.

All remaining full-text articles were subject to a formal evaluation to extract information on the following items (articles that did not contain at least one of the following items were excluded):

1. Type of PMS data sources on medical devices:

-Spontaneous reporting systems (SRS) are reactive systems that contain reports on patient harms and product problems collected from healthcare professionals, patients, healthcare authorities and manufacturers whether reported directly or through published articles.

-A medical device registry is defined by the IMDRF as an “Organized system with as primary aim to increase the knowledge on medical devices contributing to improve the quality of patient care that continuously collects relevant data, evaluates meaningful outcomes and comprehensively covers the population defined by exposure to particular device(s) at a reasonably generalizable scale. (eg international, national, regional, and health system)”1

2. Methodologies used for signal detection for medical devices.

3. Coding dictionaries for medical devices.

Each article was scored 1, 2 or 3 points depending on its content. The total score represents the sum of all three topics; 1 point being attributed to articles containing one of the three topics, 2 points being attributed to articles containing two of three topics, and finally, 3 points being attributed to articles containing all three topics. This score serves as a measure of the articles’ completeness.

Following the author’s full-text review, KV conducted a second review of the full-text articles. KV agreed with the initial selection of the 24 articles, and the assigned score based on the described inclusion and exclusion criteria.

Results

Our initial search strategy identified a total of 20,819 articles (10,199 Embase, 8,374 Medline Ovid, 1,501 Web of Science, 545 Cochrane and 200 Google Scholar). After applying specific search restrictions, a total of 996 abstracts were identified. During the abstract review, 951 articles were excluded, due to lack of information on any of the three key contents: post-market data sources for medical devices, signal detection methodologies for medical devices and coding dictionaries for medical devices. A total of 45 articles were included for full-text review. During the review cycle, 21 articles were excluded due to the lack of information on any of the three key topics. Details of the 24 remaining articles are shown in Table 2.

Table 2

Characteristics of the 24 Selected Articles

Title of Article	Type of Article	Authors	Journal	Year of Publication	Study Period	Number of Participants or Number of Reports	Post-Market Data Sources Medical Devices	Signal Detection Methodology for Medical Devices	Coding Dictionary for Medical Devices	Completeness Score
Adverse event Triggered Event Reporting for Devices:Report of a Food and Drug administration-supported feasibility pilot of automated adverse event reporting15	Review article, feasibility study	Reed, T. L.Levy, D.Steen, L. T.Roach, J.Taylor, T.Call, K.Marion, J.Drozda, J. P.	J Clin Eng	2016	NA	NA	Yes, Adverse Event Triggered Reporting for Devices (ASTER-D)	No	No	1
An analysis of implantable cardiac device reliability. The case for improved post-marketing risk assessment and surveillance21	Retrospective study	Laskey, W.Awad, K.Lum, J.Skodacek, K.Zimmerman, B.Selzman, K.Zuckerman, B.	Am J Ther	2012	January 2003- December 2007	256,392 CRT-D=≤ 1,925 malfunctioning; 459,000 ICD=≤ 10,593 malfunctioning	Yes, FDA annual reports (ICDs and CRT-D implants)	No	No	1
An evaluation of a distributed medical device safety surveillance system: The DELTA network study24	Multicenter prospective study	Vidi, V. D.Matheny, M. E.Donnelly, S.Resnic, F. S.	Contemp Clin Trials	2011	January 2010 - December 2011	Not specified but this is a paper discussing a study that still needs to be performed	Yes, American College of Cardiology’s National Cardiovascular Registry (ACC-NCDR)	Yes, DELTA	No	2
Analysis of the US Food and Drug Administration Manufacturer and User Facility Device Experience database for adverse events involving Amplatzer septal occluder devices and comparison with the Society of Thoracic Surgery congenital cardiac surgery database16	Retrospective study	DiBardino, D. J.McElhinney, D. B.Kaza, A. K.Mayer Jr, J. E.	J Thorac Cardiovasc Surg	2009	January 2002 −30 June 2007	274 AE reports in MAUDE and 1537 cardiac surgery	Yes, MAUDE database	No	No	1
Application of a temporal reasoning framework tool in analysis of medical device adverse events11	Retrospective study	Clark, K. K.Sharma, D. K.Chute, C. G.Tao, C.	AMIA Annu Symp Proc	2011	January 2009 to December 2010	15 reports were selected	Yes, MAUDE database	No	No	1
Automated surveillance to detect post procedure safety signals of approved cardiovascular devices26	Retrospective study	Resnic, F. S.Gross, T. P.Marinac-Dabic, D.Loyo-Berrios, N.Donnelly, S.Normand, S. L. T.Matheny, M. E.	J Am Med Assoc	2010	April 2003 - October 2017	74,427 interventional coronary procedures	Yes, Massachusetts angioplasty registry	Yes, DELTA	No	2
Cochlear implant complications: utility of federal database in systematic analysis12	Retrospective study	Tambyraja, R. R.Gutman, M. A.Megerian, C. A.	Arch Otolaryngol Head Neck Surg	2005	Pre 1998 and 2002	654 events on cochlear implant in 2002 and 129 pre-1998	Yes, MAUDE database	Yes, DPA; Frequentist (chisquare)	No	2
Disproportionality analysis for signal detection of implantable cardioverter-defibrillator-related adverse events in the Food and Drug Administration Medical Device Reporting System19	Review article	Duggirala, H. J.Herz, N. D.Caños, D. A.Sullivan, R. A.Schaaf, R.Pinnow, E.Marinac-Dabic, D.	Pharmacoepidemiol Drug Saf	2012	NA	NA	Yes, MAUDE database	Yes, DPA - Bayesian (MGPS)	No	2
Early detection of an underperforming implantable cardiovascular device using an automated safety surveillance tool22	Retrospective study	Hauser, R. G.Mugglin, A. S.Friedman, P. A.Kramer, D. B.Kallinen, L. M.McGriff, D. M.Hayes, D. L.	Circ Cardiovasc Qual Outcomes	2012	November 2001 until December 2008	1035 Fidelis and 1675 Quattro leads	Yes, database of Sprint Fidelis and Quattro Secure implantable cardioverter defibrillator leads	Yes, DELTA	No	2
Estimates of medical device-associated adverse events from emergency departments10	Retrospective study	Hefflin, B. J.Gross, T. P.Schroeder, T. J.	Am J Prev Med	2004	July 1999 through June 2000	10,395 medical device–associated adverse events	Yes, National Electronic Injury Surveillance System (NEISS)	No	No	1
Evaluating the Safety Profile of Non-Active ImplantableMedical Devices Compared with Medicines14	Review article	Pane, J.Coloma, P. M.Verhamme, K. M. C.Sturkenboom, M. C. J. M.Rebollo, I.	Drug Safety	2017	NA	NA	Yes, MAUDE, MEDSUN,MHRA, SCAAR, EUREQUO, AOANJRR,	No	Yes, Product problem: FDA and ISOPatient Outcome: SNOMED, MedDRA, ICD, FDA	2
FDA adverse event problem codes: Standardizing the classification of device and patient problems associated with medical device use32	Review article	Reed, T. L.Kaufman-Rivi, D.	Biomed Instrum Technol	2010	NA	NA	No	No	Yes, FDA Adverse Event Problem Codes	1
FDA MAUDE data on complications with lasers, light sources, and energy-based devices17	Review article	Tremaine, A. M.Avram, M. M.	Lasers Surg Med	2015	NA	NA	Yes, FDA MAUDE	No	No	1
Identifying optimal post-market surveillance strategies for medical and surgical devices: Implications for policy, practice and research30	Review article	Gagliardi, A. R.Umoquit, M.Lehoux, P.Ross, S.Ducey, A.Urbach, D. R.	BMJ Qual Saf	2013	NA	NA	No	Yes, optimal post-market surveillance strategies for medical and surgical devices	No	1
Issues with medical device spontaneous reporting and improvements through MedSun13	Review article	Ostuni, M.	Drug Inf J	2010	NA	NA	Yes, MEDSUN	No	No	1
Medical device regulation in Australia: Safe and effective?18	Review article	McGee, R. G.Webster, A. C.Rogerson, T. E.Craig, J. C.	Med J Aust	2012	NA	NA	Yes, TGA	No	No	1
Methods and Issues to Consider for Detection of Safety Signals From Spontaneous Reporting Databases: A Report of the DIA Bayesian Safety Signal Detection Working Group31	Retrospective study	Gould, A. L.Lystig, T. C.Lu, Y.Fu, H.Ma, H.	Ther Innov Regul Sci	2015	Not specified	Not specified	No	Yes, review article that describes methods for signal detection	No	1
Mini-Sentinel’s systematic reviews of validated methods for identifying health outcomes using administrative data: summary of findings and suggestions for future research28	Review article	Carnahan, R. M.	Pharmacoepidemiol Drug Saf	2012	NA	NA	Yes, Medicare	No	No	1
Participatory surveillance of diabetes device safety: A social media-based complement to traditional FDA reporting29	Retrospective study	Mandl, K. D.McNabb, M.Marks, N.Weitzman, E. R.Kelemen, S.Eggleston, E. M.Quinn, M.	J Am Med Informatics Assoc	2014	October 2011 to September 2012	549 diabetes patients with device, 75 adverse events	Yes, online social networking community of people with diabetes and their caregivers or family members	No	No	1
Postmarketing surveillance of medical devices using medicare claims25	Retrospective study	Malenka, D. J.Kaplan, A. V.Sharp, S. M.Wennberg, J. E.	Health Aff	2005	1 February and 31 November 2001	23.049	Yes, Medicare	No	Yes, ICD-9	2
Signal detection using change point analysis in postmarket surveillance20	Review article	Xu, Z.Kass-Hout, T.Anderson-Smits, C.Gray, G.	Pharmacoepidemiol Drug Saf	2015	NA	NA	Yes, FDA MAUDE	Yes, multivariate method; change point analysis	No	2
The Data Extraction and Longitudinal Trend Analysis network study of distributed automated post-market cardiovascular device safety surveillance23	Retrospective study	Kumar, A.Matheny, M. E.Ho, K. K. L.Yeh, R. W.Piemonte, T. C.Waldman, H.Shah, P. B.Cope, R.Normand, S. L. T.Donnelly, S.Robbins, S.Resnic, F. S.	Circ Cardiovasc Qual Outcomes	2015	January 2008 through December 2012	23,333 PCI cases	Yes, National Cardiovascular Data Registry (NCDR)	Yes, DELTA	No	2
The Kaiser Permanente implant registries: effect on patient safety, quality improvement, cost effectiveness, and research opportunities27	Retrospective study	Paxton, E. W.Inacio, M. C.Kiley, M. L.	Perm J	2012	Not mentioned	Not mentioned	Yes, Kaiser Permanente Orthopedic implant registries	No	Yes, ICD-9	2
Towards a Prototype Medical System for Devices Vigilance and Patient Safety9	Review article	Deligiannakis, A.Giatrakos, N.Pallikarakis, N.	2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health	2014	NA	NA	Yes, MEdical DEvices VIgilance and Patient Safety (MEDEVIPAS)	Yes, multivariate method; entity matching algorithm	No	2

Papers were categorized into two different categories: 11 review articles, and 13 studies (12 retrospective studies and 1 prospective study). Completeness scoring yielded 13 articles with a score of 1, 11 with a 2 point score, whereas no article scored a 3 point rating. Twenty-one articles included information on post-market data sources of medical devices, 10 articles included information on signal detection methodologies for medical devices and 4 articles included information on coding dictionaries for medical devices (Figure 1).

Figure 1

PRISMA flow diagram outlining all steps for the inclusion of articles in the review.

PMS Data Sources

Of the 21 articles including post-market data sources, 12 articles discussed SRS (one of the articles also included information on medical device registries), 9 articles discussed medical device registries (one of the articles also included information on SRS) and 1 article described a Non-Standard Data Source.

Of the 12 articles including different SRS, the following SRS were discussed: FDA MAUDE database (US), TGA DAEN database (Australia), the future European Databank on Medical devices (Eudamed) (EU), MHRA database (UK), MEDSUN database (US), Adverse Event Triggered Reporting for Devices (ASTER-D) (US), MEdical DEvices VIgilance and Patient Safety (MEDEVIPAS) (Greece), and the National Electronic Injury Surveillance System (NEISS) (US)9–20 (see Table 2).

Of the nine articles including registries, the following were discussed: American College of Cardiology’s National Cardiovascular Registry (US), Massachusetts Angioplasty Registry (US), Kaiser Permanente Orthopedic Implant registries (US), National Cardiovascular Data Registry (NCDR) (US), database of Sprint Fidelis and Quattro Secure implantable cardioverter defibrillator leads (US), Swedish Coronary Angiography and Angioplasty Registry (SCAAR) (Sweden), European Registry of Quality Outcomes for Cataract and Refractive Surgery (EUREQUO) (EU), Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) (Australia), Data Extraction and Longitudinal Trend Analysis (DELTA) Registry (US), and Medicare database (US claims database constituting a person-specific registry of medical histories recording the use of all hospital services that are eligible for payment, including use of medical devices)14,21–28 (see Table 2). One article described a non-standard data source, namely, an online social networking community of people with diabetes and their caregivers or family members. This diabetes safety network captured data entered by patients in apps (see Table 2) and contained patient case reports of medical device events.29

Signal Detection Methodologies

Ten articles described safety signal detection methodologies for medical devices, four articles discussed signal detection methodologies applied to SRS,9,12,19,20 four articles discussed signal detection methodologies applied to registries,22–24,26 and two articles discussed optimal signal detection methodologies for medical devices without applying the methodology to a specific type of PMS data source.30,31 No articles associated with signal detection methodologies applied to non-standard data sources were identified.

Of the four articles using signal detection methodologies applied to SRS, two articles discussed disproportionate analysis (DPA) methodologies (Frequentist and Bayesian),12,19 and two articles discussed multivariate methods (change point analysis and entity matching algorithm).9,20 Of the four articles that included signal detection methodologies applied to registries, all four articles discussed methodologies associated with the Data Extraction and Longitudinal Trend Analysis (DELTA) network.22–24,26

Coding Dictionaries

Of the four articles that included information on coding dictionaries for medical devices, different dictionaries and nomenclatures were used, namely, FDA codes and International Organization for Standardization [ISO], IMDRF codes for product problems and investigation results, and Systematized Nomenclature of Medicine-Clinical Terms [SNOMED CT], MedDRA_, International Classification of Diseases [ICD], FDA Patient Problem and IMDRF Patient Codes for patient outcomes.14,25,27,32

Discussion

From this review, it is clear that spontaneous reporting systems and registries are primarily used for the medical device safety signal detection. Coding is remarkably diverse, no agreement on the preferred methods for signal detection currently exists, and no gold standard for signal detection has been established thus far.

The main publicly available SRS are the FDA MAUDE (US), TGA DAEN (Australia) and the future EU Eudamed (EU)33 (see Table 3; available PMS data sources for medical devices). The organization and content of each SRS varies, some are based on voluntary reporting and others on mandatory reporting, and usually track suspected medical devices, suspected patient harms or product problems, and patient data collected in a centralized and structured format.13

Table 3

Available PMS Data Sources for Medical Devices

Type of Available PMS Data Source	Database
Spontaneous Reporting Systems9–20	MAUDE (US, FDA), DAEN (Australia, TGA), Eudamed (EU, EC)
Registries14,21–24,26,27	Orthopedic: NJR (England, Wales and Northern Ireland), CJRR (Canada), KPOR (US), LROI (Netherlands), RNI (Brazil), AOANJRR (Australia) Vascular: VQI (US), AVA registry (Australia and New Zealand), NVR (UK), JREAR (Japan) Cardiac: SCAAR (Sweden), J-PCI (Japan), Cath-PCI (US), the US TVT (US), Japanese TVT (Japan), JACVSD (Japan), Ophthalmology: EUREQUO (EU)
Non-Standard Sources29	Safety networks: diabetes device safety network Social networks: twitter, facebook, instagram, LinkedIn Software devices: data entered by patients in mobile applications

Abbreviations: MAUDE, Manufacturer and User Facility Device Experience; DAEN, Database of Adverse Event Notifications; TGA, Therapeutics Goods Administration; FDA, Food and Drug Administration; EU, European Union; EC, European Commission; NJR, National Joint Registry; CJRR, Canadian Joint Replacement Registry; KPOR, Kaiser Permanente Orthopedic Registry; LROI, Dutch Arthroplasty Registry; RNI, National Implants Registry; AOANJRR, Australian Orthopaedic Association National Joint Replacement Registry; SCAAR, Swedish Coronary Angiography and Angioplasty Registry; VQI, Vascular Quality Initiative; AVA, Australasian Vascular Audit; NVR, National Vascular Registry; JREAR, Japanese Registry of Endovascular Aneurysm Repair, abdominal and thoracic; PCI, percutaneous coronary intervention; TVT, trans-catheter valve therapies; JACVSD, Japan Adult Cardiovascular Surgery Database; EUREQUO, European Registry of Quality Outcomes for Cataract and Refractive Surgery.

Per our literature review, the identified SRS are organized based on the relationship between medical devices and events. The data is available for assessment and located in a repository or database.11,12,16,17,21 Nevertheless, SRS suffer from different limitations including: lack of harmonized global standard data set for reporting which makes integration of data from different databases challenging,14,34 difficulty to determine root causes for individual events conclusively due to limited information and no access to the actual device, with a large part of investigation results being inconclusive,35 missing and incomplete data that impacts the evaluation of the case, underreporting due to different reasons including lack of time, uncertainty about the medical device causing the adverse event, difficulty in accessing reporting forms, lack of awareness of the requirements for reporting, and lack of understanding of the purpose of SRS,36 and over-reporting where medical devices with well-known adverse event/product problems are more likely to be reported based on influence from media coverage – the so-called notoriety bias.37

Despite SRS being a standard and required source for signal detection, we identified that medical device registries are important for signal detection as well and may also be used for risk quantification. Registries typically contain valuable information such as medical device information, diagnoses, medications, medical narratives and surgical interventions. Unlike spontaneous reports, medical device registries are not restricted to patients experiencing medical device product problems or patient harms. Therefore, medical device registries data provide some advantages that can be used to complement the more traditional PMS data sources (SRS), particularly the possibility to perform active PMS. In our literature search, we found that some retrospective studies have demonstrated the feasibility of an early warning detection system using medical device registries. For example, it has been demonstrated that the fracture of the Fidelis implantable cardioverter defibrillator (ICD) leads that caused inappropriate ICD shocks could have been detected much earlier if a medical device registry would have been created.19,22

Our literature review identified different types of methodologies (depending on the type PMS data source used) that can be applied to calculate reporting associations for all medical device-event combinations. Disproportionality analysis (DPA) was used as the main signal detection method for SRS – some used frequentist and other Bayesian approaches.12,19 These methods are well established for signal detection in drug safety. For complex types of SRS analysis, multivariate approaches have been proposed: change point analysis20 or entity matching algorithm.9 These are not yet used for medical device safety signal detection. Methods applied to medical device registries can be categorized into those based on modified DPA ported from spontaneous reporting, and those based on the DELTA network methodology.22 Signal detection methods applied to medical device registries based on the Data Extraction and Longitudinal Trend Analysis (DELTA) network are considered automated safety surveillance tools that can competently support the detection of new potential post-market safety issues,15 complementing existing signal detection strategies and providing an additional tool to evaluate the safety of marketed medical devices.26

Our literature review demonstrated that different types of coding dictionaries used for medical device signal detection currently exist; for patient harms, product problems and evaluation/investigation codes (methods, results and conclusion of the investigation). These coding systems are very heterogeneous. We conclude that there is a lack of standardization of medical device event coding across different jurisdictions. Furthermore, no mapping between some of the coding dictionaries currently exists. This issue could eventually delay the timely generation of safety signals associated with a medical device event reported in jurisdictions using different event coding dictionaries without an appropriate event code mapping.

Recommendations

Based on the analysis of the current literature on safety signal detection for medical devices and their limitations we have developed some recommendations (see Table 4):

Table 4

Limitations and Recommendations on Improving the Signal Detection for Medical Devices

	Limitation	Recommendation	Owner
PMS data sources: defragmentation and harmonization	SRS: Lack of a global database of spontaneous reports and lack of harmonization of required data set and reporting requirements across jurisdictions.	SRS: Creation of a global database of medical device spontaneous reports from national/regional databases to maximize the potential of data captured in Spontaneous Reports Databases.SRS: Define a common standardized data set for reporting individual device cases to be able to link global data, and develop consistent reporting requirements across jurisdictions to ensure the same type of reportable spontaneous reports are received globally.	IMDRF, HAs, WHO
	Registries: Lack of harmonization of medical device registries	Registries: Harmonization of registry networks by using the international Coordinated Registry Network (iCRNs).Registries: Define common data elements to be able to link the data.	IMDRF, HAs, WHO
Agreement on signal detection methodologies	There is no gold standard for the methodologies used for medical device signal detection.	Develop guidance on gold standard methodologies used to mine data from the different types of PMS data sources.	IMDRF, HAs
Standardization of coding dictionaries	Lack of harmonization and consistency of event codes used for patient harm, device problem and device evaluation codes.	Coding harmonization across all jurisdictions. IMDRF coding dictionary should be the gold standard used for coding purposes. HAs should adopt this new coding dictionary or map their national coding dictionary to the IMDRF coding.	HAs
	IMDRF codes are very high level with many events categorized as “no code available”.	Develop IMDRF coding guidelines classified by therapeutic area, and additional IMDRF codes to increase specificity, when appropriate.IMDRF needs to ensure maintenance of the IMDRF coding dictionary, and establish the right balance between having meaningful event code categories but not too much granularity.	IMDRF

Abbreviations: HA, Health Authority; IMDRF, International Medical Device Regulators Forum; PMS, Post-Market Surveillance; SRS, Spontaneous Reporting Systems; WHO, World Health Organization.

PMS Data Sources: Defragmentation and Harmonization

Currently different national SRS exist; however, no global database to access spontaneous reports on medical devices has been introduced. The inconsistency in post-market reporting requirements between regions leads to different levels of completeness that makes a comparison between different SRS databases challenging. The most significant one is adverse event reporting exemption applications in the EU (amongst other exemptions, expected side-effects are not reportable in EU but subject to event trending,8 in Canada38 and Australia,18 however, no exemptions are applicable in the US39). Together with the standardization of SRS databases, harmonization of reporting criteria is needed. Otherwise, it will not be possible to identify signals from the National Competent Authority (NCA) SRS databases globally when some datasets completely exclude certain types of spontaneous reports. A multicomponent global database including reporting by manufacturers, clinicians and patients collecting spontaneous reports on adverse events related to medical devices, should be established for the assembly of all medical device reports from all National Spontaneous reports databases. The analysis of the collected data can then be performed by region or by country.

In addition, harmonization of medical device registries databases is recommended. This harmonization could be established by using the international Coordinated Registry Network (iCRNs) to maximize the potential of information collected in the international registries.1 The International Consortium of Orthopedic Registries (ICOR) is a good example of the effective use of a distributed safety data system with harmonized data definitions and data extraction followed by evaluating the data using innovative methodology across multiple national orthopedic registries. This decentralized structure system helps overcome issues related to security, operations, legal, and those related to patient privacy.40

In order to link and potentially merge the data received from the different PMS data sources (SRS or registries), it is also recommended that guidance on common data elements and a common standardized data set for reporting individual device cases are developed (both for SRS and registries), and adopted by the different stakeholders involved in the process of collection and extraction of safety data for signal detection purposes.

Agreement on Signal Detection Methodologies

At this time, there is no agreement on the preferred methods for medical device signal detection for each of the different PMS data sources (SRS, registries and nonstandard data sources), and thus no gold standard has yet been established. IMDRF and Health Authorities should work together to develop guidance on gold standard methodologies that should be used to mine data from the different types of PMS data sources (SRS, registries and nonstandard data sources).

Standardization of Coding Dictionaries

To ensure more efficient signal detection we recommend the global adoption of the IMDRF coding dictionary by all Health Authorities. All the existing coding dictionaries will need to be mapped to the IMDRF coding dictionary to allow for an efficient system of signal detection for medical devices. Taking into consideration that IMDRF codes are very high level with many events falling in the “no code available” category, we recommend IMDRF to develop a more granular level of coding developing additional IMDRF codes to increase specificity, when appropriate. In order to ensure consistency when selecting the appropriate codes, we also recommend the development of IMDRF coding guidelines classified by therapeutic area. IMDRF needs to ensure the maintenance of the coding dictionary, and establish the right balance between having meaningful event code categories while avoiding too much granularity.

To ensure successful implementation of this new system, a global harmonized system for Unique Device Identifiers (UDIs) needs to be implemented, the UDIs need to be consistently assembled within PMS data, and all stakeholders need to ensure a continual use of the SRS and registries, including UDIs. The establishment of a global medical device identification database will aid in accomplishing this objective. The identification of devices during the signal detection process will continue to be a hurdle until the UDI is standardized and widely utilized for some time. This is a long-term goal because it involves significant policy change. Active collaboration and support from all stakeholders will ultimately lead to the success of these recommendations.

Developments in Recent Years

PMS Data Sources

Recent research emphasizes that the underlying data received from the PMS data sources need to meet high-quality standards to ensure a timely safety signal generation. The authors of a recent case study describe PMS data as one of the main important publicly available SRS for medical device safety signal detection: FDA MAUDE.35 This research outlines that the completeness and the quality of the spontaneous reports in FDA MAUDE can be improved. The authors, furthermore, highlight the difficulty to determine root causes conclusively for individual events due to limited information, and no access to the actual medical device, with a large part of investigation results being inconclusive. Amongst others, it is recommended to address these challenges by considering the possibility of enriching FDA MAUDE PMS data with data from active PMS data sources such as medical device registries. In order to be able to link the registry data with the spontaneous reports, common standardized dataset including UDI should be created.

Signal Detection Methodologies

The developments regarding the applicability of new methods to the safety signal detection of medical devices have been a wide research topic over the past few years. The research in the area of passive safety surveillance (the data-mining methods used for disproportionality analysis of medical device–adverse event combinations from SRS) has become a main research focus area.

Recent developments associated with the signal detection methodologies used for medicinal products have been applied to medical device signal detection;41 for example, the likelihood ratio test (LRT) method that is applied to perform passive safety surveillance of medicines has now been successfully used to perform passive safety surveillance of medical devices. LRT is a frequentist method based on multiple 2x2 tables. It compares the reporting rate of different adverse events for a given drug or medical device of interest. The LRT method has successfully been applied for safety signal detection purposes to medical device SRS, and can also be used as spatial-cluster signal detection for an adverse event of interest from medical device registries and other databases that have patient-level geographical information. Moreover, the LRT method was compared to other frequentists and Bayesian methods, and found to be the most conservative method when evaluating the total number of detected safety signals, given its ability to control for false-positive safety signals.42,43

A big effort has been made in developing signal detection methods for medical device safety signal in passive safety surveillance. However, challenges still exist for the development of new active surveillance methods (statistical signal detection methods for medical device registries, and other longitudinal databases) for monitoring the safety of new medical devices over time. In medicines, this effort is currently being undertaken by the Observational Medical Outcomes Partnership (OMOP) and FDA Sentinel Initiative:

-OMOP: The OMOP is a public–private partnership involving the FDA, multiple pharmaceutical companies and healthcare providers. OMOP conducts methodological research on active drug safety surveillance by evaluating the performance of safety signal methods and their ability to identify true drug-adverse event associations. OMOP established a common infrastructure to collect different types of observational data from post-market data sources around the world, and successfully developed and implemented a large-scale signal detection methodology applied to medicines.44

-FDA Sentinel: The FDA Sentinel is an active surveillance program that was established in the US with the long-term objective to create a national electronic system for PMS of FDA-regulated medical products (drugs, vaccines, biologics and medical devices). Over time, Sentinel has developed the largest multisite distributed database in the world dedicated to medical product safety. This new approach can help public health officers (who depend on passive surveillance tools lacking in denominator information, ie, patient exposure data) in detecting safety signals related to medicines and medical devices, and therefore aid in the accurate comparative assessments of safety risks.45

The application of these methods in medical device safety signal detection may have the ability to address some of the challenges associated with active safety surveillance of medical devices. Further research is required to evaluate the potential applicability of these two initiatives to active safety surveillance of medical devices.

Coding Dictionaries

There have also been some initiatives to address some of the challenges associated with adverse event coding for medical devices. An IMDRF project has worked on linking IMDRF codes and MedDRA codes.46 Moreover, IMDRF has also developed the IMDRF adverse event terminology maintenance plan; a document describing how to add, modify or delete adverse event terms to the IMDRF coding dictionary.47 Although these projects have the potential to address some of the identified challenges, some work still needs to be completed. The development of IMDRF coding guidelines by therapeutic area, and the creation of additional IMDRF codes to increase the granularity of the IMDRF coding dictionary are crucial to enhance the current adverse event coding for medical devices.

Conclusions

We have shown that a wide range of PMS data sources, coding dictionaries and signal detection approaches are available for the detection and assessment of medical device problems and patient harms. Each of them offers unique opportunities that together can contribute to developing standards for robust, consistent and improved signal detection for medical devices.

New detection methodologies have been developed to utilize data that has not been used in the past, allowing for the introduction of new proactive models of medical device surveillance. Despite the increasing evidence of the benefits of medical device registries for the purpose of signal detection, spontaneous reports will remain a key data source of post-market device data and therefore a relevant source of potential signals. Standardized methods applied to similar data sources will be required. Data quality and coding harmonization will need to be improved and the UDI system will need to be fully implemented to benefit from the potential of proactive systems for the safety evaluation of medical devices. In order to succeed, all stakeholders involved in the PMS system must actively support each other and collaborate. This system will use real-world evidence to support regulatory decision-making across all jurisdictions.