Comparing sensitivity and specificity of pacemaker ID application and cardiac rhythm management device-finder application in identifying cardiac implantable electronic device manufacturer using chest radiograph - An observational study.

BACKGROUND: Smartphone-based applications to identify cardiac implantable electronic devices (CIED) are extremely useful in circumstances, where urgent device interrogation is needed, and a device identification card is not available. Few studies have provided insights regarding the utility of these applications. We have studied two widely available applications i.e., Pacemaker ID app (PMIDa) or Cardiac Rhythm Management Device-Finder (CRMD-f) to identify device manufacturers in CIEDs.
METHODS: 547 patients who underwent CIED implantation from the year 2016-2020 in our institute were enrolled. There were 438 Medtronic and 109 St. Jude's devices. All chest radiographs were de-identified and resized into 225*225 pixels focusing on the CIED. PMIDa and CRMD-f applications were used to identify the CIED. Accuracy, sensitivity, specificity, negative predictive value, and positive predictive value for both applications were calculated and compared.
RESULTS: Overall, CRMD-f application has higher specificity (93.58 vs. 82.5%) but lower sensitivity (53.6 vs. 55%) than PMIDa. The accuracy of both applications was comparable (61.6% vs. 60.5%). Accuracy varied with CIED model and type tested, and radiograph projection used. Accuracy is greatest with Cardiac-Resynchronization-Therapy (CRT) devices for both applications, followed by a single lead pacemaker.
CONCLUSION: CRMD-f has higher accuracy and specificity for CIED manufacturer identification. Both PMIDa and CRMD-f are specific tools to identify CIED but have low sensitivity.

Introduction

Cardiac implantable electronic devices (CIED) are increasingly being implanted across the world. With an expansion in the number of CIED implanted every year, physicians frequently encounter the need of device interrogation [1]. Timely interrogation of CIED helps with quicker diagnosis and management of patients with cardiovascular disease. Furthermore, patients with CIED need device interrogation and changes in the device parameters to safely perform emergency surgery. Emergency Department (ED) staff can now interrogate CIED with similar interrogation time and have no impact on the length of ED stay and similar 30-days outcomes when compared to the standard procedures of interrogation in ED [2,3]. However, significant time is spent in retrieving information about the device manufacturer from the medical records in circumstances when the device identification card is not available, which can potentially delay the necessary therapies for certain arrhythmias or delay the emergency treatment required.

Smartphone-based applications like Pacemaker ID application (PMIDa) and Cardiac Rhythm Management Device-Finder (CRMD-f) have been designed to aid with quicker recognition of device manufacturers so that manufacturer-specific equipment is arranged for the device interrogation. Either of the two applications needs validation studies to ascertain their usefulness in device manufacturer identification.

PMIDa is a smartphone-based application that uses neural network-driven model to recognize the device manufacturer (Fig. 1) [4]. It is also available as a website. CRMD-f is a mobile-based application that was developed by Dr. Ines Sherifi and Tarun Kotia and includes a flow-chart-based inspection of CIED using the CaRDIA-X algorithm (Fig. 2). CaRDIA-X is Cardiac Rhythm Device Identification Algorithm using X-rays. This algorithm utilizes the fact that every device has certain unique radiographic and morphologic features such as the shape of the battery, CAN, and header position concerning the battery, presence of coils, etc., which can facilitate device identification [5] (Fig. 3).

Fig. 1

Pacemaker ID application user interface.

Fig. 2

Cardiac Rhythm Management Device Finder application user interface.

Fig. 3

Radiographic parts of the cardiac implantable electronic device.

Methods

This was a single-center, observational study approved by the Aga Khan University Hospital Ethics Review Committee (ERC Number: 2020-5101-14156). The patient enrollment period ranged from 2015 to 2020. All subjects who underwent CIED implantation during this study period and undergoing chest radiograph after the procedure at any time were enrolled in the study. Chest radiographs were obtained from the electronic medical record system of the hospital. Commonly used manufacturers in the institute included Medtronic and St. Jude Medical.

Physicians using the applications were blinded to the manufacturer and were pre-trained to use the applications on a set of 20 radiographs. A single Nokia phone (13-megapixel camera) was used for all the devices. Screen resolution was 1366*768. Both anterior-posterior (AP) and posterior-anterior projections (PA) were used.

547 CIED implanted from the year 2015–2020 were enrolled. Manufacturer representation includes Medtronic 80.1% (438) and St. Jude Medical 19.9% (109) of patients. The unequal distribution of the device manufacturer was due to local availability and hospital contract with the device manufacturer. All chest radiographs were de-identified. Each X-ray was cropped and resized into 225*225 pixels. For every CIED, manufacturer identification was attempted using PMIDa and CRMD-f. The physician using both methods was blinded of the original manufacturer. Sensitivity and specificity for both methods were compared.

Analysis was performed using SPSS version 23. Descriptive analysis was used to calculate respective frequencies. Cross-tabulation was used for calculating individual accuracies, sensitivities, specificities, negative predictive values (NPV), and positive predictive values (PPV). Accuracies were separately calculated for individual model and device type. For a PMIDa to detect manufacturer, a cut-off of 75% was used as a prediction certainty. The answer was considered correct for the applications if the answer matched the real manufacturer (known from medical records).

The work has been reported in line with the STROCSS criteria [6]. The project has been registered with clinicaltrial.gov (UIN: NCT04957108) [7]. https://clinicaltrials.gov/ct2/show/NCT04957108.

Results

A total of 547 CIED were analyzed. There were 368 dual-chamber pacemakers, 122 single implantable cardioverter defibrillators (ICD), 7 single lead permanent pacemakers, and 15 Cardiac Resynchronization Therapy (CRT) devices. Device models interrogated are as shown in Table 1. Chest radiograph projection was anteroposterior in 87 (14.6%) and posteroanterior in 460 devices (76.8%).

Table 1

Device models tested in the study.

Device Model	Number (n)	Percentage (%)
Medtronic PPM Sensia	89	14.9
Medtronic Evera S MRI ICD	30	5
Medtronic Bravaquard CRT-D	17	2.8
Medtronic Attesta PPM MRI	4	0.7
Medtronic Ensura DR MRI PPM	265	44.2
Abbot Fortify Assura	27	4.5
Others	115	21%

PMIDa sensitivity in identifying device manufacturers was 55% and specificity was 82.5%. PMIDa negative predictive value (NPV) was 31% and the positive predictive value (PPV) was 92.6%. CaRDIA-X algorithm carried a sensitivity of 53.6% and specificity of 93.58% with an NPV of 33.4% and PPV of 97% (Table 2 and Fig. 4).

Table 2

Sensitivity, specificity, and accuracy of Pacemaker ID app and CRMD-finder app.

Application	Sensitivity (%)	Specificity (%)	Accuracy (%)
PacemakerID	55	82.5	60.5
CRMD-Finder	53.6	93.58	61.6

Fig. 4

Sensitivity, specificity, and accuracy of Pacemaker ID app and CRMD-finder app.

When compared to PMIDa, the accuracy of CRMD-f was lower for Medtronic (56.6% vs. 57.9%) and higher for St. Jude Medical (94.4% vs. 75.2%). CRMD-f had 100% accuracy for single-lead PPM and CRTs. PMIDa also correctly identified 100% of CRTs. PMIDa had higher accuracy with AP-projection when compared to PA-projection (72.4% vs. 59.3%), whereas CRMD-f had higher accuracy with PA-projection (98.5 vs. 94%). (Table 3).

Table 3

- Comparison of our study with Chudow et al. demonstrating inter-study variability of accuracies of PMIDa and CRMD-f.

Device characteristic	PMIDa was correct Shams et al.		PMIDa accuracy Chudow et al.	CRMD-Finder was correct Shams et al.		CaRDIA-X accuracy Chudow et al.
Medtronic	57.9%		96%	56.6%		72%
St. Jude Medical	75.2%		89%	94.4%		84%
Dual chamber PPM	52.1%	52.5%		49.7%	50.6%	88%
Single lead PPM	71.4%			100%
CRT	100%			100%
Single lead ICD	75.4%		95% (For all ICDs)	93.4%		88% (For all ICDs)
Anterior-posterior projection	72.4%			94%
Posterior-anterior projection	59.3%			98.5%

Discussion

Cardiology staff on-call often gets called for urgent interrogation of CIED. This entails consults from ED or peri-operative areas. Often, our patients fail to show CIED identity cards in emergency situations. In such circumstances, where urgent interrogation and programming of CIED are needed and the manufacturer name and device type are not known, either a PMIDa or CRMD-f with CaRDIA-X algorithm can be used to identify the device manufacturer.

In this study, we have used CRMD-f mobile application instead of manual CaRDIA-X flow-chart and compared it with PMIDa. Our study points towards variability concerning the manufacturer, device type, and X-ray projections used. Overall, CRMD-f had higher specificity (93.58 vs. 82.5%) but relatively lower sensitivity (53.6 vs. 55%) than PMIDa. We compared our sub-group accuracies to the one demonstrated in Chudow et al. study and found the interpretations mentioned in Table no. 3. In essence, when compared to the manual CaRDIA-X flowchart used in Chudow et al. the CRMD-f application had greater accuracy for St. Jude's Medical and single lead ICDs whereas a lesser accuracy for Medtronic CIED. Based on the device type, both the studies reported lesser accuracy with PPM in comparison to ICDs.

Chudow et al. looked at the head-to-head comparison of the accuracy of various machine learning algorithms in identifying CIED. The study used a manual CaRDIA-X flow-chart whereas our study used a mobile phone application-based CaRDIA-X algorithm. Chudow et al. reported an average accuracy of 88% for ICDs and 80% for PPMs with a variability of 71–99% depending upon the manufacturer being tested. Likewise, PMIDa had an overall accuracy of more than 75% (range: 51%–100%). The manual CaRDIA-X flow-chart was reported to be time-consuming and cumbersome when compared to PMIDa [8]. This brought us to the testing of CRMD-f which is an application-based CaRDIA-X flow-chart and allows quicker user interface and identification of the device. In comparison to Chudow et al. study, we demonstrated lesser sensitivity and specificity, pointing towards inter-study variability.

The respective accuracies of our study and Chudow et al. for Medtronic devices were: 57.9 vs. 96% (PMIDa) and 56.6 vs. 72% (CRMD-f vs. manual CaRDIA-X flow chart). For pacemakers overall, the accuracies were 52.5 vs. 83% (PMIDa) and 50.6 vs. 88% (CRMD-f vs. manual CaRDIA-X flow chart). Both the studies individually demonstrated relatively higher accuracy for ICD vs. PPM when using PMIDa (This study: 52.5 vs. 75.4% for PMIDa; Chudow et al.: 83% vs. 95%). The inter-study variability for PMIDa can be explained by the difference in mobile phone manufacturer and camera megapixel used in both the studies (iPhone 8 12 megapixels versus Nokia 13 megapixels), the difference in screen resolutions used, and the difference in projections used. Additionally, differences in the extent of data augmentation (such as flipping and cropping of radiograph images) can lead to variability. The variability for CRMD-f or CaRDIA-X algorithm can be due to inter-physician variability i.e., level of professional training. This aspect determines one's familiarity with device identifiers on visual inspection of the chest radiograph.

Additionally, our study concluded that for a machine-learning-based application (PMIDa) AP projection yields greater accuracy (72.4% vs. 59.3%) whereas, for a flow-chart-based application, the PA projection yields higher accuracy (98.5% vs. 94%). This is likely because PMIDa involves taking picture of the device and an AP projection theoretically magnifies anterior structures (including CIED).

The use of the CaRDIA-X algorithm is shown to have intra-operator variability with the electrophysiologists showing best performance, owing to their greater familiarity with devices and algorithms. An accuracy-variability amongst physicians of 62.3%–88.9% has been reported [9]. In our study, CRMD-f was used by two cardiology fellows-in-training (FIT) of the same professional year who were pre-trained to grow familiarity with the radiographic anatomy of devices. Our study was limited by the fact that physicians interpreting the radiographs were not of different professional levels. Our study is clinically applicable and in more correlation with the ground reality, wherein emergency situations mostly cardiology FIT need to interrogate the CIED.

The use of neural-network-based artificial intelligence (AI) for identifying CIED is gaining attention. A single centered study compared the network with that of cardiologists and found that neural network-based AI performs better than the cardiologists to identify CIED (99.6% vs. 72%) [9]. However, other studies [8,10] including ours show variable accuracies pointing towards the variable performance of neural networks. The difference in the accuracy of various neural network methods can be explained by the difference in training sets and software used for machine learning.

Despite the variable results, artificial intelligence maintains its attraction due to its ease of use and quicker results. Importantly, like with any technology, it might restrict cardiologists’ ability to manually identify devices based on certain morphologic characteristics. CaRDIA-X is superior to PMIDa in terms of output because it not only identifies the device manufacturer but also the CIED type. The impact of these device identification tools on patient outcomes in terms of time-to-diagnosis and time-to-therapy has yet to be studied. We are planning to analyze the difference in prognostic outcomes concerning different modalities and applications used for device identification as our future project. We will assess if efficient and timely device information results in better management decisions for patients with CIED.

Study limitations

Our study was a single-centered study, and we were limited by the lack of availability of CIED manufacturers other than Medtronic and St. Jude Medical. Prospective data on patient outcomes is lacking and needs further research.

Conclusion

CRMD-f has higher accuracy and specificity for CIED manufacturer identification. Both PMIDa and CRMD-f are specific tools to identify but have lesser sensitivity. Accuracy is greatest with CRTs for both applications, followed by single lead PPM. Both methods demonstrate variability across the studies. Our study is one of the few studies testing AI for CIED identification using chest radiograph and the first one from a low-middle income country.

Provenance and peer review

Not commissioned, externally peer-reviewed.

Sources of funding

No funding acquired for this study.

Ethical approval

Ethical review committee of the Aga Khan University Hospital.

ERC Number: 2020-5101-14156.

Consent

Consent not applicable as no direct intervention or interaction with human subjects.

Author contribution

Pirbhat Shams: Study conceptualization and design of the project. Manuscript writing and revision. Muhammad Mehdi and Jamshed Ali: Data collection, data analysis and presentation and literature review. Intisar Ahmed, Sheema Saadia, Sameen Iqbal: Literature review, statistical analysis and first draft. Aamir Hameed Khan: Critical review of the manuscript. Yawer Saeed: Supervised from conceptualization to execution. Final editing and critical review of the manuscript.

Registration of research studies

Name of the registry: clinicaltrial.gov

Unique Identifying number or registration ID: NCT04957108

Hyperlink to your specific registration (must be publicly accessible and will be checked): https://register.clinicaltrials.gov/prs/app/action/LoginUser?ts=1&cx=-jg9qo4(Publicly available).

Guarantor

Dr. Yawer Saeed.

Declaration of competing interest

None of the authors has any conflict of interest to reveal.