"Combined Diagnostic Tool" APPlication to a Retrospective Series of Patients Undergoing Total Joint Revision Surgery.

Background: Differentiating between septic and aseptic joint prosthesis may be challenging, since no single test is able to confirm or rule out infection. The choice and interpretation of the panel of tests performed in any case often relies on empirical evaluation and poorly validated scores. The "Combined Diagnostic Tool (CDT)" App, a smartphone application for iOS, was developed to allow to automatically calculate the probability of having a of periprosthetic joint infection, on the basis of the relative sensitivity and specificity of the positive and negative diagnostic tests performed in any given patient. Objective: The aim of the present study was to apply the CDT software to investigate the ability of the tests routinely performed in three high-volume European centers to diagnose a periprosthetic infection.
Methods: This three-center retrospective study included 120 consecutive patients undergoing total hip or knee revision, and included 65 infected patients (Group A) and 55 patients without infection (Group B). The following parameters were evaluated: number and type of positive and negative diagnostic tests performed pre-, intra- and post-operatively and resultant probability calculated by the CDT App of having a peri-prosthetic joint infection, based on pre-, intra- and post-operative combined tests.
Results: Serological tests were the most common performed, with an average 2.7 tests per patient for Group A and 2.2 for Group B, followed by joint aspiration (0.9 and 0.8 tests per patient, respectively) and imaging techniques (0.5 and 0.2 test per patient). Mean CDT App calculated probability of having an infection based on pre-operative tests was 79.4% for patients in Group A and 35.7 in Group B. Twenty-nine patients in Group A had > 10% chance of not having an infection, and 29 of Group B had > 10% chance of having an infection.
Conclusion: This is the first retrospective study focused on investigating the number and type of tests commonly performed prior to joint revision surgery and aimed at evaluating their combined ability to diagnose a peri-prosthetic infection. CDT App allowed us to demonstrate that, on average, the routine combination of commonly used tests is unable to diagnose pre-operatively a peri-prosthetic infection with a probability higher than 90%.

Introduction

Joint arthroplasty is one of the most common procedures in orthopedic surgery, and the number of both hip and knee arthroplasty procedures are expected to substantially grow over the next few years 1. Despite the advancements in implant design, fixation and infection control, prosthetic failure is still a relatively common occurrence, with a reported prevalence of approximately 4% 2. Periprosthetic joint infection (PJI) is amongst the main reasons for implant failure, and often results in prolonged and expensive treatment 2,3.

Although PJI can be clinically evident in some cases, in many others the diagnosis can be extremely challenging; in fact clinical presentation of PJI may only be characterized by unspecific pain at the operated joint and the differential diagnosis with other common reasons for implant failure, like aseptic loosening or neuropathic pain can be difficult, since no single diagnostic test is able to provide 100% sensitivity and specificity, thus the evaluation of a combination of tests becomes often necessary. Dosing of various serum and synovial markers, use of different imaging techniques, microbiological and histological analysis are then variably associated to refine the diagnosis, unfortunately often with conflicting results, whose interpretation is confusing and still generally based on an empirical evaluation 4-8. In fact, even the most recently published scores to define a PJI have not been clinically validated and rely on the “experts opinion” 9, while the proposed diagnostic protocols cannot avoid ending without the statement that an infection is “likely” or “unlikely”, without further specification of how much the likelihood is in a specific case 10. Unfortunately, however, the surgeon is requested to decide the most appropriate treatment strategy based on this likelihood.

This study utilizes a previously validated algorithm based on the known sensitivity and specificity of the performed tests11, to better calculate the relative probability of a joint prosthesis to be infected or not. Recently, we included this algorithm into a smartphone application, “Combined Diagnostic Tool” or “CDT App”, to further investigate its potential use as a readily accessible point of care and decision-making tool.

Objectives

Aim of the present study is to retrospectively evaluate through the CDT App the relative ability of the tests commonly performed in three high volume surgical departments in Europe to diagnose PJI. The secondary aim is to evaluate the number, type and results of the diagnostic tests routinely performed to diagnose a peri-prosthetic joint infection in three high volume centers.

Methods

Combined Diagnostic Tool App

The Combined Diagnostic Tool is an application developed for iOS working smartphones and tablets, based on a previously validated algorithm that combines the output of different diagnostic tests. The rationale, development and validation of the algorithm has been previously reported 11. Briefly, based on the sensitivity and specificity of each test, it is possible to calculate the relative proportion of True Negatives (TN), False Negatives (FN), True Positives (TP) and False Positives (FP) for each test. According to the formula:

We can then calculate the “Negative Test Index” (NTI), that indicates the relative chance that a subject, seen as negative by a given test, has of being truly negative, compared to falsely positive. The higher the value of NTI, the higher the chance that test correctly indicated that subject as NOT having the disease.

Similarly, we can calculate the Positive Test Index (PTI) according to

PTI indicates the relative chance that the subject is a TP compared to a FP. The higher the value of PTI, the higher the chance that test correctly indicated that subject as HAVING the disease.

Assuming that we have n independent tests, performed on a given subject, the combined chance of positive or negative results to be true is, respectively:

and

The ratio between PTI and NTI will then be indicated as the Combined Tests Index (CTI):

CTI indicates how many times the output of the combined positive tests is, compared to the output of the combined negative tests. The higher its value, the higher the chance the positive results are “more true” and that the subject HAS the disease and vice versa.

The “Combined Diagnostic Tool” App, specifically designed for the combined diagnosis of peri-prosthetic joint infections, works with reference values of sensitivity and specificity of the most common diagnostic tests as reported in the literature and in the App itself. Based on these reference values and on the results of different tests in a given subject, the software is able to automatically calculate the NTI, PTI and CTI of any tests combination. CTI values are then expressed as relative probability of having or not the infection both as a proportion and as a percentage: 50% change or 1 : 1 means equal chance of being infected or not, 3 : 1 or 75% means that a patient has 75% chance of being infected compared to 25% of being not, etc.

In addition the App offers the possibility to update the values of sensitivity and specificity of each test as they become updated in the literature, and to include new tests that may become available for clinical use. Moreover, it allows to simulate what could be the relative contribution to the final diagnosis of any further test that the physician would like to perform in a given subject. The reference values used in this study are those listed in our previous paper 11 and given in the Appendices.

Study population and pre-operative analysis

This study was undertaken across three-centers, and included the Istituto Ortopedico IRCCS Galeazzi, Milano, Italy; Istituto Clinico Humanitas, Rozzano, Italy; and Sheffield Teaching Hospital, Sheffield, UK. Given the retrospective nature of the study and the post-hoc analysis performed on the data, an informed consent from the patients for study participation was not required, according to the policy of the respective participating centers.

A total of 120 consecutive patients, treated during 2011-2012 with total hip or knee revision at a minimum of 12 months follow-up, were included in the study. Patients were classified in two groups, according to the surgical treatment received, as Group A (treated as infected: N=65) and Group B (patients treated as not-infected: N=55) (Table 1).

Table 1

Pre-operative data of the population under study.

	Group I - Patients treated as infected (N=65)	Group NI - Patients treated as not infected (N=55)
Sex (Male / Female)	28 / 37	25 / 30
Age (Mean +- S.D.)	64.3 +- 15.8	67.2 +-16.0
Joint (Hip / Knee)	31 / 34	28 / 27

For each patient, the type, number and result (positive or negative) of the diagnostic tests performed pre- and intra-operatively to confirm or to exclude a peri-prosthetic infection were recorded and analyzed with the CDT application, in order to obtain the pre-operative and post-operative calculated chance of being or not infected and to compare this with the treatment finally performed in each patient.

Pre-operative tests included serological testing, joint aspiration and imaging techniques (radiology, sonography and bone scans), depending upon each center's protocols.

Serological testing was considered positive following each center's cut off values.

Joint aspiration was considered positive if the WBC count in synovial fluid was above 3000 cells/µL or if cultural examination found microorganisms.

Intra-operative tests included histological and microbiological sampling. In particular, frozen and permanent histological analysis was performed in all participating centers on at least three samples obtained from the pseudocapsule, the membrane around the prosthesis or tissue suspected for infection (i.e. fibrotic or necrotic reactions). The samples used for frozen-section analysis were immediately frozen in carbon dioxide; 4mm section were cut and stained with standard hematoxylin eosin. Most cellular areas in each sample were chosen, and the number of neutrophils in at least ten high power field, HPF (x400) was counted. The presence of infection was defined according to Feldman criterion 12, i.e. the presence of at least five neutrophils per HPF. Results obtained from frozen-section histology were then computed with CDT app, thus obtaining an intra-operative CTI.

Permanent histological analysis and cultural examinations were performed according to each Institution protocol and surgeon preferences. Samples for histological analysis were collected in the same way already described for frozen-sections. If both frozen section and permanent histological analysis were performed on the same patient, each sample was divided in two parts, one for each type of examination. Samples for permanent histological analysis were immediately fixed in formalin and embedded in paraffin before the hematoxylin eosin staining. Criteria used for define infection were the same described above for frozen samples.

Microbiological analysis required 4 to 6 samples from different sites of periprosthetic tissue. Liquid samples were aspirated using a sterile syringe, and immediately inoculated into an automated culture system, and underwent 7 days of incubation. Positive samples were subsequently cultured in anaerobic and aerobic agar media. Solid tissue samples from periprosthetic tissue were placed into a sterile case, and subsequently cultured in anaerobic and aerobic agar media and in thyoglicolate broth enriched with vitamin K and hemin. After an incubation time of 10 days positive cultures were sent for organism identification and sensitivity testing.

Results from histology and cultural examination were computed in the CDT app, thus giving the final CTI for each patient included in the study.

Statistical analysis

The following parameters were evaluated per patient; number of tests performed pre-operatively per patient; number of positive and negative tests per patient; CDT App calculated chance of having a PJI.

Statistical analysis was performed using Stata software (version 11.0, StataCorp, Texas, USA). Percentage of positive tests in both groups were compared using Chi-square test, with a p-value of <0.05 was deemed statistically significant.

Results

Serological tests were the most commonly performed, with an average 2.7 tests per patient for Group A and 2.2 for Group B, followed by joint aspiration (0.9 and 0.8 tests per patient, respectively) and imaging techniques (0.5 and 0.2 test per patient). Serum markers were positive in 59.7% in Group A and 22.8% in Group B (p<0.01).

Joint aspiration were found positive for PJI in 65.6% in Group A and 24.4% in Group B (p<0.01). Pre-operatory imaging was suggestive for infection in 74.2% in Group A and 22.2% in Group B (p<0.001) (Table 2 and Table 3).

Table 2

Type, number and results (positive or negative) tests performed in patients eventually treated as infected (Group A, n =65)

	Positive tests (%)	Negative Tests (%)	Total	Average Tests per Patients
Serum Markers	105 (59.7)	71 (40.3)	176 (46.3)	2.7
Joint Aspiration	40 (65.6)	21 (34.4)	61 (16.1)	0.9
Imaging Techniques	23 (74.2)	8 (25.8)	31 (8.2)	0.5
Frozen Sections	10 (90.9)	1 (9.1)	11 (2.9)	0.2
Intra-op Cultures	57 (63.3)	33 (36.7)	90 (2.7)	1.4
Histology	11 (100)	0 (0)	11 (2.9)	0.2
Total Exams	243 (64.7)	134 (35.3)	380 (100)	5.8

Table 3

Type, number and results (positive or negative) tests performed in patients eventually treated as non-infected (Group B, n = 55)

	Positive tests (%)	Negative Tests (%)	Total	Average Tests per Patients
Serum Markers	28 (22.8)	95 (77.2)	123 (45.7)	2.2
Joint Aspiration	11 (24.4)	34 (75.6)	45 (17.4)	0.8
Imaging Techniques	2 (22.2)	7 (77.8)	9 (3.5)	0.2
Frozen Sections	0 (0)	1 (100)	1 (0.4)	0.02
Intra-op Cultures	8 (10)	72 (90)	80 (30.9)	1.5
Histology	0 (0)	1 (100)	1 (0.4)	0.02
Total Exams	49 (18.9)	210 (81.1)	259 (100)	4.7

On the basis of the pre-operative tests, the mean chance of having an infection, as calculated with the CDT App, was 79.4% ± 30.3% (range 0.2%-99.99%) for patients in Group A and 35.7 ± 36.0 (range 0.68%-99.86%) in Group B. Twenty-nine patients in Group A had > 10% chance of being NOT infected and 29 of Group B had > 10% chance of having an infection (see Table 4).

Table 4

Mean +- SD and minimum and maximum chance (%) of having a peri-prosthetic joint infection, calculated with the CDT App on the basis of pre-, intra- and post-operative findings. Group NI: patients treated as not infected; Group I: patients treated as infected.

	Group NI	Group I
Pre-operative	35.7 +- 36.0 (0.68 - 99.86)	79.4 +- 30.3 (0.2 - 99.99)
Intra-operative	35.8 +- 35.9 (0.68 - 99.86)	77.5 +- 32.8 (0.2 - 99.99)
Post-operative	22.9 +- 29.6 (0.05 - 97.82)	82.6 +- 33.7 (0.02 - 99.99)

Intra-operative histology (frozen sections) had been performed in only 22 patients. When implemented in the calculation of each patients probability of being infected, in combination with pre-operative findings, resulted in a chance of being infected of 95.7% ± 5.1% (min: 85.4%; max: 99.96%) for patients in Group A, and 10.2% ± 13.3% (min: 10.0%; max: 18.6%) in Group B.

A total of 170 intra-operative cultures were performed, 90 in Group A patients (mean 1.4 per patient) and 80 in Group B (mean 1.5 per patient). In Group A 63.3% of intra-operative cultures were positive for the presence of microorganisms, while in Group B 10% yielded a positive result (p<0.0001).

Eleven patients in Group A underwent tissue sampling for histological examination. All the samples obtained in this group confirmed the diagnosis of infection according to Feldman criteria.

Both histological findings and cultural data were implemented in each patient's individual score calculation, which lead to a post-operative chance of having a PJI of 90.0% ± 25.4% (min: 5.3%; max: 99.99%) for patients in Group A and 25.4% ± 24.0% (min: 0.05%; max: 69.0%) in Group B.

Discussion

This is to our knowledge the first multicenter study reporting on the number, type and results of the diagnostic tests routinely performed to diagnose a peri-prosthetic joint infection in patients undergoing revision surgery. Moreover, this is the first report on the application of a previously validated algorithm to the calculation of the relative chance of being infected, based on a combination of positive and negative tests, as often occurs in the clinical setting. Furthermore, this study shows how said algorithm can be implemented and made readily available to physicians as a smartphone application.

Concerning the type and number of tests, our data shows how, on average, each patient undergoing revision surgery undergoes more than five different examinations; in this regard, those more often investigated are serum markers, with a mean of 2.7 tests per patient in Group A and 2.2 in Group B. This appears to be in line with many recommendations, that report C-reactive protein (CRP) and erytrosedimentation rate (ESR) as a mandatory part of any protocol to rule out or to confirm a PJI 10,13,14. The second most executed test was joint aspiration for synovial fluid analysis, with a mean of 0.9 tests per patient in Group A and 0.8 in Group B. In fact, various studies evaluated the WBC in synovial fluid in patients with suspected PJI, showing consistent elevation of WBC in infected subjects when compared with non-infected cases 13,15,16. Elevated serum CRP and ESR and elevated WBC in synovial fluid have also been recently listed as useful criteria to define the presence of a PJI, however, neither are pathognomonic and hence it is recommended that all are performed and evaluated in light of the individual patients other findings 14.

The aim of this study was to evaluate the relative ability of the tests commonly performed in three high volume surgical departments in Europe to diagnose PJI through the CDT App. The App allowed us to show that, on average, the pre-operative investigations are unable to confirm or to exclude the diagnosis of infection with a probability higher than 90%. Indeed, while our data shows how each preoperative test performed is more frequently positive in patients treated as infected, the CTI calculation highlight the fact that the probability of actually having a PJI is only the 79.4% in Group A, compared with the 35.7% in Group B, with an extremely high variability. In this context, it must be noted that this diagnostic uncertainty is present every time we interpret examination results in clinical practice, and the CDT data offers just a mathematical quantification of that uncertainty.

When dealing with preoperative diagnostics, the routine use of CDT app can be extremely helpful for two main reasons. First, knowing the probability that a given patient actually has a PJI after performing the first line examination, and considering the implication that this diagnosis has on the treatment, can help the surgeon in deciding if further examination are needed to increase the accuracy of the diagnosis.

Second, the app can be used to simulate and quantify the impact that a given test has on the overall probability of the presence of a PJI. For example, we can hypothesize a situation in which, after a positive ESR and CRP, and a joint aspiration with a normal WBC count, using CDT app we found a probability of having a PJI of 62%. If we consider this probability too low to decide for treatment, we might wonder what would be the impact of performing a leukocyte bone scan on the diagnostic accuracy. Using CDT app we could calculate and quantify this data before performing the examination, thus basing decision to perform the test or not on accurate statistical information. This approach can have also a significant economical implication, since knowing the impact of a given test on the diagnostic accuracy in a given patient, could help in deciding whether performing an additional test is cost effective.

These considerations can be extended to the intra-operative investigations. Using the CDT App to calculate the impact that either frozen section and definitive histological examination have on the patient can help the surgeon in surgical planning, especially when frozen section is needed. According to our data, in Group A patients the percentage of frozen section that yielded a positive result was 90.9% (10/11), while the single patient in Group B who underwent frozen section analysis had a negative result.

Several different histological criteria for defining infection are reported in literature. In some cases more comprehensive criteria were used, such as 1-cell per HPF or even the presence of lymphocytes or plasma cells 17,18, while other studies reported more restrictive criteria, such as >5 neutrophils in at least 5 HPF 12,19. The former criteria favor sensitivity over specificity, while the latter favor specificity over sensitivity. The choice between comprehensive or restrictive criteria depends on the role frozen section has in the diagnostic process, whether used as a screening tool or as a confirmatory test. In the ultimate analysis, the efficacy of the chosen criteria depends on the pre-test probability of having the disease, with stringent criteria more efficacy in a situation of high pre-tests probability. Since the pre-test probability is easy calculated with CDT app, selecting patients who truly benefits from frozen section analysis become simpler and more precise.

One of the main finding of the present study is showing how a complex algorithm can be easily implemented in clinical practice. Regarding this point, older reports about routine application of diagnostic software in clinical practice identified the time required for data input as one of the main limiting factors20,21. Moreover, in a pivotal review on algorithm-based diagnostic supports, Kawamoto identified the availability at the time and location of decision making and the integration in routine workflow two key features needed for success of these systems 22. In recent years technological advances in smartphones have led to a widespread use of portable devices with high computing power. Their availability and easy to use interface makes them the perfect tool to facilitate the application of complex algorithms in clinical practice. In this context the development of a smartphone application fits perfectly. Indeed, the main advantages of the CDT app are its portability, quickness and applicability at the point of care.

Another important advantage of the CDT app is that the sensitivity and specificity data used in the algorithm for each test can be easily updated with the most recent publications found in the literature. This allows the tool to be always up to date.

The present study has limitations. Reference data for sensitivity and specificity used in CDT app algorithm are taken from literature studies. This introduces several bias; (a) population in which a give test was studied may differ from population in which the CDT app is used, (b) frozen section analysis, different diagnostic criteria and reference normal value for a given test differs in each laboratory, (c) the type of tests used in the participating centers and available during years 2011-2012, (d) current novel biomarkers and synovial fluid tests, such as alpha-defensin or esterase, are more often performed and were not included in this analysis, and (e) we applied the CDT app to two cohorts of patients treated respectively as infected or non-infected on the basis of a selection of tests decided by the surgeons, whilst the choice of the tests and their interpretation could have been different in each hospitals. Prospective, well designed studies are needed to further evaluate the role of the CDT app as an aid to diagnose PJI.

Conclusion

In conclusion, with this study we have shown that PJI diagnosis is the result of a combination of tests; the CDT app, a smartphone application designed to aid the surgeon in PJI diagnosis by calculating the overall probability of presence of PJI, when applied retrospectively did reveal that, on average, patients treated as infected still had approximately 20% chance of being not-infected and vice versa. The app, updated on a regular basis with new data and tests, may be extremely helpful in designing individual diagnostic approaches and in planning further examinations. Moreover, accessibility and ease of use of CDT app make it the ideal tool for point of care application of complex algorithm.

Table S1.

Click here for additional data file.