The 3-min appraisal of a diagnostic test.

INTRODUCTION

Diagnostic tests are invaluable tools used in various healthcare settings to distinguish between patients who have a disease and those who do not. It is essential for surgeons to be skilled in critically appraising published papers about a diagnostic test.1 This article will provide some simple and quick guidelines to assist in the 3-min critical appraisal process of the literature on diagnostic tests.

KEY CRITERIA FOR CRITICAL APPRAISAL

Many aspects of diagnostics need to be evaluated, and there are three specific areas that should be critically appraised in diagnostic test studies as described by Guyatt in his User's Guide to the Medical Literature: validity of the study, results, and the applicability of the diagnostic test [Table 1].

Table 1

Guidelines to critically appraising literature on diagnostic tests

Are the results of the study valid?

The validity of a diagnostic test study can be critically appraised through examining the study design. The patient population of the study should include a wide spectrum of patients with varying disease conditions and stages of treatment to ensure that there is genuine diagnostic uncertainty.23 Diagnostic uncertainty increases when symptoms of the target condition are also characteristic of other diseases.3 To minimize misjudgment of the study results, there should be a variety of patients.

Another essential component to analyze is if an independent (blind) comparison between the diagnostic test and an appropriate reference standard was done for each patient. For example, a surgeon wishing to understand whether the “Lachman” test for cruciate ligament is predictive of a tear could confirm the accuracy of the test by ensuring all patients tested received an independent “standard” confirmatory test (magnetic resonance imaging (MRI) or arthroscopic visualization). To increase the study validity and minimize potential bias in overestimating test outcomes, those interpreting the test results should be blinded and be different than those interpreting the reference standard.3 In this case, the surgeon performing the physical exam manoeuvre and grading the presence/absence of a positive Lachman would not be the same surgeon who reviewed the gold standard confirmatory test (MRI or arthroscopy).

The results of a diagnostic test should not influence a decision to perform the reference standard. This is referred to as verification bias.3 For example, verification bias results when only patients with a positive “Lachman” test get an MRI (or arthroscopy), but those with a negative test do not. A good study design should take verification bias into account and take measures for bias prevention.

What are the results?

The terms sensitivity and specificity are often used to describe the effectiveness of a test, but the literature has shown that likelihood ratios are a better statistical tool in aiding clinical decision making.2 Sensitivity is the proportion of individuals with a target condition that test positive and specificity is the proportion of individuals without a target condition that test negative. These dichotomous measures indicate either positive or negative tests, while likelihood ratios account for the cases in the middle of a wide spectrum of patients.23

Likelihood ratios combine specificity and sensitivity, which provides a better measure of the test as it links the pretest probability (prevalence of disease) to the post-test probability (chances of detecting the target condition with the diagnostic test).34 Likelihood ratios are a ratio of the proportion of positive test results versus the proportion of negative test results.5 A likelihood ratio equal or close to 1 means that the test has minimal value because it cannot differentiate between those who have the target condition and those who do not.4 A large likelihood ratio (>1) means a larger proportion of the test results will occur in positive patients, whereas a smaller ratio (<1) indicates a higher probability that the test result will occur more frequently in healthy patients.5 The ability to understand and interpret these ratios is essential in understanding quantitative features of a study.

Are the results of this study useful in my practise?

The last component to consider is the applicability of the study results to current practise. It is crucial to compare one's patient to the patient population in the study. The greater the similarities, the more appropriate and relevant the study is to one's practise. An accurate diagnostic test with minimal risks to the patient can be an invaluable stepping stone to improving healthcare. A diagnostic test that is easy to conduct and reproduce allows for a better integration into clinical practise.3

A PRACTICAL EXAMPLE

In the study by Iannotti et al., an office-based ultrasonography was examined for its accuracy in diagnosing rotator cuff tears.6

The patient population of Iannotti et al.'s study were all clinically diagnosed with rotator cuff symptoms. By having this inclusion criterion, it limits a wide spectrum of patients which may reduce the study validity.

This study clearly defined the diagnostic test (ultrasonography) and compared it with an appropriate reference standard (operative findings); however, some areas of the study design allowed for potential bias. The study design was complex involving consecutive radiographs, clinical examinations, ultrasounds, and preoperative MRI scans prior to the study. The orthopedic surgeon did the initial physical examinations and interpreted the radiographs. The surgeon's involvement in the preliminary assessment of the patient might have caused bias to his/her diagnosis after evaluating the ultrasound and preoperative MRI results. The authors’ rationale behind their design was that the diagnostic study should be carried out in a similar way similar to a typical clinical practise, where patients will undergo similar diagnostic procedures. The authors also suggested that the surgeon's bias from a previous involvement with the patient might actually lead to an increased accuracy of the test then the blinded radiologist.

The ultrasound had an accuracy of 80%, sensitivity of 88%, positive predictive value of 79%, negative predictive value of 90%, and a false-positive rate at 21%. These results were reported in a 3 × 3 table comparing three categories: no tear, partial-thickness tear only, and full-thickness tear with or without partial-thickness tear. If the latter two categories were combined to represent the target condition, and configured into a 2 × 2 contingency table, the likelihood ratios can be calculated.

The process of calculating likelihood ratios are shown in Table 2. The positive likelihood ratio is 4.8 and the negative likelihood ratio is 0.05. This indicates that a positive ultrasound is 4.8 times more like to occur in patients with a rotator cuff tear (either partial or full-thickness). Likelihood ratios are better measures of interpreting the results.

Table 2

Likelihood ratio calculations from the practical example

While this study had some limitations in its design, it concluded that office-based ultrasounds can be used effectively to diagnose rotator cuff tears when there are well-trained staff, preliminary clinical examinations, and radiographs.

SUMMARY

The ability to critically appraise the literature on diagnostic tests is essential for interpreting and understanding results of a study. The validity of the study, the results, and applicability of results to your patient population must be considered appropriately before applying the new knowledge in evidence-based orthopedics.