Merit of test: Perspective of information economics.

This article assesses the merit of a test through the lenses of economics, with applications to SARS-CoV-2. This allows us to rank distinct tests and to show that this ranking is not universal; it depends on the pre-test information available to the decision-maker and the losses stemming from incorrect actions. We provide a method to select, from multiple tests with different sensitivity and specificity, the test that helps the decision-maker the most to achieve her objective.

Introduction

This article assesses the merit of a test through the lenses of economics, with applications to SARS-CoV-2.1 We define the merit of a test to be the extent its outcome improves the choice of the decision-maker. This definition allows to rank distinct tests. We show that this ranking is not universal; it depends on the pre-test information available to the decision-maker and the losses stemming from incorrect actions. We provide a method to select, from multiple tests with different sensitivity and specificity, the test that helps the decision-maker the most to achieve her objective.

The method is suitable to describe the clinical as well as the epidemiological merit of a test. For the attending physician, the merit of the test is high if it helps providing optimal care. From an epidemiological perspective, a test should identify infected individuals before they go on to infect others. In both cases there is a decision maker that can test to achieve an objective. A portfolio of tests with distinct sensitivity and specificity parameters is useful, both for the physician and the epidemiologist, since one can adjust the choice of the test to their particular circumstance. Returns of such sophistication in the test choice will be large if careful statistical work will allow us to reliably assess the pre-test infection probabilities and when the employed tests will be carefully validated.

Framework

An individual is or is not infected by SARS-CoV-2; that is her state is C or N. If the decision-maker (DM) knew this state, she would choose action a when the individual is infected and action a when the individual is not. When the DM does not know the true state of the individual, she could choose the wrong action thereby incurring a loss. In particular, the loss is ℓ when the individual is infected but DM chooses action a; analogously, the loss is ℓ when the individual is not infected but DM chooses a. The values of the two losses are free parameters of the framework and can be adjusted to specific applications.2 Under incomplete information, the DM follows four steps: (i) she formulates a pre-test probability that the individual is infected,3 (ii) applies a test to the individual, (iii) uses the test result and the Bayes rule to formulate a post-test probability that the individual is infected, (iv) selects the action that minimises the expected loss. This motivates the following definition.

The merit of a test is the difference between the expected loss that the DM incurs if she acts solely based on the pre-test information and the expected loss that the DM incurs if she acts based on the post-test information.

We use Definition 1 to compare performances of distinct tests. We endow the DM with a portfolio of tests that differ in their sensitivities and specificities. The test sensitivity describes the probability that its result is correct when the individual is infected. The test specificity describes the probability that its outcome is correct when the individual does not have SARS-CoV-2. For example, Fig. 1 plots the merit of a test with a sensitivity and a specificity of 90%; it also assumes that both types of the decision error lead to a loss of one unit. When the pre-test probability of infection is either low or high, the test never improves the choice of the decision maker, and so it has no merit. This is because for an extreme enough pre-test probability, the result of imprecise test does not affect the DM’s choice. For the intermediate pre-test probabilities the outcome of the test informs the decision, and thus the test has a positive merit. The merit is maximal when the DM, in absence of the test, cannot do better than deciding by a coin flip (e.g. 1/2 pre-test probability for equal losses).

Fig. 1

A test with 90% sensitivity and specificitiy.

In the formal analysis below we derive the merit of a test for arbitrary specificity, sensitivity and pre-test probability. In what follows, we develop two applications.

Example 1. PCR or CT. Ai et al. [6] compare diagnostic accuracy of the PCR testing with that of radiologic CT-based diagnostics. Based on this study we set the sensitivity and specificity of the PCR test to be 70% and 99%, respectively. The CT-based diagnostics is highly sensitive but it has low specificity due to many false-positive errors; we set its sensitivity to 96% and its specificity to 25%. In this example, we assume that both types errors lead to a same loss of one unit. Fig. 2 plots the merit of each of the two tests for different pre-test probabilities. Because of its high rate of the false-positive errors, CT-based diagnostics is dominated by the PCR testing for most of the values of the pre-test probability P. However, when the pre-test probability is between 0.74 and 0.86, CT-based diagnostics dominates because the high false-positive error rate of the CT diagnostics is less detrimental than the intermediate false-negative error rate of the PCR testing. If the health professional can choose only one of these two diagnostic approaches, perhaps due to the time constraints, then CT scan is optimal for this group of patients. If both diagnostic procedures are done and they return contradictory results, then under these circumstances, it is optimal to follow the CT result.

Fig. 2

PCR test (blue) vs. CT diagnostic (yellow). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Example 2. PCR analysis: oropharyngeal (OP) vs. nasopharyngeal (NP) swabs. Carver and Jones [7] compare accuracy of the PCR analysis of the oropharyngeal (OP) and nasopharyngeal (NP) swabs. Based on this study, we assume a 70% sensitivity of NP, 60% sensitivity of OP, and a high specificity of 99% for both. Without further considerations, the NP method has unambiguously higher merit. In practice, however, the NP test is more unpleasant for the patient. When the test is being done for an epidemiological purpose and only asymptomatic individuals are targeted for the test, this could lead to a lower compliance when a NP method is used. For concreteness, we assume that only 3 out of four 4 targeted individuals agree to be tested with the NP swab, whereas the OP swab has full compliance. In this epidemiological application, we assume that the loss stemming from the false-negative error is four times bigger than the loss from the false-positive error. Fig. 3 plots the merit of the two tests where the test merit for NP swab is scaled by factor . For low pre-test probabilities the OP swab method dominates since the minor increase in the false-positive rate is dominated by the benefit of the larger compliance.

Fig. 3

PCR NP swab (blue) vs. PCR OP swab (yellow). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Conclusion

This article provides a methodology that ranks available diagnostic tests as a function of their sensitivities, specificities, the losses of both types of the decision error, and of the pre-test probability of infection.

Additional information: formal analysis

The state of the individual is θ ∈ {N, C} where N stands for not having SARS-CoV-2, and C for the infection. The decision-maker (DM) chooses between two available actions, a and a, where the first is optimal when and the latter is optimal when . If the DM does not choose the optimal action, then she suffers a loss ℓ or ℓ for the state of the individual N or C, respectively. No loss is incurred if the action is chosen optimally.

The DM does not know the individual’s state. Let Q ∈ [0, 1] denote the probability that the DM attaches to the individual having SARS-CoV-2 at the point of decision (and let be the probability the DM attaches to the individual not having SARS-CoV-2). As explained in the main text the DM, who chooses whichever action leads to a smaller expected loss, incurs the expected loss given her belief Q equal to

Let P ∈ [0, 1] be the pre-test probability that the DM attaches to the individual having SARS-CoV-2 before she tests her. In absence of the test, the DM cannot learn further information, and her expected loss is therefore L(P).

We endow the DM with a test with sensitivity Se ∈ [0, 1] and specificity Sp ∈ [0, 1]. The DM applies the test to the individual, observes its result, and updates her belief about the individual’s SARS-CoV-2 status according to the Bayes law. If the DM observes the positive test result, she will assign positive post-test probabilityto the individual having SARS-CoV-2. Similarly, if the DM observes the negative test result, she assigns negative post-test probabilityto the individual having SARS-CoV-2.

The imprecise test returns positive result with probability which accounts for both the true positives and for the false-positives. Similarly, the test returns negative result with probability . Overall, the expected loss of the DM who has access to the test isThe merit of the test,is the difference between the expected loss L(P) that the DM would have incurred in absence of the test, and the expected loss in Eq. (1) obtained with testing.4

Author Statements

Funding

None

Declaration of Competing Interest

None declared

Ethical approval

Not required

CRediT authorship contribution statement

Andrea Galeotti: Conceptualization, Formal analysis, Data curation, Writing - original draft, Writing - review & editing. Jakub Steiner: Conceptualization, Formal analysis, Data curation, Writing - original draft, Writing - review & editing. Paolo Surico: Conceptualization, Formal analysis, Data curation, Writing - original draft, Writing - review & editing.