Validity of evaluation approaches for outbreak detection methods in syndromic surveillance systems.

Dear Editor in Chief

Timely response to health events such as emerging diseases and outbreaks are a major public health priority. Outbreak detection methods and algorithms as the main tools for public health surveillance systems are under the umbrella of temporal and spatial methods (1). “There are three different approaches which might be used by syndromic surveillance systems to examine the performances of outbreak detection algorithms including real data testing, fully synthetic simulation and semi-synthetic simulation(2).” The first approach, i.e. real data testing, provide the highest degree of validity (3). Nevertheless, surveillance data for many of disease outbreaks or bioterrorist threats are not existent (4). Accordingly, there are few published studies in literature which evaluated the efficacy of the outbreak detection methods using real data testing approach (5–7). Consider to lack of surveillance data and need to know the performances of such outbreak detection algorithms under a wide range of outbreaks, semi-synthetic simulation approach were used by researchers (8–11). This evaluation approach allows the researcher to measure the performance of the algorithms at different circumstances at the expense of lower degree of validity in comparison to real data testing approach.

During the past ten years both simulated datasets and simulation software have been developed to evaluate outbreak detection methods with their own limitations including the hypothetical basis (12–16). Watkins RE and his colleagues developed a simulation method that allows evaluator to consider the distribution, size and shape of the real outbreaks in order to achieving higher degree of validity (16).

In the remainder of the letter, three strategies to improve the validity of the semi-synthetic simulation approach according to the author knowledge are explained. Considering the similarity of historical data on disease outbreaks in the literature and the size, shape and distribution of the injected spikes into non-outbreak baseline data can support the validity of your evaluation results as the first strategy. The similarity of the injected spikes and the previous outbreaks according to surveillance data, if the surveillance data on the interested disease or syndrome are available, should be considered as the second strategy. Considering the dynamic of disease’s transmission for different locations and circumstances through expert’s opinions is the last strategy to make a valid evaluation.