Matthew D Mitchell1. 1. Technology Assessment Group, ECRI, 5200 Butler Pike, Plymouth Meeting, PA 19462, USA.
Abstract
RATIONALE AND OBJECTIVES: The author performed this study to test a technique for validating the logit regression method for summary receiver operating characteristic (ROC) meta-analysis, perform initial validation studies, and identify areas for further investigation. MATERIALS AND METHODS: Monte Carlo simulation was performed by using a custom macro program for a personal computer spreadsheet. The program creates simulated data sets based on user-specified parameters, performs a meta-analysis on the data sets, and logs the results so the accuracy and variability of the method can be measured. The program can also be used to measure the effects of changes in study design and meta-analysis parameters. RESULTS: For the base case of a small meta-analysis (10 studies) of small trials (mean, 50 patients), the meta-analysis results closely matched the input sensitivity and specificity when they were less than 80%. Systematic errors, if any, were small. At sensitivities and specificities greater than 80%, the true sensitivity or specificity was underestimated by up to 2% in the meta-analysis. Confidence intervals calculated with the summary ROC curve were reasonably conservative, although they too fell below the true results when sensitivity or specificity was greater than 80%. The underestimation was eliminated when the simulation was repeated for a much larger trial (mean, 1,000 cases per study)--even with a sensitivity and specificity of 98%. CONCLUSION: The Littenberg-Moses method for summary ROC meta-analysis is effective for obtaining an accurate summary estimate of diagnostic test performance, although the continuity correction introduces a small downward bias in the meta-analysis of small trials.
RATIONALE AND OBJECTIVES: The author performed this study to test a technique for validating the logit regression method for summary receiver operating characteristic (ROC) meta-analysis, perform initial validation studies, and identify areas for further investigation. MATERIALS AND METHODS: Monte Carlo simulation was performed by using a custom macro program for a personal computer spreadsheet. The program creates simulated data sets based on user-specified parameters, performs a meta-analysis on the data sets, and logs the results so the accuracy and variability of the method can be measured. The program can also be used to measure the effects of changes in study design and meta-analysis parameters. RESULTS: For the base case of a small meta-analysis (10 studies) of small trials (mean, 50 patients), the meta-analysis results closely matched the input sensitivity and specificity when they were less than 80%. Systematic errors, if any, were small. At sensitivities and specificities greater than 80%, the true sensitivity or specificity was underestimated by up to 2% in the meta-analysis. Confidence intervals calculated with the summary ROC curve were reasonably conservative, although they too fell below the true results when sensitivity or specificity was greater than 80%. The underestimation was eliminated when the simulation was repeated for a much larger trial (mean, 1,000 cases per study)--even with a sensitivity and specificity of 98%. CONCLUSION: The Littenberg-Moses method for summary ROC meta-analysis is effective for obtaining an accurate summary estimate of diagnostic test performance, although the continuity correction introduces a small downward bias in the meta-analysis of small trials.
Authors: Clare Davenport; Elaine Yee Lan Cheng; Yip Tung Tony Kwok; Annie Hiu On Lai; Taka Wakabayashi; Chris Hyde; Martin Connock Journal: Br J Gen Pract Date: 2006-01 Impact factor: 5.386