Prediction and cross-validation of neural networks versus logistic regression: using hepatic disorders as an example.

The authors developed and cross-validated prediction models for newly diagnosed cases of liver disorders by using logistic regression and neural networks. Computerized files of health care encounters from the Fallon Community Health Plan were used to identify 1,674 subjects who had had liver-related health services between July 1, 1992, and June 30, 1993. A total of 219 subjects were confirmed by review of medical records as incident cases. The 1,674 subjects were randomly and evenly divided into training and test sets. The training set was used to derive prediction algorithms based solely on the automated data; the test set was used for cross-validation. The area under the Receiver Operating Characteristic curve for a neural network model was significantly larger than that for logistic regression in the training set (p = 0.04). However, the performance was statistically equivalent in the test set (p = 0.45). Despite its superior performance in the training set, the generalizability of the neural network model is limited. Logistic regression may therefore be preferred over neural network on the basis of its established advantages. More generalizable modeling techniques for neural networks may be necessary before they are practical for medical research.