"Where do we teach what?" Finding broad concepts in the medical school curriculum.

BACKGROUND: Often, medical educators and students do not know where important concepts are taught and learned in medical school. Manual efforts to identify and track concepts covered across the curriculum are inaccurate and resource intensive.
OBJECTIVE: To test the ability of a web-based application called KnowledgeMap (KM) to automatically locate where broad biomedical concepts are covered in lecture documents in the Vanderbilt School of Medicine.
METHODS: In 2003, the authors derived a gold standard set of curriculum documents by ranking 383 lecture documents as high, medium, or low relevance in their coverage of 4 broad biomedical concepts: genetics, women's health, dermatology, and radiology. We compared the gold standard rankings to KM, an automated tool that generates a variable number of subconcepts for each broad concept to calculate a relevance score for each document. Receiver operating characteristic (ROC) curves and area-under-the-curve were derived for each ranking using varying relevance score cutoffs.
RESULTS: Receiver operating characteristic curve areas were acceptably high for each broad concept (range 0.74 to 0.98). At relevance scores that optimized sensitivity and specificity, 78% to 100% of highly relevant documents were identified. The best results were obtained with the application of 63 to 1437 subconcepts for a given broad concept. The search time was fast.
CONCLUSIONS: The KM tool capably and automatically locates the detailed coverage of broad concepts across medical school documents in real time. Use of KM or similar tools may prove useful for other medical schools to identify broad concepts in their curricula.