Failure analysis of MetaMap Transfer (MMTx).

A pilot study was conducted to evaluate the performance of the MetaMap Transfer (MMTx), a tool that extracts terms from free text and suggests matches to concepts in the Unified Medical Language System' (UMLS'). Five participants, including a content domain expert and a UMLS Expert, manually extracted and mapped terms to UMLS concepts for two disease summary documents from NLM's consumer health site, Genetic Home Reference. The resulting adjudicated annotations were used as a gold standard. Differences in auto-mated term extraction and mapping between MMTx and MetaMap were noted. A failure analysis was conducted to categorize the types of terms not correctly mapped by MMTx. The most frequent type of failure (30%) resulted from missing inferential or world knowledge. Characteristics of each category are discussed. We distinguish between classes of failures that may be easily rectified, such as alternative retrieval strategies to extract exact matches, and ones that require additional research, such as coordinating conjunctions, co-reference resolution, and word sense disambiguation