Building and evaluating annotated corpora for medical NLP systems.

We report efforts from a pilot annotation project in which we annotated sixty clinical notes with disorder mentions. Four retrieval experts annotated the same clinical notes so that four-way inter-annotator agreement could be calculated. We find the inter-annotator agreement results encouraging to scale up the project using the same guidelines and annotation schema.