Evidence-based medicine (EBM) in practice: agreement between observers rating esophageal varices: how to cope with chance?

In clinical medicine, several tests that are commonly performed (i.e., physical exams, x-rays, or endoscopy) rely on a degree of subjective interpretation by observers. A limitation for clinicians is the concept of agreement between two observers that is present and beyond chance. Clinical decisions based on the presence or absence of a finding in a diagnostic test (i.e., size of esophageal varices on endoscopy) are commonly reached depending on the agreement between observers. Observations that measure the agreement between two or more observers should include a formula that takes into account the fact that observers will sometimes agree or disagree by chance. The kappa statistic corrects for this chance agreement and lets the reader know how much of the agreement beyond chance the reviewers achieve. Kappa is widely used to measure interobserver variability, that is, how often two or more observers agree in their interpretations. A kappa of 1 indicates perfect agreement, whereas a kappa of 0 indicates agreement equivalent to chance. A limitation of kappa is that it is affected by the prevalence of the finding under observation.