The standard error of Cohen's Kappa.

This paper gives a standard error for Cohen's Kappa, conditional on the margins of the observed r x r table. An explicit formula is given for the 2 x 2 table, and a procedure for the more general situation. A parsimonious log-linear model is suggested for the general case and an approximate confidence interval for kappa is based on that model. Numerical illustrations are given comparing the standard results with exact conditional results and those of the present paper. The conclusions are, first, that the standard error of kappa under the null hypothesis (that kappa is zero) should not be used except when the null is plausible; secondly, that the usual formula for the standard error appears appropriate except in the unusual circumstance of kappa taking large negative values; and, thirdly, in small samples the distribution of the estimate of kappa appears very non-symmetric, and it is preferable to base confidence intervals on transformations of kappa. A PC program with source code implementing the suggested procedures is available from the author.