Agreement between experts: an answer, but is it the answer for you?

The question of how should agreement between experts be measured, when their opinions are expressed on an ordinal scale, is discussed. The example used is that of quality of articles submitted to Journal of Clinical Neuroscience. Polychoric correlation is a useful concept, and there are now computer programs available to estimate this. However, it is also emphasised that the definition of agreement is a matter of choice, and the choice should be made carefully: even error-free measurements should not automatically be assumed to possess all the properties of numbers, and ordinal categories bring further difficulties.