Application of psychometric standards to scoring system evaluation: is "new" necessarily "improved"?

Psychometric guidelines are suggested that expand the evaluation of scoring systems beyond the most frequently (and often singly) considered criterion, that of interrater reliability. Principles of reliability and validity are elaborated and discussed with regard to scoring systems. Also addressed are assays for efficiency in applying a new system, the effects of rater proficiency on outcome scores, the comparison of costs and benefits of new scoring systems, and the comparison of a new scoring system to an existing system to demonstrate its advantage. As a timely example to illustrate these principles, a new scoring system is introduced for the Rey-Osterrieth Complex Figure Test and is compared to the most popular scoring system for this test (Lezak, 1983, p. 400). The proposed principles would be relevant to any scoring system that codes subjective behavioral data into objective values.