Variability of judgments of capacity: experience of capacity evaluators in a study of research consent capacity.

BACKGROUND: Assessment of decision-making capacity is a common and important function of psychiatric consultants. However, the sources of variability in evaluators' judgments have not been well characterized.
OBJECTIVE: To examine the degree and potential sources of variability in the categorical capacity judgments of experienced psychiatrists.
METHOD: The setting was a study comparing the decision-making capacities of 188 persons with Alzheimer's disease to appoint a research proxy and to consent to two hypothetical randomized controlled trials for dementia (a new drug RCT and a neurosurgical RCT). We compared five experienced consultation psychiatrists' capacity judgments for 555 videotaped capacity interviews. Both quantitative and qualitative data were used.
RESULTS: Pair wise kappa statistics ranged from slight agreement (0.17) to substantial agreement (0.64) with group kappa statistics ranging from fair to moderate agreement (0.40 to 0.45) for the psychiatrists' judgments regarding the three capacities. The sources of variability included varying "strictness" among judges, moderate test-retest reliability within judges, the relative novelty of assessing decision-making capacity for research participation decisions, as well as the limitations of the methods used to obtain capacity judgments in the study. DISCUSSION: There is considerable variability in capacity judgments of experienced consultation psychiatrists regarding the capacities to appoint a research proxy and to consent to research. The potential sources of variability identified in this study may provide starting points for more effective training in capacity assessment.