Inconsistent visual analyses of intrasubject data.

Visual inspection has been the method of analysis most widely employed to evaluate the functional control demonstrated by any given set of intrasubject replication data. To identify the influence of certain graphic characteristics on these evaluative behaviors, 36 "ABAB reversal" figures were constructed. They were sent to 250 reviewers of behavioral journals. Their evaluation of each figure was expressed as a rating on a 100-point scale of "experimental control." Mean interrater agreement was 0.61. In addition to this rating, a verbal description of evaluation criteria was requested. It was also found that graphic characteristics determine evaluative judgments in concert rather than singly. For example, phase mean changes had to be a pattern consistent with the hypothesized effect of the experimental variables, while degrees of mean shift and variability were less important. A description of the following evaluative criteria was presented: (a) topographic characteristics, (b) format of data presentation, (c) intra-experimental, and (d) extra-experimental circumstances.